From 240eaa081a665d1010b916b067930d3f9f407c9f Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 13:27:49 -0600 Subject: [PATCH 01/28] docs: add background jobs + push notifications design doc Gatekeeper-SEALED + Oracle-APPROVED v1.8 (2026-08-24 gates; 2026-08-25 accuracy refresh against the MCP resources/prompts merge). --- plans/background-jobs-design.md | 1200 +++++++++++++++++++++++++++++++ 1 file changed, 1200 insertions(+) create mode 100644 plans/background-jobs-design.md diff --git a/plans/background-jobs-design.md b/plans/background-jobs-design.md new file mode 100644 index 0000000..c4a52fc --- /dev/null +++ b/plans/background-jobs-design.md @@ -0,0 +1,1200 @@ +# Design: Background Tool Jobs & Completion Push Notifications + +Status: DRAFT v1.6 — grounded in src/function/mod.rs (eval_tool_calls, escalation +injection), src/supervisor/mod.rs (Supervisor/AgentHandle), src/function/ +supervisor.rs (agent__* handlers, turn-end guardrail), and Oracle rulings +(agent_oracle_10936145, persisted as memory `coyote-bg-jobs-notifications-rulings`). +User rulings R6/R7 added 2026-08-21; R7 amended + R8/R9 and §9 (regression +parity) added same day after injection-gating and test-coverage exploration. +§13 items ALL RESOLVED (final code audits 2026-08-21: JobCtx, JobEnvSnapshot, +sbx process model, truncation). Gate audit 2026-08-21 (LEAKY, 8 findings, +11/11 receipts verified) remediated here. Oracle plan review 2026-08-21 +(agent_oracle_f0d3932c): architecture + T0–T7 decomposition APPROVED; +findings folded in: shared JobState cell w/ pgid-clear (pid-reuse guard), +use_agent reset-site correction (there is NO full-context /clear), explicit +shutdown mechanism, capacity-0 supervisor consumer audit, panic-notification +semantics, tail-side UTF-8 flooring, lock discipline, parallelization map. +Re-gate on v1.5 (all prior findings verified fixed, receipts 100%): 3 +residual gaps fixed in v1.6 — lazy plain-session supervisor init (R9/§6), +T2∥T3 function/mod.rs caveat (§11), collect-side cap ownership (§3/§6). + +## 0. Decision record + +Rulings already made (Oracle 2026-08-21, user same day) — do not relitigate +during implementation: + +- **R1 (registry)**: One unified `Supervisor` whose `handles` map holds + `TaskHandle = enum { Agent(AgentHandle), Job(JobHandle) }`. REJECTED: a + kind-flag field on a generalized struct (Option-soup; a Job with a + `child_supervisor` becomes representable) and a parallel `JobRegistry` + (guardrail, `cancel_recursive`, and all 5 turn-end sites already traverse + `ctx.supervisor`; a second registry duplicates traversal and creates two + cleanup paths that drift). +- **R2 (invocation shape)**: Meta-tool `job__start { tool, arguments }`, + mirroring the `mcp_invoke_*` wrapper shape the model already sees in its own + transcripts. REJECTED: injecting a `background: true` param into every tool + schema — coyote doesn't own MCP passthrough or argc-generated schemas, the + param must be stripped before dispatch, and strict parsers + (`additionalProperties: false` MCP servers, argc bins) hard-fail on leaks. +- **R3 (push mechanism)**: Notifications are delivered by (a) merging a + `system_notifications` key into the LAST real ToolResult of a batch (the + proven `pending_escalations` channel) and (b) the turn-end guardrail user + message. REJECTED: a mid-turn synthetic `[SYSTEM NOTIFICATION]` user + message — attribution confusion (model reads it as user intent and breaks + off its tool chain), no clean seam (`merge_tool_results` at 5 sites vs one + function in `eval_tool_calls`), fake "user said" artifacts persisting in + saved sessions, and ZERO coverage gain: the only moment with no ToolResult + to merge into is an empty batch, which IS turn-end, which the guardrail + owns. This is the same transcript-integrity principle as the + `__escalation_notification` phantom-call fix (see memory + `coyote-escalation-notification-bug`): never fabricate transcript content + the model didn't produce. +- **R4 (agents too)**: Completion notifications are ALWAYS-ON for both jobs + and spawned agents. REJECTED: per-spawn opt-in (asymmetry doubles prompt + guidance; the model polls anyway; volume is one terse event per + completion). A `notify: false` opt-out may be added later if fanout spam + materializes — not in v1. NOTE: this is one of the four deliberate + behavior deltas that apply even when jobs are disabled (§9.1). +- **R5 (persistence)**: Jobs die with the coyote process. No pid files, no + output spooling, no reattach-after-restart in v1. Half-building persistence + is worse than not having it. +- **R6 (check semantics — user-ruled 2026-08-21)**: `job__check` AND + `agent__check` are pure status probes; they NEVER consume the handle. + `collect` is the single retrieval+reclaim verb. REJECTED: today's + agent__check finished→collect delegation (supervisor.rs:844) — it returns + an unbounded payload the model didn't opt into (job results can be huge), + makes check's return shape bimodal, breaks the documented check-then-collect + pattern (follow-up collect errors on the consumed id), and silently defeats + the H7 guardrail backstop (once consumed, an overlooked result is + unrecoverable). With push notifications the round-trip saving is marginal: + completion usually arrives as a notification naming the collect command, + not via polling. Finished JOB checks return status + output_tail preview + + the exact collect command; finished AGENT checks return status + the exact + collect command only (agents have no ring buffer — nothing to preview). + Deliberate delta, see §9.1. +- **R7 (concurrency knob — user-ruled 2026-08-21, AMENDED same day)**: + `max_concurrent_jobs` is BOTH a global `Config` field (concrete, default + **5**) AND an `Option` override on `AgentConfig`, resolved + agent-override-first exactly like `max_tool_result_chars` + (src/function/mod.rs:332-337 pattern: + `agent.field().or(global)`). Rationale for the amendment: plain (non-agent) + sessions have NO `AgentConfig` at all (§2), and jobs — unlike agent + spawning — ARE available in plain sessions (R9), so the global field is + the only knob there; a serde default alone is not user-tweakable. + Role/session-runtime overrides remain DEFERRED (parity with + `max_concurrent_agents`). Implementation gotcha: `AppConfig` duplicates + `Config` fields across FOUR touch points (struct ~app_config.rs:66, + Default ~:151, From ~:237, env override ~:573) — miss one and the + field silently stays default. +- **R8 (feature-off gating — user-ruled 2026-08-21; amended 2026-08-24)**: + the job feature is OFF for a context when EITHER (a) the effective + `max_concurrent_jobs` is 0, OR (b) `function_calling_support` is false + (config mod.rs:232 / AppConfig app_config.rs:36, env-overridable + app_config.rs:523). When OFF, the `job__*` tool declarations are NOT + injected, the job prompt instructions are NOT injected, and no job + capacity exists — mirroring exactly how `can_spawn_agents: false` omits + the agent__* family today (declarations gated at Agent::init + agent.rs:224-226; prompt text at agent.rs:439-441; supervisor creation at + rc.rs:4139-4143). A model that has never seen a job__ declaration cannot + call one; `max_concurrent_jobs: 0` = the feature does not exist for that + context. The function-calling gate mirrors how skill/memory/rag function + injection already checks `app.function_calling_support` at Agent::init + (agent.rs:231, 238, 252) — NOT how agent__* does it (agent__* injects + unconditionally and only refuses at runtime, supervisor.rs:526/710; + job__* must gate at INJECTION so neither declarations nor instructions + ever reach the prompt). The model-level `supports_function_calling()` + strip at input.rs:260 remains the generic per-request safety net but is + NOT the gate — it strips declarations only, never injected instructions. +- **R9 (plain sessions)**: jobs ARE available outside agent contexts + (plain REPL/role sessions), governed by the global config value alone. + This requires creating the Supervisor outside `use_agent`: init condition + becomes `can_spawn_agents || jobs_enabled` — where `jobs_enabled` is + `function_calling_support && effective_max_concurrent_jobs > 0` (R8) — with + agent capacity 0 when `can_spawn_agents` is false (Supervisor::register + already rejects at capacity — tested behavior). `exit_agent`'s Functions + rebuild (rc.rs:4177-4186) must retain job declarations when jobs are + enabled. Creation sites (gate-ruled 2026-08-21): EAGER in `use_agent` + with the amended condition (existing site rc.rs:4139-4143); LAZY in plain + sessions — `job__start` get-or-inits `ctx.supervisor` (agent capacity 0) + on first use (job handlers run on the sequential `&mut ctx` path, so they + can set it); `exit_agent`'s unconditional `self.supervisor = None` + (rc.rs:4192) STAYS as-is — `cancel_recursive` already ran (rc.rs:4190) + and the next `job__start` lazily recreates. A plain session that never + starts a job keeps `supervisor: None` — bit-identical to today (§9.2.3). +- **R10 (REPL surfaces — user-ruled 2026-08-24)**: `job__*` are BUILT-IN + functions — always on when `jobs_enabled`, never individually + toggleable: + - `.info tools` MUST list them when enabled. This is FREE: `tools_info` + (rc.rs:700) renders the `select_functions` output, so it falls out of + §3 injection + carve-outs — no dedicated code. + - `.list tools`, `.tool enable/disable` validation, AND the `.tool` + tab completions MUST all exclude them. ONE change covers all three + surfaces: add `job__` to the built-in prefix exclusion list in + `concrete_tool_names()` (rc.rs:1283-1308; today user__/mcp_/todo__/ + agent__/memory__/skill__/rag__) — `.list tools` (rc.rs:2732), + `toggle_tool` validation (rc.rs:1356), and `repl_complete`'s `.tool` + arm (rc.rs:3458) all draw from that one pool. This also keeps `job__` + out of `toggle_tool`'s disable-path pool materialization, so job + tools can never leak into a persisted `enabled_tools` list. + - Consequently `.tool enable job__start` errors with the existing + "Unknown tool" teaching error — identical to agent__/todo__ today. +- **R11 (subagents & tool-filtered contexts — user-ruled 2026-08-24)**: + - An explicit `enabled_tools` list (role, session, agent, or graph LLM + node `tools:` — node lists arrive as role-level enabled_tools, see the + comment at rc.rs:2043-2046) can neither GRANT nor REVOKE `job__*`; + presence is governed solely by `jobs_enabled` (R8). Mechanism: + `JOB_FUNCTION_PREFIX` joins BOTH infra carve-outs in + `select_enabled_functions` (the §3 bullet: agent-path retain + ~rc.rs:2139-2150, non-agent builtin re-add ~rc.rs:2109-2128); extend + `select_functions_preserves_infra_tools_under_agent_filter` + (rc.rs:5642). + - A context WITH a tool list CAN use `job__*` (user-confirmed) — but + `job__start` additionally validates that the requested tool is + AVAILABLE in the calling context (§3), so a narrowed node can only + background tools it could call in the foreground; `job__start` must + not be an `enabled_tools` bypass. + - A context with NO enabled tools still SEES `job__*` — consistent with + how agent__/todo__/user__ infra prefixes survive an empty filter + today. Harmless: every `job__start` there rejects with the + context-availability teaching error, since nothing is startable. + NAMED CONSEQUENCE (Oracle N2, accepted): this flips previously + TOOLLESS contexts from `tools: None` to `Some([five job__* decls])` + request shape — `select_functions` returns None only when the merged + set is empty (rc.rs:2277-2287; pinned by + `select_functions_returns_none_when_no_tools_enabled`, rc.rs:5493, + which needs a jobs-off guard). E.g. a pure-text graph node with + `tools: []` now sends a tools array. T7 pins both states. + - Subagents: NO special machinery. Each spawned agent runs Agent::init + → R8 gating with its own effective `max_concurrent_jobs` (agent + override → global); its jobs register in its OWN supervisor; parent + `cancel_recursive` already cascades through child supervisors (T1). + +Explicit non-goals: job persistence across restarts (R5); backgrounding +internal state-mutating tools (§4 whitelist); a general mid-turn user-message +injection mechanism (R3); job-to-job messaging (jobs have no inbox — they are +processes/futures, not conversants). + +## 1. Summary + +Today every tool call blocks the turn: `eval_tool_calls` (src/function/ +mod.rs:262) is awaited inline from `call_chat_completions` (src/client/ +common.rs:526), so a 10-minute build or a slow MCP call pins the model until +it finishes. Coyote already solved this exact problem for *agents* +(`agent__spawn`/`check`/`collect`/`cancel` on tokio tasks) — this design +generalizes that machinery to *tool invocations*: + +1. A **`job__*` tool family** — `job__start`, `job__check`, `job__collect`, + `job__cancel`, `job__list` — that runs a whitelisted tool call as a + background tokio task registered in the existing `Supervisor`. +2. A **per-context `NotificationQueue`** so job and agent completions are + *pushed* to the model (merged into the next batch's last ToolResult, and + enumerated at turn-end by the guardrail) instead of relying on the model + remembering to poll. +3. A **guardrail predicate fix** so finished-but-uncollected handles block + turn-end with exact collect commands — which also fixes a latent bug where + a finished-but-uncollected *agent's* output is silently abandoned today. + +## 2. What exists today (verified) + +Turn loop & dispatch: +- `ask()` (src/repl/mod.rs:1416) recurses: `call_chat_completions[_streaming]` + → `eval_tool_calls` → non-empty `tool_results` re-enter `ask()` via + `input.merge_tool_results` (repl/mod.rs:1469-1476). Empty results = turn + end. Same loop duplicated in src/main.rs:581, src/acp/server.rs:214, + src/graph/llm.rs:258-302, and the child loop `run_child_agent` + (src/function/supervisor.rs:435-504). +- `eval_tool_calls` (mod.rs:262) partitions a batch on `is_mcp_meta_function` + (mod.rs:291-293): calls carrying any of the FIVE MCP meta-prefixes + (`mcp_invoke_/mcp_search_/mcp_describe_/mcp_read_/mcp_prompt_` — the + catalog grew read/prompt with the 2026-08-25 MCP resources/prompts merge, + src/mcp/mod.rs:36-48) run CONCURRENTLY via + `future::join_all` with `&RequestContext` (`eval_mcp` takes `&ctx`, + mod.rs:296-300); everything else runs SEQUENTIALLY because `ToolCall::eval` + (mod.rs:1420) takes `&mut RequestContext`. Results re-sorted by original + index; per-call errors soft-fail as `{"tool_call_error": ...}`; nulls + normalize to `"DONE"` (mod.rs:358-364) because empty results end the turn. +- Post-processing order matters: dedup/loop check (mod.rs:271-284) → + `max_tool_result_chars` truncation (mod.rs:332-337) → escalation injection + into the last result at depth 0 (mod.rs:343-347). + +Supervisor & agents: +- `agent__spawn` (`handle_spawn`, supervisor.rs:649) builds a child ctx + (`RequestContext::new_for_child`, request_context.rs:465 — note: + `supervisor: None` fresh per child, but `escalation_queue` CLONE-INHERITED + from parent at rc.rs:497) and `tokio::spawn`s `run_child_agent` + (supervisor.rs:778-795). The result lives in the + `JoinHandle>` inside `AgentHandle { id, agent_name, + depth, inbox, abort_signal, join_handle, child_supervisor }` + (src/supervisor/mod.rs:30-38), registered in `Supervisor { handles: + HashMap }` (mod.rs:40-45). +- `agent__check` polls `is_finished()` (supervisor.rs:827); on a finished + agent it silently DELEGATES to `handle_collect` (supervisor.rs:844), i.e. + check can consume the handle. `agent__collect` = 200ms poll loop that + breaks out early with `status: "pending"` if escalations are pending + (supervisor.rs:894-937), then `Supervisor::take` + await, optional LLM + summarization above a threshold (supervisor.rs:1401-1451). `agent__cancel` + = take + `cancel_recursive()` on child supervisor + abort signal + ≤5s + join wait (supervisor.rs:1044-1067). +- Turn-end guardrail `check_pending_agents_guardrail` (supervisor.rs:71-99): + called at all 5 turn-end sites; injects a `[SYSTEM GUARDRAIL]` user message + via `Input::from_str` and recurses; after `PENDING_AGENTS_GUARDRAIL_MAX = 3` + reminders, `ForceTerminate` + `cancel_recursive()`. CRITICAL: + `pending_agent_ids` (supervisor.rs:41-53) filters + `is_finished == Some(false)` — finished-but-uncollected handles pass the + guardrail silently and their results are dropped. +- `run_child_agent` ends with `supervisor.read().cancel_recursive()` + (supervisor.rs:498-500) — child-owned handles get cleanup for free IFF job + cancellation is wired into `cancel_recursive`. +- Context-reset sites (Oracle-corrected 2026-08-21): there is NO + full-context `/clear` command — the REPL has only `.clear todo` + (repl/mod.rs:1320-1335). The real supervisor-reset sites are `use_agent` + (rc.rs:4155 — REPLACES `self.supervisor` WITHOUT calling + `cancel_recursive` today) and `exit_agent` (rc.rs:4190 — calls it). Also + relevant: `process::exit` bypasses destructors at main.rs:672 (shell + execute), main.rs:273, logs.rs:18, config/mod.rs:783. + +Tool-injection gating & config layering (verified 2026-08-21): +- agent__* declarations exist ONLY inside an agent's `Functions`: appended by + `Functions::append_supervisor_functions()` (mod.rs:616-621), called at + `Agent::init` iff `can_spawn_agents` (agent.rs:224-226). `spawnable_agents` + and depth limits do NOT gate injection — enforced at spawn time + (supervisor.rs:663-670, Supervisor depth check). +- Per-request tool list: `RequestContext::select_enabled_functions` + (rc.rs:2039-2172); agent path copies agent declarations (:2098-2109) with + a `SUPERVISOR_FUNCTION_PREFIX` carve-out surviving role `enabled_tools` + filters (:2111-2121); non-agent path has a builtin-prefix filter + (:2077-2096). + A job__ prefix needs the same carve-outs. +- Prompt gating: `interpolated_instructions()` pushes + `DEFAULT_SPAWN_INSTRUCTIONS` (prompts.rs:74) iff + `can_spawn_agents && inject_spawn_instructions` (agent.rs:439-441). +- Supervisor exists ONLY when `use_agent` runs with `can_spawn_agents` + (rc.rs:4139-4143); plain REPL/role sessions have `supervisor: None`, no + agent__* declarations, and CANNOT spawn agents. `exit_agent` + (rc.rs:4175-4192) rebuilds `Functions::init` + user-interaction functions + only (:4113-4116). +- Global-vs-agent fallback precedent: `Option` on AgentConfig + concrete + global Config field + `agent.field().or(global)` at the resolution site + (max_tool_result_chars: mod.rs:332-337; compression_threshold: + session.rs:569-575). `max_concurrent_agents` does NOT follow it (agent-only, + serde default 4) — R7 makes the jobs knob follow the Option-fallback + pattern instead. + +Safe injection channels (the only two): +- Key-merge into the last real ToolResult: `inject_escalation_notification` + (mod.rs:366-381) adds `pending_escalations` + `escalation_instruction` keys + to the output object, wrapping non-objects as `{"output": old, ...}`. Runs + AFTER truncation. Gated `ctx.current_depth == 0` on the SHARED root + escalation queue. +- Synthetic user message at turn-end only (guardrail, auto-continue at + repl/mod.rs:1477-1524, `.recover`). + +Loop detection: root `ToolCallTracker::default()` = `new(2,3)` (mod.rs: +2336-2353) — TWO identical consecutive calls trip it. Unmitigated, a model +calling `job__check {id}` twice in a row is flagged as a loop. + +External argc tools: results are returned via the `LLM_OUTPUT` temp file, not +stdout (run_llm_function protocol; see memory +`coyote-execute-command-swallowed-output` for the hardened error semantics). +Stdout/stderr are human-facing side output. + +Test coverage today (inventoried 2026-08-21; all inline `#[cfg(test)]`, no +tests/ dir): ~153 tests across the area. FULL coverage: escalation.rs (11), +mailbox.rs (11), taskqueue.rs (17), Supervisor register/take/capacity/depth +unit tests (11), single-call eval_tool_calls behaviors (soft-fail, DONE +normalization, escalation injection incl. non-object wrap), tracker + dedup +unit tests, declaration/registry tests (function/mod.rs ≈55, +function/supervisor.rs 48). ZERO coverage (ranked by regression risk — the +basis for T0 in §11): handle_collect (poll loop, escalation early-out, +take+await, summarize threshold); guardrail ForceTerminate arm + counter +reset; multi-call eval_tool_calls (MCP/sequential partition, index re-sort, +loop-alert-in-batch); handle_spawn (capacity/depth/allow-list through the +handler); run_child_agent; turn-loop call sites + merge_tool_results; +max_tool_result_chars truncation; handle_check finished→collect delegation; +handle_cancel of a RUNNING agent / cancel_recursive; ToolCall::eval prefix +routing. + +## 3. The `job__*` tool family + +New prefix `JOB_FUNCTION_PREFIX = "job__"` routed in `ToolCall::eval`'s +prefix chain (mod.rs:1420) before the external-command fallthrough, handled +in src/function/jobs.rs (new), mirroring src/function/supervisor.rs. + +### Availability gating (R8, R9) + +Effective value: `agent.max_concurrent_jobs().or(global).unwrap_or(5)`. +`jobs_enabled = function_calling_support && effective_max_concurrent_jobs > 0` +(R8); when `jobs_enabled` is false the feature is OFF for that context: +- No `job__*` declarations appended (`append_job_functions()` called + conditionally at Agent::init, mirroring agent.rs:224-226, AND at the + non-agent `Functions::init` sites — rc.rs:3836, rc.rs:4177, + app_state.rs:71 — so plain sessions get jobs per R9). Every one of these + call sites checks the full `jobs_enabled` predicate, including + `function_calling_support` (pattern: agent.rs:231/238/252, where + skill/memory/rag functions already gate on it). +- No job prompt instructions in `interpolated_instructions()` (mirror + agent.rs:439-441, but on the full `jobs_enabled` predicate — with + function calling off, NO job instructions appear anywhere, since a model + that cannot emit tool calls must not be taught tool syntax). Plain + sessions get NO injected job prompt text at + all — guidance there comes solely from the job__* tool descriptions + (no plain-session analog of interpolated_instructions exists; this is + the simplest R8-consistent ruling). +- Prefix carve-outs added at rc.rs:2139-2150 (agent path role filter) and + rc.rs:2109-2128 (non-agent builtin filter) apply only when enabled. + These carve-outs are what make `.info tools` show job__* and what makes + job__* survive role/session/agent/graph-node `enabled_tools` filters + (R11); extend `select_functions_preserves_infra_tools_under_agent_filter` + (rc.rs:5642) accordingly. +- REPL surfaces (R10): add `job__` to the built-in prefix exclusion list + in `concrete_tool_names()` (rc.rs:1283-1308) — that ONE pool feeds + `.list tools` (rc.rs:2732), `.tool enable/disable` validation + (toggle_tool rc.rs:1356, incl. its disable-path pool materialization), + and the `.tool` tab completions (repl_complete rc.rs:3458). `.info + tools` (tools_info rc.rs:700) needs NO dedicated code — it renders + `select_functions` output. +- Supervisor init condition becomes + `can_spawn_agents || jobs_enabled`; when only jobs + are enabled, agent capacity is 0 (register-at-capacity rejection is + existing tested behavior). + +### `job__start { tool: string, arguments: object }` +- Description mirrors `mcp_invoke_*`: "`arguments` is the same object the + tool takes when called directly." `arguments` schema is free-form object. +- Validates the tool against the backgroundable whitelist (§4). Rejection is + a TEACHING error: names the tool, why it can't background (mutates agent + state / interactive), and the whitelist categories — the model WILL try + `job__start {tool: "memory__write"}`. +- Validates the tool is AVAILABLE in the calling context (R11): the + requested tool must appear in the PER-REQUEST DECLARED-NAMES STASH — + the set of function names actually sent to the model for the current + request, captured in `before_chat_completion` (see Validation hardening + rule 1 for the mechanism and why recomputing via `extract_role` is + WRONG). This inherently applies role/session/agent/LLM-node + `enabled_tools` filters and enabled-MCP-server filters, because the + stash IS the filtered output. Whitelisted-but-filtered-out → + TEACHING error: "'X' is not enabled in this context". Without this + check, `job__start` would be an `enabled_tools` bypass for narrowed + contexts (e.g. graph LLM nodes with a `tools:` list). +- Runs any pre-execution gate SYNCHRONOUSLY before returning — a detached + task cannot touch the tty (H3). AUDITED (§13.2): NO gates exist on the + external-tool path today (no approval prompts/denylists; the only + pre-spawn checks are dedup/loop/unknown-tool inside eval_tool_calls), so + this rule is future-proofing, not migration work. +- Returns immediately: `{ status: "ok", job_id: "job_", tool, message: + "Running in background. Check with job__check, block with job__collect, + cancel with job__cancel. You will receive a system_notifications entry on + completion. Jobs do not survive coyote exiting." }` (The + system_notifications sentence is transiently aspirational on a T3-only + tree — notifications land in T4; both merge in the same PR, accepted + window.) +- Capacity: separate per-context `max_concurrent_jobs` budget (R7: agent + override → global config → default 5) — jobs must not consume the agent + `max_concurrent` budget. Jobs skip the agent depth check (no depth + semantics). + +### `job__check { id }` +- Returns `{ status: running|completed|failed, id, tool, elapsed_secs, + output_tail, output_bytes_captured, tail_truncated }`. `status` is read + from the JobHandle's shared JobState cell (§6) — the JoinHandle result is + never peeked (it can't be without consuming it). `cancelled` is + deliberately absent from the enum: job__cancel removes the handle from + the registry, so a later check on that id hits the unknown-id error + below. +- Unknown id → teaching error: "No job 'X' is registered — it may have + already been collected or cancelled. job__list shows active jobs." +- `output_tail` = tail of the live ring buffer (§5) — progress telemetry, + NOT the result. Tail extraction uses `String::from_utf8_lossy` (the ring + can split a multibyte char at the wrap point). +- NEVER consumes the handle (R6). `agent__check` is ALIGNED to the same + pure-probe semantics in T2 (today it delegates finished→collect at + supervisor.rs:844); a finished job check returns status + tail preview + + the exact collect command; a finished agent check returns status + the + exact collect command (no preview — no ring buffer). +- Per-handle consecutive-check counter: after ~5 checks with no state + change — defined as the `(status, output_bytes_captured)` tuple + unchanged — append hint: "still running; call job__collect to block, or + do other work — a system notification will fire on completion" (H8's + semantic rate-limit, replacing tracker special-casing). + +### `job__collect { id, tail_lines?: number }` +- Mirrors `agent__collect`: poll loop with the same early-out that surfaces + `pending_escalations` instead of deadlocking (supervisor.rs:904-914 + pattern), then `Supervisor::take` + await. +- Result assembly per tool class (§5): argc/external → read `LLM_OUTPUT` + with the hardened missing/empty semantics, plus final `output_tail` and + exit status; MCP → the future's return value. Ownership split (gate-ruled): + the JOB TASK reads LLM_OUTPUT after `wait()` (protocol semantics §4) and + returns the RAW output in `JobResult`; the COLLECT HANDLER applies the + default tail cap / `tail_lines` and reports `result_truncated` in its own + response. +- `JoinError` (panic) maps to `status: "failed"` with the panic message — + never hangs — and still returns whatever ring-buffer content exists (H6). +- Output capping (RESOLVED §13.6): `max_tool_result_chars` CANNOT be the + backstop — its global default is null/no-cap (config mod.rs:266, + config.example.yaml:206) and `truncate_if_needed` keeps the HEAD + (mod.rs:404-413), the wrong end for build logs (failures land at the + tail). Job results get their own TAIL-biased cap: default keep the LAST + 50,000 chars with a `[truncated: kept last N of M chars]` header, plus + the optional `tail_lines` param for explicit control (`tail_lines` + applies to the RESULT text — LLM_OUTPUT contents / MCP return — never to + the ring buffer). Both the default cap and `tail_lines` cuts floor to a + char boundary (do not reintroduce the §9.1 UTF-8 bug on the tail side). + No LLM summarization in v1 (supervisor.rs:1401-1451 ships the entire + output to the summarizer as one message — a huge log would blow its + context window anyway). + +### `job__cancel { id }` +- Take handle → kill process group per §6 platform strategy (only if the + JobState pgid is still set) → abort JoinHandle → ≤5s join wait → + `{ status: "cancelled", id, output_tail }` (partial output included). + +### `job__list` +- All registered jobs for THIS agent's supervisor: id, tool, status (from + the JobState cell), elapsed, bytes captured. + +Cross-kind misuse errors teach: `agent__collect("job_x")` → "'job_x' is a +background job, not an agent — use job__collect"; `job__collect("agent_...")` +→ inverse. IDs are namespaced (`job_` prefix) to make this cheap. + +### Validation hardening (loophole audit, 2026-08-24) + +WHY this matters more than it looks: in the FOREGROUND, tool access is +enforced by request declaration — providers only let the model call tools +sent in the request payload, so `ToolCall::eval` merely checks POOL +membership (`extract_call_config_from_ctx` bails "Unexpected call" for +undeclared names, mod.rs:1814-1825; test mod.rs:2564) and then resolves +the name against bin dirs + PATH (`run_llm_function`). `job__start`'s +`tool` parameter is a FREE STRING — it bypasses the provider-level +constraint entirely, so the R11 declared-names check is THE enforcement +point (and is strictly tighter than foreground's pool check: enabled set, +not pool). Binding rules: + +1. **Exact-match only, validated name drives everything.** The R11 check + is exact string equality (post-trim) against the set of tool names + ACTUALLY DECLARED to the model for the CURRENT request. It must NOT + recompute `select_functions(&extract_role())` at eval time: + `extract_role` layers session → agent → role (rc.rs:996-1019), and + during a graph LLM-node run it returns the UN-narrowed + `agent.to_role()` while the request was built from the node's + swapped-in role (`ctx.role` — graph/llm.rs:174-188, 251-252 → + input.rs:433 `functions: ctx.select_functions(role)`), so recomputing + would validate against the FULL pool and reopen the exact bypass R11 + closes (Oracle B1). Mechanism (Oracle-ruled 2026-08-24): + `before_chat_completion` (rc.rs:896) — already called at ALL SEVEN + turn-loop sites (repl/mod.rs:1435/1444, main.rs:562/573/632, + acp/server.rs:204, supervisor.rs:456 child loop, graph/llm.rs:257) — + stashes the request's declared-function NAME SET (a cheap + `HashSet` from the Input's select_functions output, NOT a + Role clone) into ctx; `job__start` validates against the stash. + REFRESHED on every request — a stale stash from a prior turn must be + impossible. Desirable side effect: a mid-batch `skill__load` widening + the pool does NOT grant `job__start` access until the next request — + the check is literally "declared to the model this turn". + `mapping_tools` aliases are NOT expanded — concrete + declaration names only. The raw model string is NEVER passed to + `run_llm_function`/bin-dir/PATH resolution or to the agent-tool + mapping (mod.rs:1785-1812): class dispatch (external vs `mcp_invoke_*`) + and cmd resolution derive from the MATCHED declaration, not from the + model's string. (Otherwise `job__start {tool: "bash"}` or a + path-shaped name would execute an arbitrary PATH binary that was + never a declared tool.) +2. **Both gates always run**: §4 whitelist AND R11 context availability; + a name must pass both regardless of order (each has its own teaching + error). +3. **Disabled state never falls through.** VERIFIED eval order (Oracle + B2): `extract_call_config_from_ctx/from_agent` runs BEFORE the prefix + chain (mod.rs:1420-1428) and bails "Unexpected call" for any name not + in the declaration pool (mod.rs:1814-1825) — so with jobs disabled + (job__* never declared, R8) a hallucinated `job__start` dies at that + bail as a per-call soft-fail and can never reach the external-command + fallthrough. RULING (B2 option (b)): this existing "Unexpected call" + soft-fail IS the specified disabled-state behavior — exactly what + agent__* produces in non-agent contexts today. Option (a) + (pre-extract routing on the raw name to emit a nicer teaching error) + REJECTED: reordering eval semantics for an error-message nicety, zero + security gain; the mod.rs:2564 test must stay green as-is. The + `job__` prefix arm in the chain consequently only ever executes when + declarations exist (jobs enabled). +4. **MCP server derivation from the validated name.** `JobCtx`'s + source, the full `McpRuntime` map, contains every STARTED server + (src/config/tool_scope.rs:44-47), not just context-enabled ones. The + server is derived solely from the validated `mcp_invoke_` + declaration name (the invoke_mcp_tool pattern, mod.rs:1614-1636, + prefix-strip at :1619) — and to + make the discipline STRUCTURAL (Oracle N3): `JobCtx` is constructed + at `job__start` AFTER validation, so it snapshots a SINGLE-ENTRY + `McpRuntime` holding only the validated server's + `Arc` — the job task cannot reach any other server + even by bug. The + inner `tool` argument stays foreground-parity (opaque server-side + name — no new gate, none exists today). +5. **Handle isolation across contexts.** All five `job__*` handlers + operate EXCLUSIVELY on `ctx.supervisor` — never `parent_supervisor` + (which exists in child contexts for escalations) — so a subagent + cannot check/collect/cancel its parent's jobs or vice versa. Forged + or stale ids hit the cross-kind/unknown-id teaching errors. +6. **Residual, accepted + documented**: job OUTPUT is untrusted text — + the same prompt-injection surface as foreground tool output. It cannot + mint declarations, grant tools, or forge handles; worst case is + persuading the model to call tools it already has. No v1 mitigation + beyond the existing one (output is data, delivered inside a + ToolResult). + +## 4. Backgroundable-tool whitelist + +Forced by the architecture: `ToolCall::eval` takes `&mut RequestContext`, +which cannot move into a detached task (H2). Backgroundable = tools whose +execution can run from an owned snapshot: + +| Class | Backgroundable | Execution path in job task | +|---|---|---| +| `execute_command` + ALL external command tools — bash (argc), JavaScript/TypeScript, Python (user-ruled 2026-08-21; no per-tool opt-in flag) | YES (the whole point) | Extract the spawn logic out of the eval path into a free function taking `(Arc, JobEnvSnapshot, args)` — do NOT route through `ToolCall::eval`. `JobEnvSnapshot` = the audited field set below. | +| `mcp_invoke_*` | YES | Already `&ctx` (`eval_mcp`); job task owns `JobCtx { mcp_runtime: McpRuntime, current_depth: usize }` — the runtime is a SINGLE-ENTRY snapshot holding only the validated server's `Arc` (§3 hardening rule 4, Oracle N3). Job MCP path = `invoke` → `render_tool_result` (mod.rs:1635 foreground parity; FREE function, see §13.1). AUDITED (§13.1): that is eval_mcp's COMPLETE transitive ctx surface. McpRuntime is `#[derive(Clone)]`, shallow Arc map (src/config/tool_scope.rs:44-47); OAuth refresh is transport-embedded (auth_client.rs) and needs no AppState. NOTE: MCP jobs have NO timeout (`COYOTE_TOOL_TIMEOUT` is process-path only); a hung MCP job is recoverable via `job__cancel` (abort drops the future) — accepted v1, stated in the tool description. | +| `mcp_search_/mcp_describe_/mcp_read_/mcp_prompt_` | NO (pointless — fast) | Teaching error: "sub-second call; invoke directly." (All four non-invoke meta-families; meta-declarations are capability-gated per server — `gated_meta_function_prefixes`, function/mod.rs:416-426: invoke⇔tools, read⇔resources, prompt⇔prompts — which composes with the declared-names stash automatically: a capability the server never advertised is never declared, so `job__start` rejects it with the standard not-declared error.) | +| `agent__*`, `job__*` | NO | "Already asynchronous — use them directly." | +| `todo__*`, `memory__*`, `skill__*`, `user__*` and other internal `&mut ctx` tools | NO | "Mutates agent/session state; must run in-turn." | +| fs_* / ast_grep / grep-class builtins | NO in v1 | Fast; not worth the snapshot surface. Revisit only with evidence. | + +Snapshot semantics (document in tool description): config/model/env changes +made after `job__start` do not affect a running job. The job must not mutate +shared session state — it only produces output. + +Directionality ruling (user, 2026-08-24): the async systems compose ONE WAY +ONLY — agents (and graph LLM nodes) may start jobs, but a job can NEVER +start an agent, another job, or invoke any built-in — anything else would +duplicate/defeat the agents' own parallelization system. This is enforced +twice: (1) the whitelist rows above (policy, with teaching errors); (2) +architecturally — every built-in handler requires `&mut RequestContext`, +which cannot move into a detached task (H2); a job task owns only +`JobEnvSnapshot`/`JobCtx`, so there is no ctx, supervisor, or eval loop +inside a job for a built-in to run against, even if the whitelist check +were bypassed. + +### JobEnvSnapshot & runner mechanics (§13.2 audit, 2026-08-21) + +| Field | Source (foreground receipt) | +|---|---| +| `cmd_name`, `cmd_args` | tool name / agent-tool mapping + JSON args pushed as last arg (mod.rs:1785-1812, 1269) | +| `envs` | `agent.variable_envs()` — `LLM_AGENT_VAR_*` with vault-secret interpolation (agent.rs:469-485); resolved INTO the snapshot at job__start (same plaintext-in-memory exposure as foreground) | +| `agent_name` | drives bin dir + AUTO_CONFIRM-for-graph (mod.rs:2136-2149) | +| `PATH` | functions/agent bin dirs prepended (mod.rs:2136-2149); env-derived dirs (paths.rs:285-287, 327-329) FROZEN at job__start per snapshot semantics | +| `LLM_OUTPUT` | fresh temp path, file not pre-created (mod.rs:2158-2159, utils/mod.rs:313-320) | +| `CLICOLOR_FORCE`/`FORCE_COLOR` | hardcoded =1 (mod.rs:2178-2179) | +| `COYOTE_TOOL_TIMEOUT` | default 1800s, 0 = unlimited (mod.rs:2247-2251); resolved at job__start; jobs honor it (expiry → kill process GROUP + `failed` status) | +| cwd | inherited (foreground sets none) | +| Windows | `LLM_TOOL_DATA_FILE` arg spill + PATHEXT polyfill (mod.rs:2161-2176) | + +Runner-mechanics rulings from the audit: +- The external branch of `ToolCall::eval` never actually uses `&mut` — + `run_llm_function(cmd_name, cmd_args, envs, agent_name)` takes zero ctx + (mod.rs:1246-1252) — so the extraction is clean, not a refactor risk. +- The foreground runner is SYNC `std::process` polled at 100ms from async + without spawn_blocking (mod.rs:2181-2316). v1 leaves the foreground path + UNTOUCHED; the background runner uses `tokio::process` + + `process_group(0)`. (Foreground sync-in-async is a pre-existing latent + issue — note as follow-up, out of scope.) +- Foreground TEES stdout/stderr live to the terminal via threads + (mod.rs:2193-2245); background jobs are CAPTURE-ONLY into the ring + buffer — no live tee (it would interleave with the foreground UI). +- `stdin = Stdio::null()` is already foreground behavior (mod.rs:2184) — no + delta for jobs. +- LLM_OUTPUT protocol semantics to replicate when the job task reads the + file after `wait()` (mod.rs:2283-2316): + nonzero exit → `tool_call_error` + stderr/stdout + partial output; zero + exit + missing/empty file → null → "DONE"; unreadable existing file → + hard error; timeout → kill + `tool_call_error`. + +## 5. Output channels: ring buffer vs result (H1 — do not conflate) + +For process-backed jobs there are TWO distinct channels: + +1. **Live telemetry**: stdout+stderr streamed into a bounded ring buffer on + the `JobHandle` (`Arc>`, default 64 KiB). Carries a + monotonic total-bytes counter and a truncation marker so the model knows + the tail is clipped. This is what `job__check` returns and what makes + "did it produce any output yet?" answerable mid-run. +2. **The tool result**: for argc/external tools this is the `LLM_OUTPUT` + file, read ONLY once — by the job task after `wait()` (§3 ownership + split; the model sees it at collect) — with the same missing/empty-file + error semantics as the foreground path. The ring buffer is never substituted + for the result. For MCP jobs the result is simply the future's return + value; the ring buffer is unused/empty. + +Background processes get `stdin = /dev/null` — a script that prompts must +fail fast, not hang the job forever (H3 corollary; already true in the +foreground runner — `Stdio::null()` at mod.rs:2184 — so no behavior delta). + +## 6. JobHandle, JobState, process lifecycle, and cleanup (H4) + +```rust +// src/supervisor/mod.rs (alongside AgentHandle) +pub struct JobHandle { + pub id: String, // "job_" + pub tool: String, + pub started_at: Instant, + pub join_handle: JoinHandle>, + pub abort_signal: AbortSignal, + pub state: Arc>, // shared with the job task + pub output_buf: Arc>, // telemetry channel (§5) + pub no_change_checks: u32, // §3 job__check counter +} + +pub struct JobState { + pub status: JobStatus, // Running | Completed | Failed + pub pgid: Option, // Unix pgid; None on Windows & MCP jobs; + // CLEARED by the job task right after wait() + // reaps the child (pid-reuse guard, see below) +} + +pub enum JobStatus { Running, Completed, Failed } +// job__check/job__list read this cell. Cancelled is unrepresentable here — +// cancel removes the handle from the registry (§3). The job task writes the +// final status (and clears pgid) BEFORE it returns. + +pub struct JobResult { + pub output: Value, // RAW tool result (job task reads LLM_OUTPUT after wait(); MCP return) + pub exit_code: Option, // process jobs only + pub output_bytes_captured: u64, +} +// The COLLECT HANDLER applies the tail cap / tail_lines (§3) and reports +// `result_truncated` in its own response — deliberately NOT a JobResult +// field (capping is a presentation concern owned by collect). + +pub enum TaskHandle { Agent(AgentHandle), Job(JobHandle) } // R1 +``` + +- `Supervisor.handles` becomes `HashMap`; `register()` + branches on variant for capacity; existing agent paths pattern-match. +- Supervisor creation per R9: `can_spawn_agents || jobs_enabled` (R8), + agent capacity 0 in jobs-only contexts — EAGER in `use_agent`, LAZY via + `job__start` + get-or-init in plain sessions (see R9 for the full site ruling incl. the + exit_agent rc.rs:4192 rule). Never-jobbing plain sessions keep + `supervisor: None`. +- **Kill discipline (pid-reuse guard — Oracle finding, MANDATORY)**: every + killer (`impl Drop for JobHandle`, `job__cancel`, `cancel_recursive`, + timeout expiry) kills the process group ONLY IF `state.pgid` is still + `Some`, and the job task sets `pgid = None` immediately after `wait()` + returns (the child is reaped; with `process_group(0)`, pgid == child pid, + so a stale killpg after reap can SIGTERM an innocent recycled pid). + Without this guard, the normal collect path (take → await → handle drop) + fires a killpg against a reaped group on EVERY successful job. +- Spawn processes with `process_group(0)`; cancellation = `killpg(SIGTERM, + grace 5s, SIGKILL)` then `JoinHandle::abort()` — `abort()` alone does NOT + kill OS processes, and `kill_on_drop` misses grandchildren. +- Context-reset sites (corrected — there is NO full-context `/clear`): + `use_agent` (rc.rs:4155) currently REPLACES `self.supervisor` without + cleanup — T1 adds `cancel_recursive()` on the old supervisor before the + replacement (Arc-drop alone is unreliable: child ctxs hold + `parent_supervisor` clones, rc.rs:494, deferring the Drop while the job + runs on with no transcript). `exit_agent` already calls it (rc.rs:4190). + A job surviving a context switch has no transcript to report into. +- **Shutdown mechanism**: explicit kill-all (cancel_recursive over the root + supervisor) on the REPL quit path, where destructors run today. Hard-exit + paths that bypass destructors (`process::exit` at main.rs:672/257, + logs.rs:18, config/mod.rs:783) and panics MAY orphan a process group — + documented, accepted v1 (§9.4.11 checks the normal path). Note: + `process_group(0)` detaches jobs from coyote's group, so the shell will + not reap them either. +- `cancel_recursive`/`cancel_all` must handle the Job variant (kill group + per the discipline above, not just abort signal). +- R5: nothing survives process exit; `job__start`'s response and the system + prompt say so. +- **Lock discipline** (copy the existing pattern, supervisor.rs:894-942): + NEVER hold the parking_lot supervisor lock across an await point in any + job handler; take what you need under a scoped read/write and drop the + guard before awaiting. The ring-buffer mutex is locked only for the + memcpy (never across awaits in the stdout pump task). + +Module placement: `TaskHandle`/`JobHandle`/`JobState`/`JobStatus`/`JobResult` +live in src/supervisor/mod.rs alongside AgentHandle; the five job__* +handlers, `RingBuf`, `JobEnvSnapshot`, `JobCtx`, and the extracted runner +live in src/function/jobs.rs (new file, mirroring +src/function/supervisor.rs); `NotificationQueue` lives in +src/supervisor/notifications.rs (new file, sibling and structural template: +src/supervisor/escalation.rs). + +### Platform strategy (gate finding — killpg is Unix-only) + +- Unix: set the group with std's `CommandExt::process_group(0)` + (std::os::unix::process — no crate needed); kill with + `libc::killpg(pgid, SIGTERM)` → 5s grace → SIGKILL. NEW SANCTIONED + DEPENDENCY: `[target.'cfg(unix)'.dependencies] libc = "0.2"` — Cargo.toml + has no nix/libc direct dep today; this is the approved addition. +- Windows: jobs remain ENABLED; cancellation/timeout uses tokio's + `Child::start_kill()` + `kill_on_drop(true)` — single-PID, grandchildren + may leak, which is exactly the foreground timeout path's behavior today + (mod.rs:2258-2266): parity, not regression. All group-kill code is + `#[cfg(unix)]`-gated. Win32 Job objects are out of scope v1. +- MCP jobs have no OS process: cancellation = abort signal + + `JoinHandle::abort()` on both platforms. + +## 7. Push notifications: per-context `NotificationQueue` + +### Ownership model (critical — opposite of escalations) +`NotificationQueue` follows the `supervisor` pattern in `new_for_child` +(fresh per child, rc.rs:493), NOT the `escalation_queue` pattern +(clone-inherited, rc.rs:497). Each agent gets notifications for ITS OWN +spawned jobs/agents. A shared inherited queue = first-drainer-wins race +delivering the root's notifications into a child's transcript. +Consequently: escalations stay `depth == 0` on the shared root queue; +notifications drain the ctx's OWN queue at ANY depth. + +### Event shape (terse — one line per event) +```json +{ "event": "job_completed" | "job_failed" | "agent_completed" | "agent_failed", + "id": "job_a1b2", "tool_or_agent": "execute_command", "status": "success", + "next_action": "job__collect --id job_a1b2 for output" } +``` + +### Producers +- Job task: pushes on completion/failure before returning. +- Agent task: the `tokio::spawn` wrapper in `handle_spawn` + (supervisor.rs:778-795) pushes into the SPAWNING ctx's queue before + returning. Always-on (R4). +- NO event on explicit `job__cancel`/`agent__cancel` (the cancel's own + ToolResult confirms it). YES on failure/panic — with one caveat: a PANIC + in the job task skips the push (the producer never runs). This is + INTENTIONAL and covered: the turn-end guardrail's + finished-but-uncollected predicate still surfaces the handle, and + collect's JoinError→failed mapping (H6) reports the panic. Do NOT "fix" + this with a Drop-guard push — it reopens double-delivery. + +### Delivery point 1 — mid-turn key-merge (the "push") +Generalize `inject_escalation_notification` (mod.rs:366-381) into ONE +`merge_system_channel(last: &mut ToolResult, escalations, notifications)` +applied at the end of `eval_tool_calls`, keeping the existing +AFTER-truncation ordering. Single pass is MANDATORY: two independent mergers +each applying the non-object wrap double-nest the output +(`{"output": {"output": ...}}`) (H9). Key order: `pending_escalations` first +(children are BLOCKED; completions are not urgent), then +`system_notifications` + a one-line instruction. + +Drain-time stale suppression: filter events against current supervisor +registration — if the handle was already `take()`n (model collected before +the drain), DROP the event. Otherwise the model chases a dead id into an +error loop. + +### Delivery point 2 — turn-end guardrail (H7 predicate fix) +`pending_agent_ids` → `pending_task_ids` returning `(id, kind, finished)`, +INCLUDING finished-but-uncollected handles (today's `Some(false)` filter +excludes them — supervisor.rs:48). Guardrail prompt renders two sections: +- still running: existing reclaim language, kind-specific commands; +- completed — collect NOW: exact `job__collect`/`agent__collect` commands + (instant on a finished handle, so this is cheap for the model). +On the `ForceTerminate` strike (3 reminders), discard finished results with +a logged warning — no infinite loop. NOTE: this predicate change also fixes +the latent agent bug where finished-but-uncollected output is silently +abandoned at turn end. All 5 guardrail call sites get this for free. +Deliberate delta, see §9.1. + +### Delivery point 3 — none +No other injection point exists or is needed (R3). If the model has nothing +else to do, blocking `job__collect`/`agent__collect` remains the correct +primitive; notifications improve the working-meanwhile case only. + +### Race sweep (Oracle-reviewed 2026-08-21 — all benign, no further holes) +- Job completes between drain and merge → event delivers on the next batch + drain or the turn-end guardrail catches the finished handle. Covered. +- Model collects before drain → stale suppression drops the event. Covered. +- Guardrail-prompted collect, then next batch drains the old event → stale + suppression. Covered. +- Duplicate mention (event + guardrail, same handle) → benign redundancy. +- check-then-collect on the JobState cell → no consuming race under R6; + collect's take-under-write-lock is atomic. + +## 8. Loop-detection & polling ergonomics (H8) + +- Exempt `job__check`, `job__list`, `agent__check`, `agent__list_running` + from `tool_tracker.check_loop` AND from `record_call` — recording them + would let `[check, X, check, X]` mask a real X-loop; they must be invisible + to the tracker. (Root tracker `new(2,3)` trips on just 2 identical calls.) +- Unbounded-polling backstop is the per-handle no-change counter hint (§3), + a semantic limit where it belongs — not a tracker special case. + +## 9. Regression parity: guarantee when jobs are disabled (user requirement) + +Hard requirement (user, 2026-08-21): with effective `max_concurrent_jobs: 0`, +ALL existing function-calling and agent__* behavior works IDENTICALLY to +today — as if the feature does not exist. + +### 9.1 The ONLY intentional behavior deltas (jobs-independent) + +Four approved changes apply even when jobs are disabled. Everything else is +bit-identical. Each ships as its own commit with its own dedicated tests so +it can be reviewed/reverted in isolation: + +| Delta | Ruling | What changes | +|---|---|---| +| Agent completion push notifications | R4 | `system_notifications` key can appear on the last ToolResult of a batch after an agent finishes; guardrail prompt gains a "completed — collect now" section. Own commit within T4. | +| `agent__check` never consumes | R6 | Finished-agent check returns status + the exact collect command (no preview — agents have no ring buffer) instead of delegating to collect (supervisor.rs:844). T2 commit (b). | +| Guardrail counts finished-but-uncollected | H7 | Turn-end with an uncollected finished agent now Injects instead of silently dropping the result (fixes latent output-abandonment bug). T2 commit (a). | +| `truncate_if_needed` UTF-8 boundary fix | §13.6 audit | Edge-case BUG today: a cap landing mid-UTF-8-char makes `s.get(..max_chars)` return None → falls back to the FULL untruncated string while still prepending the truncation marker (mod.rs:404-413). Fix: floor the cut to a char boundary. Foreground-visible only in the broken edge case. T2 commit (c). | + +If any of these must ALSO be gated off, say so before task +materialization — they are separable. + +### 9.2 Zero-diff invariants when jobs are OFF (encode as tests) + +1. Tool list byte-identical: no `job__*` declarations in agent or plain + sessions (`select_enabled_functions` output compared with effective + `max_concurrent_jobs` 0 vs >0, AND with `function_calling_support: + false` at any `max_concurrent_jobs` value — both legs of `jobs_enabled` + independently force the OFF state). +2. Prompt byte-identical: no job instructions in + `interpolated_instructions()` (plain sessions never get job prompt text + at any setting — §3); also byte-identical with + `function_calling_support: false` regardless of `max_concurrent_jobs`. +3. Supervisor creation condition unchanged for agent-only contexts: + `can_spawn_agents: false` + jobs 0 → `supervisor: None`, exactly today. +4. `eval_tool_calls` behavior on any batch without job__ calls: identical + partition/order/soft-fail/truncation/injection (pinned by T0 tests). +5. `Supervisor` agent paths (register/capacity/depth/take/cancel_recursive) + behave identically with the `TaskHandle::Agent` variant — the enum + refactor is mechanical; T0 tests written BEFORE T1 must pass unmodified + after it (except type-name churn). +6. `merge_system_channel` with zero notifications + pending escalations + produces byte-identical output to today's + `inject_escalation_notification` (both object and non-object wrap cases). +7. Loop-tracker behavior unchanged for all non-exempt tools; exemption list + is exactly {job__check, job__list, agent__check, agent__list_running} + (agent-check exemption is part of delta R6's ergonomics; verify it + cannot mask real loops via the interleave test). +8. No `NotificationQueue` allocation side effects in sessions that never + spawn/background anything (drain of an empty queue = no-op, no key + added). + +### 9.3 T0 characterization tests (write BEFORE T1; the refactor safety net) + +Pin current behavior for every §2 coverage gap (ranked by risk): +1. `handle_collect`: finished-agent take+await happy path; escalation + early-out returns `status: "pending"` without consuming; summarization + under/over threshold. +2. Guardrail: `ForceTerminate` after 3 strikes + `cancel_recursive` called; + counter reset on collect/cancel/no-pending; Inject prompt content (both + with and without escalations — exists, keep). +3. `eval_tool_calls` multi-call: MCP/sequential partition, result re-sort by + original index, per-call soft-fail isolation (one failing call doesn't + poison the batch), loop-alert result shape inside a batch, bail on + empty-after-dedup. +4. `handle_spawn`: capacity rejection, depth rejection, spawnable_agents + allow-list rejection — through the HANDLER, not just Supervisor units. +5. `max_tool_result_chars` truncation through eval_tool_calls (agent + override + global fallback + 0-disables). +6. `merge_tool_results` message-shape test. +7. `handle_check`: pin CURRENT finished→collect delegation, marked as + deliberately rewritten by T2 commit (b) so the T2 diff is explicit. +8. `handle_cancel` of a RUNNING (not pre-finished) agent: abort + wait path; + direct `cancel_recursive` unit test. +9. `ToolCall::eval` prefix-routing table test (each prefix → expected + handler family, unknown → catalog-hint error). +10. `run_child_agent`: BEST-EFFORT — needs a mock client; if infeasible + without large scaffolding, document as manual case (§9.4) instead of + faking it. + +T0 merges before any refactor commit; T1+ must keep T0 green (allowing only +mechanical type renames), except tests explicitly marked for T2's deliberate +deltas. + +### 9.4 Manual verification checklist (user-runnable, post-implementation) + +Run once with `max_concurrent_jobs: 0` (global), once unset (default 5), +and once with `function_calling_support: false` (any `max_concurrent_jobs`) +— the last leg must show NO `job__*` declarations and NO job instructions +anywhere in the assembled prompt (R8): +1. Plain REPL: chat + `execute_command` + an fs_* call + an MCP call — works + as today; `job__*` absent from the tool list (0-case) / present + (default case). +2. Agent session (e.g. a spawning-capable agent): fan out 2 explores → + check → collect both; verify outputs and summarization. +3. Escalation round-trip: child asks a user__* question → parent sees + `pending_escalations` on last tool result → `agent__reply_escalation` + unblocks child. +4. Guardrail: force a turn-end with a running agent → `[SYSTEM GUARDRAIL]` + message; let it strike 3 times → force-cancel. +5. Task queue: `task_create` with deps + auto-dispatch agent on + `task_complete`. +6. Mailbox: `send_message` → child `check_inbox`. +7. Ctrl-c mid-stream (partial text kept, turn ends), then `.recover`. +8. Auto-continue with todo list pending. +9. Load a pre-existing saved session (incl. one containing the old phantom + `__escalation_notification` if available) — replays benignly. +10. `.agent` enter/exit — tool list correct on both sides of `exit_agent`; + with a job running, entering/exiting an agent KILLS the job (§6 + reset-site rule). +11. (jobs enabled) `job__start` a 30s `execute_command` → `job__check` + twice (no loop alert; tail visible) → do other tool work → observe + `system_notifications` on completion → `job__collect` → result correct; + then a cancel case; then quit coyote (normal REPL quit) mid-job → + verify no orphan process (`pgrep`). +12. REPL surfaces (R10) + filtered contexts (R11), jobs enabled: + `.info tools` lists the five `job__*` entries; `.list tools` omits + them; `.tool enable`/`.tool disable` tab completion never offers them; + `.tool enable job__start` → "Unknown tool". An agent/role with an + `enabled_tools` filter still sees `job__*`; `job__start` of a + filtered-out (but whitelisted) tool → context-availability error; + `job__start` of an in-filter tool works. + +## 10. Prompt & docs updates + +- Roles/system prompts (src/config/prompts.rs:118-119,180 region + agent + definitions in assets/): document the `job__*` family alongside `agent__*`; + update the wait-protocol guidance: "for long-running commands, `job__start` + and keep working; completion arrives as a `system_notifications` entry on + your next tool result; collect blocks only when you have nothing else to + do." Note jobs die with the process (R5) and snapshot semantics (§4). Job + instructions injected only when enabled (R8). Graph-node line (§12 + iteration-burn hazard): "in graph LLM nodes, collect or cancel your jobs + before ending your final node turn — an uncollected job at node turn-end + burns node iterations via the guardrail and can fail the node." +- Repo assets are canonical. Sync mechanism: T6 produces the list of every + modified file under assets/; after merge, each is copied (plain `cp`) to + its mirrored path under ~/.config/coyote/ (e.g. assets/agents// + index.yaml → ~/.config/coyote/agents//index.yaml). That list is + recorded as a follow-up item in the PR body. Built-in prompt text in + src/config/prompts.rs is compiled into the binary and needs no sync. +- config.example.yaml: `max_concurrent_jobs` next to + compression_threshold/max_tool_result_chars (~lines 200-206) with the + `0 = disabled` semantics documented. +- CHANGELOG + README/docs section for the new tools. +- **Wiki (user-required 2026-08-24)** — users must be able to discover and + understand the subsystem. Target: the GitHub wiki + (github.com/Dark-Alex-17/coyote/wiki — a SEPARATE git repo, + `coyote.wiki`, OUTSIDE this run's write boundary). Required content, new + page `Background-Jobs.md`: + 1. What background jobs are (whitelisted tool calls running as detached + tasks) and when to use them vs. agents (§4 directionality ruling: + agents can start jobs, never the reverse — jobs run single tool + calls; agents think); + 2. The five `job__*` tools with a worked example (start a long build → + keep working → notification arrives → collect); + 3. The backgroundable whitelist + the teaching errors users will see; + 4. Push notifications: the `system_notifications` key and the turn-end + guardrail, in user-visible terms; + 5. Configuration: global `max_concurrent_jobs`, per-agent override, + `0` disables, and the function-calling requirement (R8) — + jobs simply don't appear otherwise; + 6. Behavioral fine print: snapshot semantics, jobs die with the coyote + process (R5, no persistence), output tail cap + `tail_lines`, + `job__*` visible in `.info tools` but deliberately not toggleable + via `.tool` (R10), graph LLM-node guidance (collect before the + final node turn). + Cross-links: Home.md feature blurb + README features list entry + pointing at the new page, and a "jobs vs. subagents" note on the + existing agents wiki page. Process (macros-run precedent, coyote.wiki + 198f82c): T6 DRAFTS the full page content and cross-link diffs as a + task artifact (task log.md); actual publication to coyote.wiki is + recorded in the PR body's Follow-up section and executed AFTER merge — + the wiki must only ever describe merged behavior, and the wiki repo is + outside the run branch's write target. + +## 11. Implementation sketch (task-shaped) + +0. **T0 — characterization tests (§9.3).** Pin current behavior for every + coverage gap. Merges FIRST; no production code changes. +1. **T1 — TaskHandle enum + Supervisor generalization.** `TaskHandle`, + `HashMap`, per-kind capacity (R7 resolution: + agent-override → global → 5; 0 disables), namespaced ids, JobState cell + + kill discipline (§6), `cancel_recursive`/`cancel_all` Job-variant + handling, Drop-kill (pgid-guarded), cross-kind teaching errors, + supervisor init condition (R9), `use_agent` gains `cancel_recursive()` + on the old supervisor before replacement (rc.rs:4155 — §6 reset-site + fix), explicit kill-all on the REPL quit path (§6 shutdown). Config + plumbing: global Config field + AgentConfig Option + all four AppConfig + touch points. T0 stays green. +2. **T2 — Deliberate deltas (THREE separate commits — §9.1 isolation).** + Commit (a): guardrail predicate fix — `pending_task_ids` incl. + finished-but-uncollected, kind-aware prompt, ForceTerminate + discard-with-warning (H7). Commit (b): align `agent__check` to + pure-probe semantics (R6, supervisor.rs:844) with tool-description/ + prompt updates. Commit (c): `truncate_if_needed` UTF-8-boundary fix. + Each commit rewrites its own T0-marked tests — explicitly, in that + commit. Independently shippable; fixes the agent output-abandonment + bug even without jobs. +3. **T3 — job runner + `job__*` handlers + conditional injection (R8/R9).** + Extracted process runner (`JobEnvSnapshot`, process_group(0) per §6 + platform strategy, stdin=null, ring buffer, LLM_OUTPUT read post-wait()), + `JobCtx` for MCP invokes, the five handlers (lock discipline per §6), + whitelist + teaching errors, sync pre-execution gates in `job__start`, + context-availability validation (R11) against the per-request + declared-names stash captured in `before_chat_completion` (rc.rs:896; + all seven turn-loop call sites — §3 hardening rule 1, Oracle B1), + single-entry JobCtx McpRuntime (§3 rule 4, Oracle N3), graph-LLM-node + lifecycle semantics (§12, Oracle B3), REPL-surface wiring (R10: + `job__` in `concrete_tool_names()` exclusion; `.info tools` verified + free), + declaration/prompt gating at all injection sites (agent init, + plain-session Functions::init sites, exit_agent rebuild, + select_enabled_functions carve-outs). Per the §13.2 audit: background + runner on `tokio::process` (foreground stays sync-std, untouched); + capture-only, no live tee; `COYOTE_TOOL_TIMEOUT` resolved at start and + enforced with a pgid-guarded group kill; env-derived bin dirs + + vault-interpolated agent envs frozen into the snapshot at start. + Includes the tail-biased result cap + `tail_lines` param (§3, char- + boundary floored). PLUS (Oracle finding 8): an explicit audit item — + enumerate every `ctx.supervisor`/`parent_supervisor` consumer + (guardrail, taskqueue/mailbox handlers, REPL displays, session + save/load) and verify each behaves correctly with an agent-capacity-0 + supervisor (the novel R9 default-on state in plain sessions). +4. **T4 — NotificationQueue + merge_system_channel.** Per-ctx queue, + producers (jobs, then the agent spawn wrapper), single-pass merger + refactor replacing `inject_escalation_notification` (byte-identical + output when notifications are empty — §9.2.6), stale suppression, + no-notify-on-cancel, panic-skip semantics per §7. Wire drain into + `eval_tool_calls` at any depth. The agent-producer (R4 — a §9.1 delta) + is its own commit within T4. +5. **T5 — loop-tracker exemptions + no-change check hint.** +6. **T6 — prompts/docs/CHANGELOG/config.example.yaml + config sync (§10), + including the README features-list entry and the DRAFT of the + `Background-Jobs.md` wiki page + cross-link diffs (§10 wiki bullet; + publication to the separate coyote.wiki repo is a post-merge follow-up + recorded in the PR body, never done on the run branch).** +7. **T7 — feature tests**: §9.2 invariants (0 vs >0 states), double-wrap + regression (escalation+notification same batch, non-object output), + stale-notification suppression, orphan process-group kill (incl. + grandchild, Unix), pgid-guard (no kill after normal collect), + LLM_OUTPUT-vs-ring-buffer split, JoinError→failed mapping, guardrail + enumeration of finished handles, tracker exemption masking test + (`[check, X, check, X]` still detects X), cross-kind id errors, + use_agent/exit_agent kills jobs, jobs-only supervisor (agent capacity 0) + rejects agent__spawn with today's capacity error, tail-cap char-boundary + tests. PLUS R10/R11 surface tests: `concrete_tool_names()` excludes + `job__`; extended infra-preservation test (job__* survive agent/role + `enabled_tools` filters, incl. the empty-list case); `job__start` + rejects a context-filtered tool and accepts an in-filter one; toggle + of `job__start` errors as unknown. PLUS §3 validation-hardening tests: + `job__start {tool: "bash"}` / a path-shaped name / a PATH-resolvable + binary that is not a declared tool → rejected with NO process spawned + (assert the runner is never invoked); jobs-disabled context + + hallucinated `job__start` → eval's existing "Unexpected call" + soft-fail (B2 option (b) ruling; mod.rs:2564 stays green); + `job__start {tool: "mcp_invoke_X"}` for + a started-but-not-context-enabled server X → R11 rejection; subagent + `job__check/collect/cancel` cannot reach a parent-supervisor job id + (isolation → unknown-id error); declared+enabled but NOT whitelisted + (e.g. `memory__write`, `fs_read`, and the directionality cases + `agent__spawn`/`user__select` — §4 ruling) → whitelist teaching error, + no spawn + (Oracle N1 — rule 2's owning test); `mapping_tools` alias name → + rejected (concrete names only, rule 1); stash freshness — stash + refreshed per request, mid-batch `skill__load` does not grant until + next request (B1); None→Some flip pinned both states + jobs-off guard + for the rc.rs:5493 None-test (N2); `.info tools` positive assertion + automated via the tools_info unit tests (rc.rs:6025-6079, N5); + graph-node job lifecycle: node-started job registers in the shared + ctx.supervisor, survives node completion, notification drains on a + later turn of the same ctx; guardrail iteration-burn characterized + (B3). + +Dependency order & worktree parallelization (Oracle-confirmed): +T0 → T1 (strictly sequential; T0 is the safety net) → **T2 ∥ T3** (disjoint +file sets EXCEPT function/mod.rs — T2 commit (c) touches :404-413 while T3 +touches the :1420 routing region: non-overlapping hunks, rebase-safe, not +conflict-free; otherwise T2 = function/supervisor.rs, T3 = function/jobs.rs ++ agent.rs/rc.rs gating + config) → T4 (after BOTH — touches +mod.rs's merger and supervisor.rs's spawn wrapper) → **T5 ∥ T6** → T7 last +(may overlap T6 only). + +## 12. Risks + +- **Prompt-budget creep**: `system_notifications` + guardrail enumeration + + `pending_escalations` can stack; keep entries one-line terse. +- **Snapshot drift**: JobEnvSnapshot must capture exactly what the foreground + runner reads, or background runs behave differently — enumerate the field + set during T3 with a test comparing foreground/background env of the same + tool. +- **AppConfig four-touch-point trap** (R7): struct, Default, From, + env override — missing one silently pins the default. +- **Sandbox mode**: RESOLVED (§13.5) — no risk. Sandboxed coyote runs INSIDE + the container (kit entrypoint IS the coyote binary; `sbx run` attach at + sandbox/mod.rs:1093-1100); tools are always local children of coyote; + src/function/ has ZERO sandbox awareness. killpg works identically in and + out of sbx. (Pre-existing caveat, unchanged by this design: the foreground + timeout path uses single-PID `child.kill()`, so grandchildren can survive + a foreground timeout — background jobs fix this for themselves via + process groups on Unix.) +- **MCP server lifecycle vs running jobs**: a job's `Arc` + keeps the old service instance alive across a registry restart/shutdown + mid-job — the job finishes against the OLD server. Acceptable v1; + document in the tool description. +- **Hard-exit orphans**: `process::exit`/panic paths skip destructors and + jobs are in their own process group — accepted v1 (§6 shutdown + mechanism); normal REPL quit kills all. +- **ACP/graph surfaces** — CORRECTED + RULED (Oracle B3, 2026-08-24): the + old "graph mode out of scope" claim conflated two node kinds. The + supervisor bypass (`run_agent_for_graph`, supervisor.rs:510-596) means + the jobs/notifications exclusion applies ONLY + to graph AGENT nodes (the `run_agent_for_graph` path). Graph **LLM + nodes** are IN SCOPE — consistent with R11's user-confirmed text: they + are one of the §2 turn-loop sites, run on the parent session's + `&mut RequestContext`, and already call the guardrail (graph/llm.rs:271). + Semantics (owned by T3, tested in T7): a node-started job registers in + the SHARED ctx.supervisor and OUTLIVES the node run — mechanically + coherent: notifications drain into later turns on the same ctx, and the + turn-end guardrail surfaces still-running handles. Iteration-burn + hazard: the guardrail Inject arm inside the node's run_chat_loop + consumes node `max_iterations` and bails the WHOLE node at the limit + (graph/llm.rs:281-291) — a node that backgrounds a long job with + nothing else to do converts success into an error. Mitigation is + prompt guidance (§10: collect/cancel before the final node turn), NOT + a mechanical carve-out in v1. + +## 13. Open questions / VERIFY — ALL RESOLVED + +1. **JobCtx vs full RequestContext clone** — RESOLVED 2026-08-21 (audit): + purpose-built `JobCtx { mcp_runtime: McpRuntime, current_depth: usize }` + WINS decisively. eval_mcp's complete transitive ctx surface is exactly + `ctx.tool_scope.mcp_runtime` (+ `current_depth` for print gating) — + the single partition + concurrent join in `eval_tool_calls` + (mod.rs:291-326) and the five `eval_mcp` arms (mod.rs:1367-1418); + `McpRuntime::invoke` is self-contained and returns the raw + `CallToolResult` (tool_scope.rs:289-304). Post-invoke, the foreground + routes that result through `render_tool_result` (mod.rs:1989, called at + :1635) — a FREE function (spill under `paths::cache_dir()`/mcp-resources, + `TEXT_MAX_BYTES_CLAMP` paging; zero ctx) — and the job path calls the + SAME function for parity, so the audit conclusion is UNCHANGED after the + 2026-08-25 MCP resources/prompts merge. RequestContext has NO Clone impl + at all; the nearest equivalent `fork_for_branch` (rc.rs:433-463) + deep-copies the ENTIRE session transcript (Vec ×2) + Functions + and would drag parking_lot supervisor locks into the detached task. + OAuth needs nothing from ctx: per-request bearer injection + 401 + force-refresh-retry-once live INSIDE the transport (`McpOAuthClient`, + auth_client.rs:42-99) with a process-global token store and per-server + single-flight locks — mid-job refresh works through JobCtx unchanged. +2. **JobEnvSnapshot field set** — RESOLVED 2026-08-21 (audit): exact field + table + runner-mechanics rulings in §4. Notable: NO pre-execution gates + exist (H3 is future-proofing); the external eval branch never uses + `&mut ctx` (clean extraction); foreground runner is sync-std polled from + async (background uses tokio::process; foreground untouched v1); + live-tee threads must not run for jobs; vault-interpolated agent envs + resolve into the snapshot at start (same exposure as foreground). +3. **`agent__check` consume-on-finished quirk** — RESOLVED 2026-08-21 + (user, R6): align — check never consumes, for agents AND jobs; + deliberate behavior change, tool descriptions/prompts updated in T2/T6. +4. **Defaults & knob placement** — RESOLVED 2026-08-21 (user, R7 amended): + `max_concurrent_jobs` = global Config field (default 5, `0 = disabled`) + + Option AgentConfig override, resolved agent-first + (max_tool_result_chars pattern); ring buffer 64 KiB, no-change hint + threshold 5, SIGTERM grace 5s stand. Role/session-runtime overrides + deferred for both concurrency knobs together. +5. **Sandbox (sbx) execute_command** — RESOLVED 2026-08-21 (audit): clean + answer — sandbox mode does NOT touch tool execution. Coyote itself runs + inside the container (kit entrypoint = coyote; the only `sbx exec` calls + are launch-time setup, sandbox/mod.rs:1051-1128); tools are plain local + `std::process` children everywhere; `grep sandbox src/function/` = zero + matches. killpg is fully viable in sbx; no whitelist carve-out needed. +6. **Huge job outputs at collect** — RESOLVED 2026-08-21 (audit): plain + `max_tool_result_chars` truncation is INADEQUATE (default null = no cap; + head-keeping = wrong end for logs; UTF-8 boundary bug — §9.1). Ruling: + tail-biased job-result cap (default keep last 50,000 chars, explicit + header, char-boundary floored) + optional `tail_lines` param on + `job__collect` (§3); no LLM summarization v1. +7. **Whitelist boundary for custom tools** — RESOLVED 2026-08-21 (user): + ALL external command tools are backgroundable — bash (argc), + JavaScript/TypeScript, and Python — no per-tool opt-in flag; they are + process-isolated by construction. +8. **Feature-off gating** — RESOLVED 2026-08-21 (user, R8/R9): effective + `max_concurrent_jobs == 0` → job__* declarations and prompt text not + injected (can_spawn_agents-style gating); jobs available in plain + sessions via the global value (R9). From bfc3b7bfeaaecbeaff7647fef09bd0d4174ceeb5 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 13:55:15 -0600 Subject: [PATCH 02/28] test: pin current tool-eval, guardrail, and truncation behavior ahead of background-jobs work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit T0 characterization safety net per plans/background-jobs-design.md §9.3: 43 tests pinning handle_collect/check/cancel/spawn, the pending-agents guardrail (incl. ForceTerminate + counter resets), eval_tool_calls partition/re-sort/soft-fail/ loop-alert/truncation, truncate_if_needed's UTF-8 boundary edge, ToolCall::eval prefix routing, merge_tool_results shape, and cancel_recursive recursion. Known-buggy behaviors deliberately pinned for visible later diffs: handle_check consumes finished handles, guardrail ignores finished-but-uncollected agents, mid-char truncation returns the full string with marker prepended. Not covered (findings): empty-after-dedup bail is unreachable from non-empty input; run_child_agent needs a mock LLM client (none exists) — manual case; over-threshold summarization pinned via deterministic unknown-model failure. --- src/config/input.rs | 68 +++++++ src/function/mod.rs | 313 +++++++++++++++++++++++++++- src/function/supervisor.rs | 408 ++++++++++++++++++++++++++++++++++++- src/supervisor/mod.rs | 21 ++ 4 files changed, 808 insertions(+), 2 deletions(-) diff --git a/src/config/input.rs b/src/config/input.rs index 71f82d5..5d35064 100644 --- a/src/config/input.rs +++ b/src/config/input.rs @@ -593,6 +593,8 @@ mod tests { use super::*; use crate::config::request_context::RequestContext; use crate::config::{AppState, WorkingMode}; + use crate::function::ToolCall; + use serde_json::json; use std::fs; use std::sync::Arc; use std::time::SystemTime; @@ -973,4 +975,70 @@ mod tests { )); assert!(result.is_err()); } + + fn tool_result(id: &str, output: &str) -> ToolResult { + ToolResult::new( + ToolCall::new("t".into(), json!({}), Some(id.to_string())), + json!(output), + ) + } + + #[test] + fn merge_tool_results_first_merge_creates_container() { + let ctx = create_test_ctx(); + let input = Input::from_str(&ctx, "test", None).unwrap(); + + let input = + input.merge_tool_results("assistant text".into(), vec![tool_result("id-1", "ok")]); + + let tool_calls = input.tool_calls().as_ref().unwrap(); + assert_eq!(tool_calls.text, "assistant text"); + assert!(!tool_calls.sequence); + assert_eq!(tool_calls.tool_results.len(), 1); + assert!(tool_calls.tool_results[0].text.is_none()); + } + + #[test] + fn merge_tool_results_second_merge_marks_sequence_and_tags_text() { + let ctx = create_test_ctx(); + let input = Input::from_str(&ctx, "test", None) + .unwrap() + .merge_tool_results("assistant text".into(), vec![tool_result("id-1", "ok")]); + + let input = + input.merge_tool_results("second text".into(), vec![tool_result("id-2", "ok2")]); + + let tool_calls = input.tool_calls().as_ref().unwrap(); + assert!(tool_calls.sequence); + assert_eq!(tool_calls.tool_results.len(), 2); + assert_eq!(tool_calls.text, "assistant text"); + assert!(tool_calls.tool_results[0].text.is_none()); + assert_eq!( + tool_calls.tool_results[1].text, + Some("second text".to_string()) + ); + } + + #[test] + fn build_messages_wraps_tool_results_in_single_assistant_message() { + let ctx = create_test_ctx(); + let input = Input::from_str(&ctx, "test", None) + .unwrap() + .merge_tool_results("assistant text".into(), vec![tool_result("id-1", "ok")]) + .merge_tool_results("second text".into(), vec![tool_result("id-2", "ok2")]); + + let messages = input.build_messages().unwrap(); + + let tool_call_messages: Vec<_> = messages + .iter() + .filter(|m| matches!(m.content, MessageContent::ToolCalls(_))) + .collect(); + assert_eq!(tool_call_messages.len(), 1); + let message = tool_call_messages[0]; + assert!(matches!(message.role, MessageRole::Assistant)); + let MessageContent::ToolCalls(tool_calls) = &message.content else { + unreachable!(); + }; + assert_eq!(tool_calls.tool_results.len(), 2); + } } diff --git a/src/function/mod.rs b/src/function/mod.rs index d32a24c..f552854 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -2474,7 +2474,7 @@ mod tests { FIXTURE_ANNOTATED_TEXT, FIXTURE_ANNOTATED_URI, FIXTURE_BLOB_BYTES, FIXTURE_BLOB_URI, FIXTURE_LOG_TEXT, FIXTURE_LOG_URI, FixtureServer, fixture_runtime, }; - use crate::config::{AppState, WorkingMode}; + use crate::config::{Agent, AgentConfig, AppConfig, AppState, WorkingMode}; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; use base64::Engine; use base64::engine::general_purpose::STANDARD; @@ -4004,4 +4004,315 @@ mod tests { assert!(dir.is_dir()); fs::remove_dir_all(&dir).unwrap(); } + + #[test] + fn eval_tool_calls_partitions_mcp_and_sequential_then_resorts() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let calls = vec![ + call("unknown_first", Some("id-1")), + ToolCall::new( + "mcp_search_foo".into(), + json!({"query": "q"}), + Some("id-2".into()), + ), + call("unknown_last", Some("id-3")), + ]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + assert_eq!(results.len(), 3); + assert_eq!(results[0].call.name, "unknown_first"); + assert_eq!(results[1].call.name, "mcp_search_foo"); + assert_eq!(results[2].call.name, "unknown_last"); + + for sequential in [&results[0], &results[2]] { + let err = sequential.output["tool_call_error"].as_str().unwrap(); + assert!( + err.contains("use only tools listed in your catalog"), + "{err}" + ); + } + let mcp_err = results[1].output["tool_call_error"].as_str().unwrap(); + assert!(mcp_err.starts_with("MCP search failed"), "{mcp_err}"); + assert!(!mcp_err.contains("use only tools listed in your catalog")); + } + + #[test] + fn eval_tool_calls_isolates_failures_within_a_batch() { + let app = AppState { + config: Arc::new(AppConfig { + auto_continue: true, + ..Default::default() + }), + ..AppState::test_default() + }; + let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd); + ctx.tool_scope.functions.append_todo_functions(); + let calls = vec![ + ToolCall::new( + "todo__init".into(), + json!({"goal": "ship it"}), + Some("id-1".into()), + ), + call("unknown_tool", Some("id-2")), + ]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + assert_eq!(results.len(), 2); + assert_eq!(results[0].call.name, "todo__init"); + assert_eq!(results[0].output["status"], "ok"); + assert!(results[0].output.get("tool_call_error").is_none()); + let err = results[1].output["tool_call_error"].as_str().unwrap(); + assert!(err.contains("Unexpected call"), "{err}"); + } + + #[test] + fn eval_tool_calls_reports_loop_alert_without_executing() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let looped = call_with_args("looped_tool", json!({"a": 1})); + ctx.tool_scope.tool_tracker.record_call(looped.clone()); + ctx.tool_scope.tool_tracker.record_call(looped.clone()); + let calls = vec![ + looped, + ToolCall::new("other_tool".into(), json!({}), Some("id-2".into())), + ]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + assert_eq!(results.len(), 2); + let alert = results[0].output.as_str().unwrap(); + assert!(alert.starts_with("{\"tool_call_loop_alert\":"), "{alert}"); + let err = results[1].output["tool_call_error"].as_str().unwrap(); + assert!(err.contains("Unexpected call"), "{err}"); + } + + #[test] + fn eval_tool_calls_truncates_with_global_max_chars() { + let app = AppState { + config: Arc::new(AppConfig { + max_tool_result_chars: Some(50), + ..Default::default() + }), + ..AppState::test_default() + }; + let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd); + let calls = vec![ToolCall::new( + "unknown_tool".into(), + json!({"padding": "x".repeat(200)}), + Some("id-1".into()), + )]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + let out = results[0].output.as_str().unwrap(); + assert!( + out.starts_with("[truncated: tool output exceeded 50 chars]\n"), + "{out}" + ); + } + + #[test] + fn eval_tool_calls_agent_max_chars_overrides_global() { + let app = AppState { + config: Arc::new(AppConfig { + max_tool_result_chars: Some(5000), + ..Default::default() + }), + ..AppState::test_default() + }; + let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd); + ctx.agent = Some(Agent::test_new(AgentConfig { + max_tool_result_chars: Some(30), + ..Default::default() + })); + let calls = vec![ToolCall::new( + "unknown_tool".into(), + json!({"padding": "x".repeat(200)}), + Some("id-1".into()), + )]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + let out = results[0].output.as_str().unwrap(); + assert!( + out.starts_with("[truncated: tool output exceeded 30 chars]\n"), + "{out}" + ); + } + + #[test] + fn eval_tool_calls_zero_max_chars_disables_truncation() { + let app = AppState { + config: Arc::new(AppConfig { + max_tool_result_chars: Some(0), + ..Default::default() + }), + ..AppState::test_default() + }; + let mut ctx = RequestContext::new(Arc::new(app), WorkingMode::Cmd); + let calls = vec![ToolCall::new( + "unknown_tool".into(), + json!({"padding": "x".repeat(200)}), + Some("id-1".into()), + )]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + assert!(results[0].output["tool_call_error"].is_string()); + assert!(!results[0].output.to_string().contains("[truncated")); + } + + #[test] + fn eval_tool_calls_no_max_chars_configured_never_truncates() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let calls = vec![ToolCall::new( + "unknown_tool".into(), + json!({"padding": "x".repeat(200)}), + Some("id-1".into()), + )]; + + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + assert!(results[0].output["tool_call_error"].is_string()); + assert!(!results[0].output.to_string().contains("[truncated")); + } + + /// Pins current behavior: when the char cap lands inside a multi-byte + /// UTF-8 character of the serialized output, no prefix can be taken, so + /// the truncation marker is prepended to the FULL original output and the + /// "truncated" result is longer than the input. + #[test] + fn truncate_if_needed_utf8_boundary_returns_full_output_with_marker() { + let serialized = json!("aé").to_string(); + let result = ToolResult::new(call("t", Some("id-1")), json!("aé")); + + let truncated = result.truncate_if_needed(3); + + let out = truncated.output.as_str().unwrap(); + assert_eq!( + out, + format!("[truncated: tool output exceeded 3 chars]\n{serialized}") + ); + assert!(out.len() > serialized.len()); + } + + #[test] + fn eval_routes_agent_prefix_to_supervisor_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_supervisor_functions(); + + let out = + run_async(call_with_args("agent__check", json!({"id": "x"})).eval(&mut ctx)).unwrap(); + + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with("Supervisor tool failed"), "{err}"); + assert!(err.contains("No supervisor active"), "{err}"); + } + + #[test] + fn eval_routes_todo_prefix_to_todo_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_todo_functions(); + + let out = run_async(call_with_args("todo__list", json!({})).eval(&mut ctx)).unwrap(); + + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with("Todo tool failed"), "{err}"); + assert!(err.contains("Auto-continue is not enabled"), "{err}"); + } + + #[test] + fn eval_routes_memory_prefix_to_memory_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_memory_functions(); + + let out = run_async(call_with_args("memory__read", json!({})).eval(&mut ctx)).unwrap(); + + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with("Memory tool failed"), "{err}"); + assert!(err.contains("name is required"), "{err}"); + } + + #[test] + fn eval_routes_skill_prefix_to_skill_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_skill_functions(); + + let out = run_async(call_with_args("skill__load", json!({})).eval(&mut ctx)).unwrap(); + + assert_eq!(out["error"], "name is required"); + } + + #[test] + fn eval_routes_user_prefix_to_user_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_user_interaction_functions(); + + let out = run_async(call_with_args("user__confirm", json!({})).eval(&mut ctx)).unwrap(); + + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with("User interaction failed"), "{err}"); + assert!(err.contains("'question' is required"), "{err}"); + } + + #[test] + fn eval_routes_rag_prefix_to_rag_handler() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_rag_query_functions(); + + let out = + run_async(call_with_args("rag__query", json!({"query": "x"})).eval(&mut ctx)).unwrap(); + + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with("RAG query failed"), "{err}"); + assert!(err.contains("No RAG is attached"), "{err}"); + } + + #[test] + fn eval_unknown_name_errors_with_unexpected_call() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + + let err = run_async(call_with_args("nope", json!({})).eval(&mut ctx)).unwrap_err(); + + assert!(err.to_string().contains("Unexpected call"), "{err}"); + } + + #[test] + fn eval_mcp_empty_runtime_returns_distinct_error_per_prefix() { + let ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let cases = [ + ( + "mcp_invoke_ghost", + json!({"tool": "t"}), + "MCP tool invocation failed", + ), + ( + "mcp_search_ghost", + json!({"query": "q"}), + "MCP search failed", + ), + ( + "mcp_describe_ghost", + json!({"tool": "t"}), + "MCP describe failed", + ), + ( + "mcp_read_ghost", + json!({"uri": "file:///x"}), + "MCP read failed", + ), + ( + "mcp_prompt_ghost", + json!({"prompt": "p"}), + "MCP prompt failed", + ), + ]; + + for (name, args, expected) in cases { + let out = run_async(call_with_args(name, args).eval_mcp(&ctx)).unwrap(); + let err = out["tool_call_error"].as_str().unwrap(); + assert!(err.starts_with(expected), "{name}: {err}"); + } + } } diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index 32c5d48..87dfbe5 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -1473,14 +1473,24 @@ mod tests { } fn register_fake_agent(ctx: &mut RequestContext, id: &str, name: &str) { + register_fake_agent_with_output(ctx, id, name, "fake output"); + } + + fn register_fake_agent_with_output( + ctx: &mut RequestContext, + id: &str, + name: &str, + output: &str, + ) { let rt = tokio::runtime::Runtime::new().unwrap(); let id_owned = id.to_string(); let name_owned = name.to_string(); + let output_owned = output.to_string(); let join_handle = rt.spawn(async move { Ok(AgentResult { id: id_owned, agent_name: name_owned, - output: "fake output".into(), + output: output_owned, exit_status: AgentExitStatus::Completed, }) }); @@ -1511,6 +1521,48 @@ mod tests { .block_on(f) } + fn register_running_agent(ctx: &mut RequestContext, id: &str, name: &str) -> AbortSignal { + let abort = create_abort_signal(); + let id_owned = id.to_string(); + let name_owned = name.to_string(); + let join_handle = tokio::spawn(async move { + time::sleep(Duration::from_secs(60)).await; + Ok(AgentResult { + id: id_owned, + agent_name: name_owned, + output: String::new(), + exit_status: AgentExitStatus::Completed, + }) + }); + let handle = AgentHandle { + id: id.to_string(), + agent_name: name.to_string(), + depth: 1, + inbox: Arc::new(Inbox::new()), + abort_signal: abort.clone(), + join_handle, + child_supervisor: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + abort + } + + fn wait_until_finished(ctx: &RequestContext, id: &str) { + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while ctx.supervisor.as_ref().unwrap().read().is_finished(id) != Some(true) { + assert!( + std::time::Instant::now() < deadline, + "agent '{id}' never finished" + ); + std::thread::sleep(Duration::from_millis(10)); + } + } + #[tokio::test] async fn sync_agent_functions_gates_meta_functions_on_live_capabilities() { let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); @@ -2127,4 +2179,358 @@ mod tests { } }); } + + #[test] + fn handle_collect_finished_agent_returns_output_and_consumes_handle() { + let mut ctx = ctx_with_supervisor(4, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + ctx.pending_agents_guardrail_count = 2; + + let result = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(result["status"], "completed"); + assert_eq!(result["id"], "a1"); + assert_eq!(result["agent"], "explore"); + assert_eq!(result["exit_status"], "Completed"); + assert_eq!(result["output"], "fake output"); + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), + None + ); + } + + #[test] + fn handle_collect_pending_escalations_early_out_keeps_handle() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_supervisor(4, 3); + let _abort = register_running_agent(&mut ctx, "slow", "test"); + let queue = ctx.ensure_root_escalation_queue(); + let (tx, _rx) = tokio::sync::oneshot::channel(); + queue.submit(EscalationRequest { + id: "esc_1".into(), + from_agent_id: "a1".into(), + from_agent_name: "explore".into(), + question: "What do?".into(), + options: None, + reply_tx: tx, + }); + + let result = handle_collect(&mut ctx, &json!({"id": "slow"})) + .await + .unwrap(); + + assert_eq!(result["status"], "pending"); + assert!(result["pending_escalations"].is_array()); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("slow"), + Some(false) + ); + }); + } + + #[test] + fn handle_collect_unknown_agent_errors() { + let mut ctx = ctx_with_supervisor(4, 3); + let result = run_async(handle_collect(&mut ctx, &json!({"id": "missing"}))).unwrap(); + assert_eq!(result["status"], "error"); + assert!(result["message"].as_str().unwrap().contains("not found")); + } + + #[test] + fn handle_collect_without_agent_passes_long_output_through_verbatim() { + let mut ctx = ctx_with_supervisor(4, 3); + let long_output = "x".repeat(10_000); + register_fake_agent_with_output(&mut ctx, "a1", "explore", &long_output); + + let result = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(result["output"], long_output); + } + + #[test] + fn handle_collect_output_below_agent_threshold_passes_through() { + let mut ctx = ctx_with_supervisor(4, 3); + ctx.agent = Some(Agent::test_new(AgentConfig { + summarization_threshold: 1_000_000, + ..Default::default() + })); + register_fake_agent(&mut ctx, "a1", "explore"); + + let result = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(result["status"], "completed"); + assert_eq!(result["output"], "fake output"); + } + + #[test] + fn handle_collect_over_threshold_with_unknown_summarization_model_errors() { + let mut ctx = ctx_with_supervisor(4, 3); + ctx.agent = Some(Agent::test_new(AgentConfig { + summarization_threshold: 1, + summarization_model: Some("nonexistent_client:model".into()), + ..Default::default() + })); + register_fake_agent(&mut ctx, "a1", "explore"); + + let err = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap_err(); + + assert!(err.to_string().contains("nonexistent_client")); + } + + #[test] + fn guardrail_no_supervisor_is_no_action_and_resets_counter() { + let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); + ctx.pending_agents_guardrail_count = 2; + + assert!(matches!( + check_pending_agents_guardrail(&mut ctx), + GuardrailAction::NoAction + )); + assert_eq!(ctx.pending_agents_guardrail_count, 0); + } + + /// Pins current behavior: a finished-but-uncollected agent is not counted + /// as pending (only still-running agents are), so the turn-end guardrail + /// takes no action and the finished agent's result can be silently dropped. + #[test] + fn guardrail_ignores_finished_but_uncollected_agents() { + let mut ctx = ctx_with_supervisor(4, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + wait_until_finished(&ctx, "a1"); + ctx.pending_agents_guardrail_count = 2; + + assert!(matches!( + check_pending_agents_guardrail(&mut ctx), + GuardrailAction::NoAction + )); + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), + Some(true) + ); + } + + #[test] + fn guardrail_force_terminates_at_max_and_cancels_agents() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_supervisor(4, 3); + let abort = register_running_agent(&mut ctx, "slow", "test"); + ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::ForceTerminate(ids) => { + assert_eq!(ids, vec!["slow".to_string()]); + } + _ => panic!("expected ForceTerminate action"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert!(abort.aborted()); + }); + } + + #[test] + fn guardrail_injects_prompt_and_increments_counter_below_max() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_supervisor(4, 3); + let _abort = register_running_agent(&mut ctx, "slow", "test"); + ctx.pending_agents_guardrail_count = 1; + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::Inject(prompt) => { + assert!(prompt.contains("slow")); + assert!(prompt.contains("agent__collect")); + } + _ => panic!("expected Inject action"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 2); + }); + } + + #[test] + fn handle_cancel_resets_guardrail_counter() { + let mut ctx = ctx_with_supervisor(4, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + ctx.pending_agents_guardrail_count = 2; + + let result = run_async(handle_cancel(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(result["status"], "ok"); + assert_eq!(ctx.pending_agents_guardrail_count, 0); + } + + #[test] + fn handle_spawn_missing_agent_arg_errors() { + let mut ctx = ctx_with_supervisor(4, 3); + let err = run_async(handle_spawn(&mut ctx, &json!({}))).unwrap_err(); + assert!(err.to_string().contains("'agent' is required")); + } + + #[test] + fn handle_spawn_missing_prompt_arg_errors() { + let mut ctx = ctx_with_supervisor(4, 3); + let err = run_async(handle_spawn(&mut ctx, &json!({"agent": "explore"}))).unwrap_err(); + assert!(err.to_string().contains("'prompt' is required")); + } + + #[test] + fn handle_spawn_rejects_agent_outside_whitelist() { + let mut ctx = ctx_with_supervisor(4, 3); + ctx.agent = Some(Agent::test_new(AgentConfig { + spawnable_agents: Some(vec!["allowed".into()]), + ..Default::default() + })); + + let result = run_async(handle_spawn( + &mut ctx, + &json!({"agent": "notallowed", "prompt": "p"}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("spawnable_agents") + ); + } + + #[test] + fn handle_spawn_at_capacity_errors() { + let mut ctx = ctx_with_supervisor(1, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + + let result = run_async(handle_spawn( + &mut ctx, + &json!({"agent": "x", "prompt": "p"}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert_eq!( + result["message"], + "At capacity: 1/1 agents running. Wait for one to finish or cancel one." + ); + } + + #[test] + fn handle_spawn_exceeding_depth_errors() { + let mut ctx = ctx_with_supervisor(4, 0); + + let result = run_async(handle_spawn( + &mut ctx, + &json!({"agent": "x", "prompt": "p"}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("Max agent depth exceeded") + ); + } + + #[test] + fn handle_spawn_no_supervisor_errors() { + let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); + let err = run_async(handle_spawn( + &mut ctx, + &json!({"agent": "x", "prompt": "p"}), + )) + .unwrap_err(); + assert!(err.to_string().contains("No supervisor active")); + } + + /// Pins current behavior: checking a finished agent does not report a + /// "finished, ready to collect" status; it silently delegates to collect, + /// returning the full result and consuming the handle. + #[test] + fn handle_check_finished_agent_delegates_to_collect_and_consumes_handle() { + let mut ctx = ctx_with_supervisor(4, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + wait_until_finished(&ctx, "a1"); + + let result = run_async(handle_check(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(result["status"], "completed"); + assert_eq!(result["output"], "fake output"); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), + None + ); + } + + #[test] + fn handle_cancel_running_agent_aborts_and_waits_for_cleanup() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_supervisor(4, 3); + let sig = create_abort_signal(); + let sig2 = sig.clone(); + let join_handle = tokio::spawn(async move { + loop { + if sig2.aborted() { + return Ok(AgentResult { + id: "a1".into(), + agent_name: "explore".into(), + output: String::new(), + exit_status: AgentExitStatus::Completed, + }); + } + time::sleep(Duration::from_millis(10)).await; + } + }); + let handle = AgentHandle { + id: "a1".into(), + agent_name: "explore".into(), + depth: 1, + inbox: Arc::new(Inbox::new()), + abort_signal: sig.clone(), + join_handle, + child_supervisor: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + ctx.pending_agents_guardrail_count = 2; + + let result = handle_cancel(&mut ctx, &json!({"id": "a1"})).await.unwrap(); + + assert_eq!(result["status"], "ok"); + let message = result["message"].as_str().unwrap(); + assert!(message.contains("Cancelled agent 'explore'")); + assert!(message.contains("waited for cleanup")); + assert!(sig.aborted()); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), + None + ); + assert_eq!(ctx.pending_agents_guardrail_count, 0); + }); + } } diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index dc55d21..0739404 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -294,4 +294,25 @@ mod tests { AgentExitStatus::Failed("x".into()) ); } + + #[test] + fn cancel_recursive_aborts_nested_supervisors() { + let child_sig = create_abort_signal(); + let mut child_handle = make_handle("c1", "worker", 2); + child_handle.abort_signal = child_sig.clone(); + let mut child_sup = Supervisor::new(4, 3); + child_sup.register(child_handle).unwrap(); + + let parent_sig = create_abort_signal(); + let mut parent_handle = make_handle("a1", "explore", 1); + parent_handle.abort_signal = parent_sig.clone(); + parent_handle.child_supervisor = Some(Arc::new(RwLock::new(child_sup))); + let mut sup = Supervisor::new(4, 3); + sup.register(parent_handle).unwrap(); + + sup.cancel_recursive(); + + assert!(parent_sig.aborted()); + assert!(child_sig.aborted()); + } } From 7f3f95d89d61b2d0ac0bea8e491384b2ad0a8bb5 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 15:17:57 -0600 Subject: [PATCH 03/28] feat: generalize supervisor registry to TaskHandle enum with job scaffolding, kill discipline, and max_concurrent_jobs config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements T1 of plans/background-jobs-design.md (§6, R7/R8/R9): - Supervisor.handles is now HashMap where TaskHandle = Agent(AgentHandle) | Job(JobHandle); agent-facing accessors (active_count, effective_active_count, is_finished, take, inbox, abort_signal_for, list_agents) match only Agent variants, preserving all existing external behavior byte-for-byte. - New JobHandle/JobState/JobStatus/JobResult types with pgid-guarded process-group kill discipline: Drop and cancel_all/cancel_recursive kill the group only while state.pgid is still set (pid-reuse guard), via libc::killpg on unix and JoinHandle::abort elsewhere. - Per-kind job capacity: Supervisor carries max_concurrent_jobs (builder-set, default 0); job registration rejects at capacity. - Cross-kind teaching errors at the four agent-lookup miss sites (agent__check/collect/cancel/send_message) when the id is a registered job or job_-prefixed; genuinely-unknown ids keep their existing messages. - Supervisor init condition is now can_spawn_agents || jobs_enabled in use_agent and both child-agent spawn paths, with agent capacity 0 in jobs-only contexts; use_agent cancels the old supervisor recursively before replacing it. - max_concurrent_jobs config plumbing: global Config field, AgentConfig override + accessor, all four AppConfig touch points including the COYOTE_MAX_CONCURRENT_JOBS env override; shared effective_max_concurrent_jobs/jobs_enabled predicates (agent override -> global -> default 5; 0 disables). - Stage dependency-free RingBuf (64 KiB default) in src/function/jobs.rs for the upcoming job output pump. - New sanctioned dependency: libc 0.2 under cfg(unix). --- Cargo.lock | 1 + Cargo.toml | 3 + src/config/agent.rs | 6 + src/config/app_config.rs | 49 +++++ src/config/mod.rs | 7 +- src/config/request_context.rs | 201 ++++++++++++++++++++- src/function/jobs.rs | 161 +++++++++++++++++ src/function/mod.rs | 1 + src/function/supervisor.rs | 248 +++++++++++++++++++++++--- src/supervisor/mod.rs | 325 ++++++++++++++++++++++++++++++---- 10 files changed, 942 insertions(+), 60 deletions(-) create mode 100644 src/function/jobs.rs diff --git a/Cargo.lock b/Cargo.lock index 05028a1..d856e4a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1703,6 +1703,7 @@ dependencies = [ "inquire", "is-terminal", "json-patch", + "libc", "log", "log4rs", "nu-ansi-term", diff --git a/Cargo.toml b/Cargo.toml index d8473a6..fbfa196 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -139,6 +139,9 @@ arboard = { version = "3.3.0", default-features = false, features = [ [target.'cfg(not(any(target_os = "linux", target_os = "android", target_os = "emscripten")))'.dependencies] arboard = { version = "3.3.0", default-features = false } +[target.'cfg(unix)'.dependencies] +libc = "0.2" + [dev-dependencies] pretty_assertions = "1.4.0" rmcp = { version = "3.1.2", features = ["server"] } diff --git a/src/config/agent.rs b/src/config/agent.rs index 40771fa..acddb70 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -561,6 +561,10 @@ impl Agent { self.config.max_tool_result_chars } + pub fn max_concurrent_jobs(&self) -> Option { + self.config.max_concurrent_jobs + } + pub fn compression_keep_last(&self) -> Option { self.config.compression_keep_last } @@ -735,6 +739,8 @@ pub struct AgentConfig { #[serde(default, skip_serializing_if = "Option::is_none")] pub max_tool_result_chars: Option, #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_concurrent_jobs: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] pub compression_keep_last: Option, #[serde(default)] pub description: String, diff --git a/src/config/app_config.rs b/src/config/app_config.rs index 5dd6b2d..a050234 100644 --- a/src/config/app_config.rs +++ b/src/config/app_config.rs @@ -68,6 +68,7 @@ pub struct AppConfig { pub summarization_prompt: Option, pub summary_context_prompt: Option, pub max_tool_result_chars: Option, + pub max_concurrent_jobs: Option, pub memory: Option, pub memory_cap_with_tools: Option, @@ -153,6 +154,7 @@ impl Default for AppConfig { summarization_prompt: None, summary_context_prompt: None, max_tool_result_chars: None, + max_concurrent_jobs: None, memory: None, memory_cap_with_tools: None, @@ -239,6 +241,7 @@ impl AppConfig { summarization_prompt: config.summarization_prompt, summary_context_prompt: config.summary_context_prompt, max_tool_result_chars: config.max_tool_result_chars, + max_concurrent_jobs: config.max_concurrent_jobs, memory: config.memory, memory_cap_with_tools: config.memory_cap_with_tools, @@ -574,6 +577,9 @@ impl AppConfig { { self.compression_threshold = v; } + if let Some(v) = super::read_env_value::(&get_env_name("max_concurrent_jobs")) { + self.max_concurrent_jobs = v; + } if let Some(v) = super::read_env_value::(&get_env_name("summarization_prompt")) { self.summarization_prompt = v; } @@ -844,6 +850,49 @@ mod tests { } } + #[test] + fn from_config_copies_max_concurrent_jobs() { + let cfg = Config { + model_id: "test-model".to_string(), + max_concurrent_jobs: Some(3), + clients: vec![ClientConfig::default()], + ..Config::default() + }; + + let app = AppConfig::from_config(cfg).unwrap(); + + assert_eq!(app.max_concurrent_jobs, Some(3)); + } + + #[test] + #[serial_test::serial] + fn load_envs_overrides_max_concurrent_jobs() { + let env_name = get_env_name("max_concurrent_jobs"); + let prev = std::env::var_os(&env_name); + + let mut app = AppConfig::default(); + + unsafe { std::env::set_var(&env_name, "7") }; + app.load_envs(); + assert_eq!(app.max_concurrent_jobs, Some(7)); + + unsafe { std::env::set_var(&env_name, "0") }; + app.load_envs(); + assert_eq!(app.max_concurrent_jobs, Some(0)); + + unsafe { std::env::remove_var(&env_name) }; + app.max_concurrent_jobs = Some(2); + app.load_envs(); + assert_eq!(app.max_concurrent_jobs, Some(2)); + + unsafe { + match prev { + Some(v) => std::env::set_var(&env_name, v), + None => std::env::remove_var(&env_name), + } + } + } + #[test] fn editor_returns_configured_value() { let configured = cached_editor() diff --git a/src/config/mod.rs b/src/config/mod.rs index 08992fd..6f070b4 100644 --- a/src/config/mod.rs +++ b/src/config/mod.rs @@ -42,7 +42,10 @@ pub use self::macro_policy::{ MacroAllowlistLevel, MacroPolicy, MacroSource, MacroState, RESERVED_MACRO_NAMES, ResolvedMacro, }; #[allow(unused_imports)] -pub use self::request_context::{RenderMode, RequestContext, should_inject_skill_instructions}; +pub use self::request_context::{ + RenderMode, RequestContext, effective_max_concurrent_jobs, jobs_enabled, + should_inject_skill_instructions, +}; pub use self::role::{ CODE_ROLE, CREATE_TITLE_ROLE, EXPLAIN_SHELL_ROLE, Role, RoleLike, SHELL_ROLE, }; @@ -264,6 +267,7 @@ pub struct Config { pub summarization_prompt: Option, pub summary_context_prompt: Option, pub max_tool_result_chars: Option, + pub max_concurrent_jobs: Option, pub memory: Option, pub memory_cap_with_tools: Option, @@ -346,6 +350,7 @@ impl Default for Config { summarization_prompt: None, summary_context_prompt: None, max_tool_result_chars: None, + max_concurrent_jobs: None, memory: None, memory_cap_with_tools: None, diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 2be3d37..3dc350c 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -138,6 +138,17 @@ pub fn should_inject_skill_instructions(app: &AppConfig, policy: &SkillPolicy) - app.function_calling_support && policy.skills_enabled && !policy.compatible_enabled.is_empty() } +pub fn effective_max_concurrent_jobs(agent: Option<&Agent>, app: &AppConfig) -> usize { + agent + .and_then(|a| a.max_concurrent_jobs()) + .or(app.max_concurrent_jobs) + .unwrap_or(5) +} + +pub fn jobs_enabled(agent: Option<&Agent>, app: &AppConfig) -> bool { + app.function_calling_support && effective_max_concurrent_jobs(agent, app) > 0 +} + fn print_asset_names(kind: &str, names: &[String]) -> Result<()> { if names.is_empty() { println!("No {kind} found."); @@ -4046,6 +4057,11 @@ impl RequestContext { Ok(()) } + #[allow(dead_code)] + pub fn jobs_enabled(&self) -> bool { + jobs_enabled(self.agent.as_ref(), &self.app.config) + } + pub async fn use_agent( &mut self, app: &AppConfig, @@ -4136,11 +4152,20 @@ impl RequestContext { ); } - let should_init_supervisor = agent.can_spawn_agents(); - let max_concurrent = agent.max_concurrent_agents(); + let jobs_enabled = jobs_enabled(Some(&agent), app); + let should_init_supervisor = agent.can_spawn_agents() || jobs_enabled; + let max_concurrent = if agent.can_spawn_agents() { + agent.max_concurrent_agents() + } else { + 0 + }; let max_depth = agent.max_agent_depth(); - let supervisor = should_init_supervisor - .then(|| Arc::new(RwLock::new(Supervisor::new(max_concurrent, max_depth)))); + let max_jobs = effective_max_concurrent_jobs(Some(&agent), app); + let supervisor = should_init_supervisor.then(|| { + Arc::new(RwLock::new( + Supervisor::new(max_concurrent, max_depth).with_max_concurrent_jobs(max_jobs), + )) + }); self.rag = agent.rag(); // Keep `rag_key` in lockstep with `rag`. Agent RAGs are cached under @@ -4152,6 +4177,9 @@ impl RequestContext { .is_some() .then(|| RagKey::Agent(agent.name().to_string())); self.agent = Some(agent); + if let Some(old) = self.supervisor.as_ref() { + old.read().cancel_recursive(); + } self.supervisor = supervisor; self.inbox = None; self.escalation_queue = None; @@ -5217,6 +5245,171 @@ mod tests { assert_eq!(ctx.rag_key, None); } + #[test] + fn effective_max_concurrent_jobs_resolution_precedence() { + let mut app = AppConfig::default(); + assert_eq!(effective_max_concurrent_jobs(None, &app), 5); + + app.max_concurrent_jobs = Some(9); + assert_eq!(effective_max_concurrent_jobs(None, &app), 9); + + let agent = Agent::test_new(AgentConfig { + max_concurrent_jobs: Some(2), + ..AgentConfig::default() + }); + assert_eq!(effective_max_concurrent_jobs(Some(&agent), &app), 2); + } + + #[test] + fn jobs_enabled_requires_function_calling_and_nonzero_capacity() { + let mut app = AppConfig::default(); + assert!(jobs_enabled(None, &app)); + + app.max_concurrent_jobs = Some(0); + assert!(!jobs_enabled(None, &app)); + + app.max_concurrent_jobs = None; + app.function_calling_support = false; + assert!(!jobs_enabled(None, &app)); + + app.function_calling_support = true; + let agent = Agent::test_new(AgentConfig { + max_concurrent_jobs: Some(0), + ..AgentConfig::default() + }); + assert!(!jobs_enabled(Some(&agent), &app)); + } + + #[test] + #[serial] + fn use_agent_cancels_previous_supervisor() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + let agent_name = format!( + "test_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + let old_sig = utils::create_abort_signal(); + + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async { + let join_handle = tokio::spawn(async { + Ok(crate::supervisor::AgentResult { + id: "a1".into(), + agent_name: "explore".into(), + output: String::new(), + exit_status: crate::supervisor::AgentExitStatus::Completed, + }) + }); + let handle = crate::supervisor::AgentHandle { + id: "a1".to_string(), + agent_name: "explore".to_string(), + depth: 1, + inbox: Arc::new(Inbox::new()), + abort_signal: old_sig.clone(), + join_handle, + child_supervisor: None, + }; + let old_sup = Arc::new(RwLock::new(Supervisor::new(4, 3))); + old_sup.write().register(handle).unwrap(); + ctx.supervisor = Some(old_sup); + + ctx.use_agent(&app, &agent_name, None, utils::create_abort_signal()) + .await + .unwrap(); + }); + + assert!(old_sig.aborted()); + assert!(ctx.supervisor.is_some()); + } + + #[test] + #[serial] + fn use_agent_inits_job_capable_supervisor_without_spawning() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + let agent_name = format!( + "test_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async { + ctx.use_agent(&app, &agent_name, None, utils::create_abort_signal()) + .await + .unwrap(); + }); + + let supervisor = ctx.supervisor.as_ref().expect("supervisor for jobs"); + let supervisor = supervisor.read(); + assert_eq!(supervisor.max_concurrent(), 0); + assert_eq!(supervisor.max_concurrent_jobs(), 5); + } + + #[test] + #[serial] + fn use_agent_skips_supervisor_when_jobs_disabled() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let mut app = ctx.app.config.as_ref().clone(); + app.max_concurrent_jobs = Some(0); + let agent_name = format!( + "test_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async { + ctx.use_agent(&app, &agent_name, None, utils::create_abort_signal()) + .await + .unwrap(); + }); + + assert!(ctx.supervisor.is_none()); + } + #[test] fn current_depth_default_is_zero() { let ctx = create_test_ctx(); diff --git a/src/function/jobs.rs b/src/function/jobs.rs new file mode 100644 index 0000000..05ac697 --- /dev/null +++ b/src/function/jobs.rs @@ -0,0 +1,161 @@ +use crate::supervisor::Supervisor; + +use parking_lot::RwLock; +use std::sync::Arc; + +#[allow(dead_code)] +pub fn is_agent_task(supervisor: Option<&Arc>>, id: &str) -> bool { + id.starts_with("agent_") + || id.starts_with("graph_agent_") + || supervisor.is_some_and(|sup| sup.read().has_agent(id)) +} + +pub struct RingBuf { + buf: Vec, + capacity: usize, + write_pos: usize, + total_written: u64, +} + +#[allow(dead_code)] +impl RingBuf { + pub fn new(capacity: usize) -> Self { + Self { + buf: Vec::new(), + capacity, + write_pos: 0, + total_written: 0, + } + } + + pub fn push(&mut self, bytes: &[u8]) { + self.total_written += bytes.len() as u64; + if self.capacity == 0 { + return; + } + let src = if bytes.len() > self.capacity { + &bytes[bytes.len() - self.capacity..] + } else { + bytes + }; + for &byte in src { + if self.buf.len() < self.capacity { + self.buf.push(byte); + } else { + self.buf[self.write_pos] = byte; + } + self.write_pos = (self.write_pos + 1) % self.capacity; + } + } + + pub fn total_written(&self) -> u64 { + self.total_written + } + + pub fn tail(&self) -> Vec { + if self.buf.len() < self.capacity { + return self.buf.clone(); + } + let mut out = Vec::with_capacity(self.capacity); + out.extend_from_slice(&self.buf[self.write_pos..]); + out.extend_from_slice(&self.buf[..self.write_pos]); + out + } +} + +impl Default for RingBuf { + fn default() -> Self { + Self::new(64 * 1024) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ring_buf_returns_contents_below_capacity() { + let mut buf = RingBuf::new(8); + buf.push(b"abc"); + buf.push(b"de"); + assert_eq!(buf.tail(), b"abcde"); + assert_eq!(buf.total_written(), 5); + } + + #[test] + fn ring_buf_exact_fit_keeps_everything() { + let mut buf = RingBuf::new(5); + buf.push(b"abcde"); + assert_eq!(buf.tail(), b"abcde"); + assert_eq!(buf.total_written(), 5); + } + + #[test] + fn ring_buf_wrap_around_keeps_newest_bytes() { + let mut buf = RingBuf::new(5); + buf.push(b"abcde"); + buf.push(b"fg"); + assert_eq!(buf.tail(), b"cdefg"); + assert_eq!(buf.total_written(), 7); + } + + #[test] + fn ring_buf_oversize_push_keeps_last_capacity_bytes() { + let mut buf = RingBuf::new(4); + buf.push(b"abcdefghij"); + assert_eq!(buf.tail(), b"ghij"); + assert_eq!(buf.total_written(), 10); + } + + #[test] + fn ring_buf_default_capacity_is_64_kib() { + let mut buf = RingBuf::default(); + let payload = vec![b'x'; 64 * 1024 + 1]; + buf.push(&payload); + assert_eq!(buf.tail().len(), 64 * 1024); + assert_eq!(buf.total_written(), 64 * 1024 + 1); + } + + #[test] + fn is_agent_task_matches_agent_prefixes() { + assert!(is_agent_task(None, "agent_explore_a1b2c3d4")); + assert!(is_agent_task(None, "graph_agent_explore_a1b2c3d4")); + assert!(!is_agent_task(None, "job_deadbeef")); + } + + #[test] + fn is_agent_task_matches_registered_agents() { + use crate::supervisor::mailbox::Inbox; + use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult}; + use crate::utils::create_abort_signal; + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let join_handle = rt.spawn(async { + Ok(AgentResult { + id: "a1".into(), + agent_name: "explore".into(), + output: String::new(), + exit_status: AgentExitStatus::Completed, + }) + }); + std::mem::forget(rt); + let handle = AgentHandle { + id: "a1".to_string(), + agent_name: "explore".to_string(), + depth: 1, + inbox: Arc::new(Inbox::new()), + abort_signal: create_abort_signal(), + join_handle, + child_supervisor: None, + }; + let mut sup = Supervisor::new(4, 3); + sup.register(handle).unwrap(); + let sup = Arc::new(RwLock::new(sup)); + + assert!(is_agent_task(Some(&sup), "a1")); + assert!(!is_agent_task(Some(&sup), "missing")); + } +} diff --git a/src/function/mod.rs b/src/function/mod.rs index f552854..9d7d3d6 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -1,3 +1,4 @@ +pub(crate) mod jobs; pub(crate) mod memory; pub(crate) mod rag_query; pub(crate) mod skill; diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index 87dfbe5..e69ef6c 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -1,7 +1,8 @@ use super::{FunctionDeclaration, JsonSchema}; use crate::client::{Model, ModelType, call_chat_completions}; use crate::config::{ - Agent, AppState, Input, RequestContext, Role, RoleLike, list_agents_with_descriptions, + Agent, AppState, Input, RequestContext, Role, RoleLike, effective_max_concurrent_jobs, + jobs_enabled, list_agents_with_descriptions, }; use crate::supervisor::mailbox::{Envelope, EnvelopePayload, Inbox}; use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult, Supervisor}; @@ -32,6 +33,19 @@ fn agent_permitted(whitelist: Option<&[String]>, target: &str) -> bool { } } +fn is_job_task(supervisor: Option<&Arc>>, id: &str) -> bool { + id.starts_with("job_") || supervisor.is_some_and(|sup| sup.read().has_job(id)) +} + +fn job_id_teaching_error(id: &str) -> Value { + json!({ + "status": "error", + "message": format!( + "'{id}' is a background job, not an agent — use job__check / job__collect / job__cancel" + ), + }) +} + pub enum GuardrailAction { NoAction, Inject(String), @@ -557,9 +571,15 @@ pub async fn run_agent_for_graph( let agent_mcp_servers = agent.mcp_server_names().to_vec(); let session = agent.agent_session().map(|v| v.to_string()); - let should_init_supervisor = agent.can_spawn_agents(); - let agent_max_concurrent = agent.max_concurrent_agents(); + let child_jobs_enabled = jobs_enabled(Some(&agent), app_config.as_ref()); + let should_init_supervisor = agent.can_spawn_agents() || child_jobs_enabled; + let agent_max_concurrent = if agent.can_spawn_agents() { + agent.max_concurrent_agents() + } else { + 0 + }; let agent_max_depth = agent.max_agent_depth(); + let agent_max_jobs = effective_max_concurrent_jobs(Some(&agent), app_config.as_ref()); let mut child_ctx = RequestContext::new_for_child( Arc::clone(&child_app_state), @@ -571,10 +591,10 @@ pub async fn run_agent_for_graph( child_ctx.rag = agent.rag(); child_ctx.agent = Some(agent); if should_init_supervisor { - child_ctx.supervisor = Some(Arc::new(RwLock::new(Supervisor::new( - agent_max_concurrent, - agent_max_depth, - )))); + child_ctx.supervisor = Some(Arc::new(RwLock::new( + Supervisor::new(agent_max_concurrent, agent_max_depth) + .with_max_concurrent_jobs(agent_max_jobs), + ))); } if let Some(session) = session { @@ -736,9 +756,15 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { let agent_mcp_servers = agent.mcp_server_names().to_vec(); let session = agent.agent_session().map(|v| v.to_string()); - let should_init_supervisor = agent.can_spawn_agents(); - let max_concurrent = agent.max_concurrent_agents(); + let child_jobs_enabled = jobs_enabled(Some(&agent), app_config.as_ref()); + let should_init_supervisor = agent.can_spawn_agents() || child_jobs_enabled; + let max_concurrent = if agent.can_spawn_agents() { + agent.max_concurrent_agents() + } else { + 0 + }; let max_depth = agent.max_agent_depth(); + let max_jobs = effective_max_concurrent_jobs(Some(&agent), app_config.as_ref()); let mut child_ctx = RequestContext::new_for_child( Arc::clone(&child_app_state), ctx, @@ -749,10 +775,9 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { child_ctx.rag = agent.rag(); child_ctx.agent = Some(agent); if should_init_supervisor { - child_ctx.supervisor = Some(Arc::new(RwLock::new(Supervisor::new( - max_concurrent, - max_depth, - )))); + child_ctx.supervisor = Some(Arc::new(RwLock::new( + Supervisor::new(max_concurrent, max_depth).with_max_concurrent_jobs(max_jobs), + ))); } if let Some(session) = session { @@ -861,10 +886,15 @@ async fn handle_check(ctx: &mut RequestContext, args: &Value) -> Result { Ok(result) } - None => Ok(json!({ - "status": "error", - "message": format!("No agent found with id '{id}'") - })), + None => { + if is_job_task(ctx.supervisor.as_ref(), id) { + return Ok(job_id_teaching_error(id)); + } + Ok(json!({ + "status": "error", + "message": format!("No agent found with id '{id}'") + })) + } } } @@ -883,6 +913,9 @@ async fn handle_collect(ctx: &mut RequestContext, args: &Value) -> Result let target_abort = { let sup = supervisor.read(); if sup.is_finished(id).is_none() { + if id.starts_with("job_") || sup.has_job(id) { + return Ok(job_id_teaching_error(id)); + } return Ok(json!({ "status": "error", "message": format!("Agent '{id}' not found. Use agent__check to verify it exists and is finished.") @@ -1065,10 +1098,15 @@ async fn handle_cancel(ctx: &mut RequestContext, args: &Value) -> Result "message": message, })) } - None => Ok(json!({ - "status": "error", - "message": format!("No agent found with id '{id}'"), - })), + None => { + if is_job_task(ctx.supervisor.as_ref(), id) { + return Ok(job_id_teaching_error(id)); + } + Ok(json!({ + "status": "error", + "message": format!("No agent found with id '{id}'"), + })) + } } } @@ -1115,10 +1153,17 @@ fn handle_send_message(ctx: &mut RequestContext, args: &Value) -> Result "message": format!("Message delivered to agent '{id}'"), })) } - None => Ok(json!({ - "status": "error", - "message": format!("No agent found with id '{id}'. Agent may not exist or may have already completed."), - })), + None => { + if is_job_task(ctx.supervisor.as_ref(), id) + || is_job_task(ctx.parent_supervisor.as_ref(), id) + { + return Ok(job_id_teaching_error(id)); + } + Ok(json!({ + "status": "error", + "message": format!("No agent found with id '{id}'. Agent may not exist or may have already completed."), + })) + } } } @@ -1455,7 +1500,10 @@ mod tests { use super::*; use crate::config::test_fixtures::{FixtureServer, fixture_runtime}; use crate::config::{AgentConfig, AppState, WorkingMode}; + use crate::function::jobs::RingBuf; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; + use crate::supervisor::{JobHandle, JobResult, JobState, JobStatus}; + use parking_lot::Mutex; use serde_json::json; use serial_test::serial; @@ -1472,6 +1520,59 @@ mod tests { ctx } + fn ctx_with_job_capable_supervisor() -> RequestContext { + let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); + ctx.supervisor = Some(Arc::new(RwLock::new( + Supervisor::new(4, 3).with_max_concurrent_jobs(4), + ))); + ctx + } + + fn make_fake_job(id: &str) -> JobHandle { + let rt = tokio::runtime::Runtime::new().unwrap(); + let join_handle = rt.spawn(async { + Ok(JobResult { + output: json!(null), + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + JobHandle { + id: id.to_string(), + tool: "execute_command".to_string(), + started_at: std::time::Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + } + } + + fn register_fake_job(ctx: &mut RequestContext, id: &str) { + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_fake_job(id)) + .unwrap(); + } + + fn assert_job_teaching_error(result: &Value, id: &str) { + assert_eq!(result["status"], "error"); + let message = result["message"].as_str().unwrap(); + assert_eq!( + message, + format!( + "'{id}' is a background job, not an agent — use job__check / job__collect / job__cancel" + ) + ); + } + fn register_fake_agent(ctx: &mut RequestContext, id: &str, name: &str) { register_fake_agent_with_output(ctx, id, name, "fake output"); } @@ -2533,4 +2634,101 @@ mod tests { assert_eq!(ctx.pending_agents_guardrail_count, 0); }); } + + #[test] + fn handle_check_registered_job_id_teaches_job_tools() { + run_async(async { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "bg_1"); + + let result = handle_check(&mut ctx, &json!({"id": "bg_1"})) + .await + .unwrap(); + + assert_job_teaching_error(&result, "bg_1"); + }); + } + + #[test] + fn handle_check_job_prefixed_id_teaches_job_tools() { + run_async(async { + let mut ctx = ctx_with_supervisor(4, 3); + + let result = handle_check(&mut ctx, &json!({"id": "job_deadbeef"})) + .await + .unwrap(); + + assert_job_teaching_error(&result, "job_deadbeef"); + }); + } + + #[test] + fn handle_collect_registered_job_id_teaches_job_tools() { + run_async(async { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "bg_1"); + + let result = handle_collect(&mut ctx, &json!({"id": "bg_1"})) + .await + .unwrap(); + + assert_job_teaching_error(&result, "bg_1"); + }); + } + + #[test] + fn handle_collect_job_prefixed_id_teaches_job_tools() { + run_async(async { + let mut ctx = ctx_with_supervisor(4, 3); + + let result = handle_collect(&mut ctx, &json!({"id": "job_deadbeef"})) + .await + .unwrap(); + + assert_job_teaching_error(&result, "job_deadbeef"); + }); + } + + #[test] + fn handle_cancel_registered_job_id_teaches_job_tools_and_keeps_job() { + run_async(async { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "bg_1"); + + let result = handle_cancel(&mut ctx, &json!({"id": "bg_1"})) + .await + .unwrap(); + + assert_job_teaching_error(&result, "bg_1"); + assert!(ctx.supervisor.as_ref().unwrap().read().has_job("bg_1")); + }); + } + + #[test] + fn handle_send_message_registered_job_id_teaches_job_tools() { + run_async(async { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "bg_1"); + + let result = + handle_send_message(&mut ctx, &json!({"id": "bg_1", "message": "hi"})).unwrap(); + + assert_job_teaching_error(&result, "bg_1"); + }); + } + + #[test] + fn handle_send_message_job_in_parent_supervisor_teaches_job_tools() { + run_async(async { + let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); + let mut parent_sup = Supervisor::new(4, 3).with_max_concurrent_jobs(4); + parent_sup.register(make_fake_job("bg_p")).unwrap(); + ctx.parent_supervisor = Some(Arc::new(RwLock::new(parent_sup))); + + let result = + handle_send_message(&mut ctx, &json!({"id": "bg_p", "message": "hi"})).unwrap(); + + assert_job_teaching_error(&result, "bg_p"); + }); + } } diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index 0739404..b63f608 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -2,16 +2,19 @@ pub mod escalation; pub mod mailbox; pub mod taskqueue; +use crate::function::jobs::RingBuf; use crate::utils::AbortSignal; use fmt::{Debug, Formatter}; use mailbox::Inbox; -use parking_lot::RwLock; +use parking_lot::{Mutex, RwLock}; use taskqueue::TaskQueue; use anyhow::{Result, bail}; +use serde_json::Value; use std::collections::HashMap; use std::fmt; use std::sync::Arc; +use std::time::Instant; use tokio::task::JoinHandle; #[derive(Debug, Clone, PartialEq, Eq)] @@ -37,11 +40,85 @@ pub struct AgentHandle { pub child_supervisor: Option>>, } +#[allow(dead_code)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum JobStatus { + Running, + Completed, + Failed, +} + +pub struct JobState { + #[allow(dead_code)] + pub status: JobStatus, + pub pgid: Option, +} + +#[allow(dead_code)] +pub struct JobResult { + pub output: Value, + pub exit_code: Option, + pub output_bytes_captured: u64, +} + +pub struct JobHandle { + pub id: String, + #[allow(dead_code)] + pub tool: String, + #[allow(dead_code)] + pub started_at: Instant, + pub join_handle: JoinHandle>, + pub abort_signal: AbortSignal, + pub state: Arc>, + #[allow(dead_code)] + pub output_buf: Arc>, + #[allow(dead_code)] + pub no_change_checks: u32, +} + +impl JobHandle { + // pgid == child pid under process_group(0); after wait() reaps the child + // the pid can be recycled, so never kill unless pgid is still set. + fn kill_process_group(&self) { + #[cfg(unix)] + if let Some(pgid) = self.state.lock().pgid { + unsafe { + libc::killpg(pgid, libc::SIGTERM); + } + } + } +} + +impl Drop for JobHandle { + fn drop(&mut self) { + self.kill_process_group(); + self.join_handle.abort(); + } +} + +pub enum TaskHandle { + Agent(AgentHandle), + Job(JobHandle), +} + +impl From for TaskHandle { + fn from(handle: AgentHandle) -> Self { + Self::Agent(handle) + } +} + +impl From for TaskHandle { + fn from(handle: JobHandle) -> Self { + Self::Job(handle) + } +} + pub struct Supervisor { - handles: HashMap, + handles: HashMap, task_queue: TaskQueue, max_concurrent: usize, max_depth: usize, + max_concurrent_jobs: usize, } impl Supervisor { @@ -51,17 +128,43 @@ impl Supervisor { task_queue: TaskQueue::new(), max_concurrent, max_depth, + max_concurrent_jobs: 0, } } + pub fn with_max_concurrent_jobs(mut self, max_concurrent_jobs: usize) -> Self { + self.max_concurrent_jobs = max_concurrent_jobs; + self + } + + fn agent(&self, id: &str) -> Option<&AgentHandle> { + match self.handles.get(id) { + Some(TaskHandle::Agent(handle)) => Some(handle), + _ => None, + } + } + + fn agents(&self) -> impl Iterator { + self.handles.values().filter_map(|handle| match handle { + TaskHandle::Agent(handle) => Some(handle), + TaskHandle::Job(_) => None, + }) + } + pub fn active_count(&self) -> usize { - self.handles.len() + self.agents().count() } pub fn effective_active_count(&self) -> usize { + self.agents() + .filter(|h| !h.join_handle.is_finished()) + .count() + } + + pub fn active_job_count(&self) -> usize { self.handles .values() - .filter(|h| !h.join_handle.is_finished()) + .filter(|h| matches!(h, TaskHandle::Job(job) if !job.join_handle.is_finished())) .count() } @@ -73,6 +176,11 @@ impl Supervisor { self.max_depth } + #[allow(dead_code)] + pub fn max_concurrent_jobs(&self) -> usize { + self.max_concurrent_jobs + } + pub fn task_queue(&self) -> &TaskQueue { &self.task_queue } @@ -81,59 +189,111 @@ impl Supervisor { &mut self.task_queue } - pub fn register(&mut self, handle: AgentHandle) -> Result<()> { - if self.effective_active_count() >= self.max_concurrent { - bail!( - "Cannot spawn agent: at capacity ({}/{})", - self.effective_active_count(), - self.max_concurrent - ); + pub fn register(&mut self, handle: impl Into) -> Result<()> { + match handle.into() { + TaskHandle::Agent(handle) => { + if self.effective_active_count() >= self.max_concurrent { + bail!( + "Cannot spawn agent: at capacity ({}/{})", + self.effective_active_count(), + self.max_concurrent + ); + } + if handle.depth > self.max_depth { + bail!( + "Cannot spawn agent: max depth exceeded ({}/{})", + handle.depth, + self.max_depth + ); + } + self.handles + .insert(handle.id.clone(), TaskHandle::Agent(handle)); + } + TaskHandle::Job(handle) => { + if self.active_job_count() >= self.max_concurrent_jobs { + bail!( + "Cannot start job: at capacity ({}/{})", + self.active_job_count(), + self.max_concurrent_jobs + ); + } + self.handles + .insert(handle.id.clone(), TaskHandle::Job(handle)); + } } - if handle.depth > self.max_depth { - bail!( - "Cannot spawn agent: max depth exceeded ({}/{})", - handle.depth, - self.max_depth - ); - } - self.handles.insert(handle.id.clone(), handle); Ok(()) } pub fn is_finished(&self, id: &str) -> Option { - self.handles.get(id).map(|h| h.join_handle.is_finished()) + self.agent(id).map(|h| h.join_handle.is_finished()) } pub fn take(&mut self, id: &str) -> Option { - self.handles.remove(id) + self.agent(id)?; + match self.handles.remove(id) { + Some(TaskHandle::Agent(handle)) => Some(handle), + _ => None, + } + } + + #[allow(dead_code)] + pub fn take_job(&mut self, id: &str) -> Option { + if !self.has_job(id) { + return None; + } + match self.handles.remove(id) { + Some(TaskHandle::Job(handle)) => Some(handle), + _ => None, + } + } + + pub fn has_job(&self, id: &str) -> bool { + matches!(self.handles.get(id), Some(TaskHandle::Job(_))) + } + + pub fn has_agent(&self, id: &str) -> bool { + self.agent(id).is_some() } pub fn inbox(&self, id: &str) -> Option<&Arc> { - self.handles.get(id).map(|h| &h.inbox) + self.agent(id).map(|h| &h.inbox) } pub fn abort_signal_for(&self, id: &str) -> Option { - self.handles.get(id).map(|h| h.abort_signal.clone()) + self.agent(id).map(|h| h.abort_signal.clone()) } pub fn list_agents(&self) -> Vec<(&str, &str)> { - self.handles - .values() + self.agents() .map(|h| (h.id.as_str(), h.agent_name.as_str())) .collect() } pub fn cancel_all(&self) { for handle in self.handles.values() { - handle.abort_signal.set_ctrlc(); + match handle { + TaskHandle::Agent(agent) => agent.abort_signal.set_ctrlc(), + TaskHandle::Job(job) => { + job.abort_signal.set_ctrlc(); + job.kill_process_group(); + } + } } } pub fn cancel_recursive(&self) { for handle in self.handles.values() { - handle.abort_signal.set_ctrlc(); - if let Some(child_sup) = handle.child_supervisor.as_ref() { - child_sup.read().cancel_recursive(); + match handle { + TaskHandle::Agent(agent) => { + agent.abort_signal.set_ctrlc(); + if let Some(child_sup) = agent.child_supervisor.as_ref() { + child_sup.read().cancel_recursive(); + } + } + TaskHandle::Job(job) => { + job.abort_signal.set_ctrlc(); + job.kill_process_group(); + } } } } @@ -142,7 +302,7 @@ impl Supervisor { impl Debug for Supervisor { fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_struct("Supervisor") - .field("active_agents", &self.handles.len()) + .field("active_agents", &self.active_count()) .field("max_concurrent", &self.max_concurrent) .field("max_depth", &self.max_depth) .finish() @@ -177,6 +337,33 @@ mod tests { } } + fn make_job(id: &str, abort_signal: AbortSignal) -> JobHandle { + // Keep the runtime alive so the spawned task is never polled and the + // job counts as running for capacity checks. + let rt = Builder::new_current_thread().enable_all().build().unwrap(); + let join_handle = rt.spawn(async { + Ok(JobResult { + output: Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + JobHandle { + id: id.to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal, + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + } + } + #[test] fn supervisor_new_empty() { let sup = Supervisor::new(4, 3); @@ -315,4 +502,82 @@ mod tests { assert!(parent_sig.aborted()); assert!(child_sig.aborted()); } + + #[test] + fn job_registration_rejects_when_job_capacity_zero() { + let mut sup = Supervisor::new(4, 3); + let result = sup.register(make_job("j1", create_abort_signal())); + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("at capacity")); + } + + #[test] + fn job_registration_rejects_at_job_capacity() { + let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(1); + sup.register(make_job("j1", create_abort_signal())).unwrap(); + let result = sup.register(make_job("j2", create_abort_signal())); + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("at capacity")); + } + + #[test] + fn job_capacity_is_independent_of_agent_capacity() { + let mut sup = Supervisor::new(1, 3).with_max_concurrent_jobs(1); + sup.register(make_job("j1", create_abort_signal())).unwrap(); + sup.register(make_handle("a1", "explore", 1)).unwrap(); + assert_eq!(sup.active_job_count(), 1); + assert_eq!(sup.active_count(), 1); + assert_eq!(sup.max_concurrent_jobs(), 1); + } + + #[test] + fn agent_accessors_ignore_jobs() { + let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(2); + sup.register(make_job("j1", create_abort_signal())).unwrap(); + + assert_eq!(sup.active_count(), 0); + assert_eq!(sup.effective_active_count(), 0); + assert!(sup.list_agents().is_empty()); + assert_eq!(sup.is_finished("j1"), None); + assert!(sup.inbox("j1").is_none()); + assert!(sup.abort_signal_for("j1").is_none()); + assert!(sup.take("j1").is_none()); + assert!(sup.has_job("j1")); + assert!(!sup.has_agent("j1")); + assert_eq!(sup.active_job_count(), 1); + } + + #[test] + fn take_job_removes_job_but_not_agents() { + let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(2); + sup.register(make_job("j1", create_abort_signal())).unwrap(); + sup.register(make_handle("a1", "explore", 1)).unwrap(); + + assert!(sup.take_job("a1").is_none()); + assert!(sup.has_agent("a1")); + assert!(sup.take_job("j1").is_some()); + assert_eq!(sup.active_job_count(), 0); + } + + #[test] + fn cancel_recursive_aborts_jobs() { + let sig = create_abort_signal(); + let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(1); + sup.register(make_job("j1", sig.clone())).unwrap(); + + sup.cancel_recursive(); + + assert!(sig.aborted()); + } + + #[test] + fn cancel_all_aborts_jobs() { + let sig = create_abort_signal(); + let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(1); + sup.register(make_job("j1", sig.clone())).unwrap(); + + sup.cancel_all(); + + assert!(sig.aborted()); + } } From a9df9a4dd5b7afab4f6d9c34484563b4dbe8de63 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 17:35:47 -0600 Subject: [PATCH 04/28] docs(plan): reconcile whitelist row 1 with the grep-class carve-out row T3 implementation followed the specific fs_*/ast_grep 'NO in v1' row; row 1's 'ALL external command tools' over-claimed. No design change. --- plans/background-jobs-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plans/background-jobs-design.md b/plans/background-jobs-design.md index c4a52fc..52bb456 100644 --- a/plans/background-jobs-design.md +++ b/plans/background-jobs-design.md @@ -549,7 +549,7 @@ execution can run from an owned snapshot: | Class | Backgroundable | Execution path in job task | |---|---|---| -| `execute_command` + ALL external command tools — bash (argc), JavaScript/TypeScript, Python (user-ruled 2026-08-21; no per-tool opt-in flag) | YES (the whole point) | Extract the spawn logic out of the eval path into a free function taking `(Arc, JobEnvSnapshot, args)` — do NOT route through `ToolCall::eval`. `JobEnvSnapshot` = the audited field set below. | +| `execute_command` + external command tools — bash (argc), JavaScript/TypeScript, Python (user-ruled 2026-08-21; no per-tool opt-in flag) — EXCEPT the fast grep-class carve-out row below | YES (the whole point) | Extract the spawn logic out of the eval path into a free function taking `(Arc, JobEnvSnapshot, args)` — do NOT route through `ToolCall::eval`. `JobEnvSnapshot` = the audited field set below. | | `mcp_invoke_*` | YES | Already `&ctx` (`eval_mcp`); job task owns `JobCtx { mcp_runtime: McpRuntime, current_depth: usize }` — the runtime is a SINGLE-ENTRY snapshot holding only the validated server's `Arc` (§3 hardening rule 4, Oracle N3). Job MCP path = `invoke` → `render_tool_result` (mod.rs:1635 foreground parity; FREE function, see §13.1). AUDITED (§13.1): that is eval_mcp's COMPLETE transitive ctx surface. McpRuntime is `#[derive(Clone)]`, shallow Arc map (src/config/tool_scope.rs:44-47); OAuth refresh is transport-embedded (auth_client.rs) and needs no AppState. NOTE: MCP jobs have NO timeout (`COYOTE_TOOL_TIMEOUT` is process-path only); a hung MCP job is recoverable via `job__cancel` (abort drops the future) — accepted v1, stated in the tool description. | | `mcp_search_/mcp_describe_/mcp_read_/mcp_prompt_` | NO (pointless — fast) | Teaching error: "sub-second call; invoke directly." (All four non-invoke meta-families; meta-declarations are capability-gated per server — `gated_meta_function_prefixes`, function/mod.rs:416-426: invoke⇔tools, read⇔resources, prompt⇔prompts — which composes with the declared-names stash automatically: a capability the server never advertised is never declared, so `job__start` rejects it with the standard not-declared error.) | | `agent__*`, `job__*` | NO | "Already asynchronous — use them directly." | From 7cf88c030f61972ee9634fd9d699f72c7469c9bf Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 15:45:07 -0600 Subject: [PATCH 05/28] fix(supervisor): surface finished-but-uncollected tasks in turn-end guardrail The turn-end guardrail only counted still-running agents, so an agent that finished before the turn ended was invisible: its uncollected result was silently dropped. Jobs were never counted at all. The guardrail now enumerates every registered task (running and finished, agents and jobs) via Supervisor::list_tasks and renders a kind-aware prompt with two sections: still-running tasks to reclaim, and completed-but-uncollected tasks with the exact collect command. At the force-terminate cap, finished-but-uncollected handles are explicitly discarded with a warning naming the lost ids, so the guardrail cannot loop forever on handles nobody will collect. --- src/function/supervisor.rs | 215 +++++++++++++++++++++++++++++++------ src/supervisor/mod.rs | 25 ++++- 2 files changed, 207 insertions(+), 33 deletions(-) diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index e69ef6c..ad09808 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -5,14 +5,14 @@ use crate::config::{ jobs_enabled, list_agents_with_descriptions, }; use crate::supervisor::mailbox::{Envelope, EnvelopePayload, Inbox}; -use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult, Supervisor}; +use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult, Supervisor, TaskKind}; use crate::utils::{AbortSignal, create_abort_signal, wait_abort_signal}; use crate::graph; use anyhow::{Context, Result, anyhow, bail}; use chrono::Utc; use indexmap::IndexMap; -use log::debug; +use log::{debug, warn}; use parking_lot::RwLock; use serde_json::{Value, json}; use std::pin::Pin; @@ -52,38 +52,87 @@ pub enum GuardrailAction { ForceTerminate(Vec), } -pub fn pending_agent_ids(ctx: &RequestContext) -> Vec { +pub struct PendingTask { + pub id: String, + pub kind: TaskKind, + pub finished: bool, +} + +pub fn pending_tasks(ctx: &RequestContext) -> Vec { let Some(sup) = ctx.supervisor.as_ref() else { return Vec::new(); }; - let sup = sup.read(); - sup.list_agents() + let mut tasks: Vec = sup + .read() + .list_tasks() .into_iter() - .filter_map(|(id, _)| match sup.is_finished(id) { - Some(false) => Some(id.to_string()), - _ => None, + .map(|(id, kind, finished)| PendingTask { + id: id.to_string(), + kind, + finished, }) - .collect() + .collect(); + tasks.sort_by(|a, b| a.id.cmp(&b.id)); + tasks } -pub fn build_pending_agents_guardrail_prompt(ids: &[String]) -> String { - let count = ids.len(); - let id_list = ids - .iter() - .map(|id| format!("- {id}")) - .collect::>() - .join("\n"); +pub fn build_pending_agents_guardrail_prompt(tasks: &[PendingTask]) -> String { + let running: Vec<&PendingTask> = tasks.iter().filter(|t| !t.finished).collect(); + let finished: Vec<&PendingTask> = tasks.iter().filter(|t| t.finished).collect(); + + let mut sections = Vec::new(); + if !running.is_empty() { + let id_list = running + .iter() + .map(|t| { + let (kind, collect, cancel) = match t.kind { + TaskKind::Agent => ("agent", "agent__collect", "agent__cancel"), + TaskKind::Job => ("job", "job__collect", "job__cancel"), + }; + format!( + "- {id} ({kind}): call `{collect}` (blocks until done, returns output) or \ + `{cancel}` (discards)", + id = t.id + ) + }) + .collect::>() + .join("\n"); + sections.push(format!( + "Still running ({count}):\n{id_list}\n\nThese will be abandoned if your turn ends \ + now. You MUST reclaim each one before ending your turn. Do NOT emit a text-only \ + response expecting them to 'report back' — they will not.", + count = running.len() + )); + } + if !finished.is_empty() { + let cmd_list = finished + .iter() + .map(|t| { + let collect = match t.kind { + TaskKind::Agent => "agent__collect", + TaskKind::Job => "job__collect", + }; + format!("- `{collect} --id {id}`", id = t.id) + }) + .collect::>() + .join("\n"); + sections.push(format!( + "Completed but UNCOLLECTED — collect NOW ({count}):\n{cmd_list}\n\nCollect returns \ + instantly on a finished task. Their results are LOST if your turn ends without \ + collecting.", + count = finished.len() + )); + } format!( - "[SYSTEM GUARDRAIL] You attempted to end your turn while {count} spawned background agent(s) \ - are still running:\n{id_list}\n\nThese agents will be abandoned if your turn ends now. You MUST \ - reclaim each one before ending your turn. For each agent: call `agent__collect` (blocks until \ - done, returns output) or `agent__cancel` (discards). Do NOT emit a text-only response \ - expecting them to 'report back' — they will not." + "[SYSTEM GUARDRAIL] You attempted to end your turn with {count} unreclaimed background \ + task(s).\n\n{body}", + count = tasks.len(), + body = sections.join("\n\n") ) } pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailAction { - let pending = pending_agent_ids(ctx); + let pending = pending_tasks(ctx); if pending.is_empty() { ctx.pending_agents_guardrail_count = 0; return GuardrailAction::NoAction; @@ -92,10 +141,29 @@ pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailActi if ctx.pending_agents_guardrail_count >= PENDING_AGENTS_GUARDRAIL_MAX { if let Some(sup) = ctx.supervisor.as_ref().cloned() { sup.read().cancel_recursive(); + let finished: Vec<&PendingTask> = pending.iter().filter(|t| t.finished).collect(); + if !finished.is_empty() { + let ids: Vec<&str> = finished.iter().map(|t| t.id.as_str()).collect(); + warn!( + "Turn-end guardrail: discarding uncollected result(s) for finished task(s) \ + after max reminders: {ids:?}" + ); + let mut sup = sup.write(); + for task in &finished { + match task.kind { + TaskKind::Agent => { + let _ = sup.take(&task.id); + } + TaskKind::Job => { + let _ = sup.take_job(&task.id); + } + } + } + } } ctx.pending_agents_guardrail_count = 0; - return GuardrailAction::ForceTerminate(pending); + return GuardrailAction::ForceTerminate(pending.into_iter().map(|t| t.id).collect()); } ctx.pending_agents_guardrail_count += 1; @@ -2396,27 +2464,110 @@ mod tests { assert_eq!(ctx.pending_agents_guardrail_count, 0); } - /// Pins current behavior: a finished-but-uncollected agent is not counted - /// as pending (only still-running agents are), so the turn-end guardrail - /// takes no action and the finished agent's result can be silently dropped. + /// A finished-but-uncollected agent counts as pending: the turn-end + /// guardrail tells the model to collect it instead of letting the result + /// be silently dropped, and the handle stays registered. #[test] - fn guardrail_ignores_finished_but_uncollected_agents() { + fn guardrail_surfaces_finished_but_uncollected_agents() { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); wait_until_finished(&ctx, "a1"); ctx.pending_agents_guardrail_count = 2; - assert!(matches!( - check_pending_agents_guardrail(&mut ctx), - GuardrailAction::NoAction - )); - assert_eq!(ctx.pending_agents_guardrail_count, 0); + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::Inject(prompt) => { + assert!(prompt.contains("a1")); + assert!(prompt.contains("agent__collect --id a1")); + assert!(prompt.contains("Completed but UNCOLLECTED")); + } + _ => panic!("expected Inject action"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 3); assert_eq!( ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), Some(true) ); } + #[test] + fn guardrail_force_terminate_discards_finished_uncollected_handles() { + let mut ctx = ctx_with_supervisor(4, 3); + register_fake_agent(&mut ctx, "a1", "explore"); + wait_until_finished(&ctx, "a1"); + ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::ForceTerminate(ids) => { + assert_eq!(ids, vec!["a1".to_string()]); + } + _ => panic!("expected ForceTerminate action"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), + None + ); + } + + #[test] + fn guardrail_prompt_renders_running_and_finished_sections() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_supervisor(4, 3); + let _abort = register_running_agent(&mut ctx, "slow", "test"); + register_fake_agent(&mut ctx, "a1", "explore"); + wait_until_finished(&ctx, "a1"); + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::Inject(prompt) => { + assert!(prompt.contains("Still running")); + assert!(prompt.contains("slow (agent)")); + assert!(prompt.contains("Completed but UNCOLLECTED")); + assert!(prompt.contains("agent__collect --id a1")); + } + _ => panic!("expected Inject action"), + } + }); + } + + #[test] + fn guardrail_prompt_is_kind_aware_for_jobs() { + let tasks = vec![ + PendingTask { + id: "job_1".into(), + kind: TaskKind::Job, + finished: false, + }, + PendingTask { + id: "job_2".into(), + kind: TaskKind::Job, + finished: true, + }, + ]; + + let prompt = build_pending_agents_guardrail_prompt(&tasks); + + assert!(prompt.contains("job_1 (job)")); + assert!(prompt.contains("job__cancel")); + assert!(prompt.contains("job__collect --id job_2")); + } + + #[test] + fn pending_tasks_includes_registered_jobs() { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "job_1"); + + let tasks = pending_tasks(&ctx); + + assert_eq!(tasks.len(), 1); + assert_eq!(tasks[0].id, "job_1"); + assert_eq!(tasks[0].kind, TaskKind::Job); + } + #[test] fn guardrail_force_terminates_at_max_and_cancels_agents() { let rt = tokio::runtime::Builder::new_current_thread() diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index b63f608..dfa072d 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -101,6 +101,12 @@ pub enum TaskHandle { Job(JobHandle), } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum TaskKind { + Agent, + Job, +} + impl From for TaskHandle { fn from(handle: AgentHandle) -> Self { Self::Agent(handle) @@ -236,7 +242,6 @@ impl Supervisor { } } - #[allow(dead_code)] pub fn take_job(&mut self, id: &str) -> Option { if !self.has_job(id) { return None; @@ -269,6 +274,24 @@ impl Supervisor { .collect() } + pub fn list_tasks(&self) -> Vec<(&str, TaskKind, bool)> { + self.handles + .values() + .map(|handle| match handle { + TaskHandle::Agent(agent) => ( + agent.id.as_str(), + TaskKind::Agent, + agent.join_handle.is_finished(), + ), + TaskHandle::Job(job) => ( + job.id.as_str(), + TaskKind::Job, + job.join_handle.is_finished(), + ), + }) + .collect() + } + pub fn cancel_all(&self) { for handle in self.handles.values() { match handle { From 257b06bbd4f7f43e2b6f3731248c1a4444095301 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 15:46:26 -0600 Subject: [PATCH 06/28] fix(supervisor): make agent__check a pure status probe that never consumes the handle agent__check on a finished agent delegated to agent__collect, which returned the full (unbounded) result and consumed the handle. That contradicted the tool's own docs and broke the check-then-collect pattern: a second collect on the same id failed. check now reports { status: finished } with a pointer to agent__collect and leaves the handle registered; collect is the single retrieval verb. The tool description and prompt table are updated to stop promising that check returns the result. --- src/config/prompts.rs | 2 +- src/function/supervisor.rs | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/src/config/prompts.rs b/src/config/prompts.rs index 8af226b..cb2e163 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -82,7 +82,7 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {" | Tool | Purpose | |------|----------| | `agent__spawn` | Spawn a subagent in the background. Returns an `id` immediately. | - | `agent__check` | Non-blocking check: is the agent done yet? Returns PENDING or result. | + | `agent__check` | Non-blocking status probe: running or finished. Never returns/consumes the result — use `agent__collect`. | | `agent__collect` | Blocking wait: wait for an agent to finish, return its output. | | `agent__list_available` | List all agent types you can spawn (name + description). Use this to discover specialists before calling `agent__spawn`. | | `agent__list_running` | List all subagents YOU have spawned, with their status. | diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index ad09808..2e766e2 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -255,7 +255,7 @@ pub fn supervisor_function_declarations() -> Vec { }, FunctionDeclaration { name: format!("{SUPERVISOR_FUNCTION_PREFIX}check"), - description: "Check if a spawned agent has finished. Non-blocking; returns PENDING if still running, or the result if complete.".to_string(), + description: "Non-blocking status probe: reports whether a spawned agent is still running or finished. NEVER returns or consumes the result — when finished, call agent__collect to retrieve it.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([( @@ -934,7 +934,15 @@ async fn handle_check(ctx: &mut RequestContext, args: &Value) -> Result { }; match is_finished { - Some(true) => handle_collect(ctx, args).await, + Some(true) => Ok(json!({ + "status": "finished", + "id": id, + "message": format!( + "Agent '{id}' has finished; its result is ready and has NOT been consumed. \ + Call `agent__collect --id {id}` to retrieve it (returns instantly on a \ + finished agent). The handle stays registered until collected." + ), + })), Some(false) => { let mut result = json!({ "status": "pending", @@ -2711,23 +2719,34 @@ mod tests { assert!(err.to_string().contains("No supervisor active")); } - /// Pins current behavior: checking a finished agent does not report a - /// "finished, ready to collect" status; it silently delegates to collect, - /// returning the full result and consuming the handle. + /// Checking a finished agent is a pure status probe: it reports the + /// agent as finished, points at agent__collect, and leaves the handle + /// registered so a subsequent collect still returns the result. #[test] - fn handle_check_finished_agent_delegates_to_collect_and_consumes_handle() { + fn handle_check_finished_agent_reports_status_and_keeps_handle() { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); wait_until_finished(&ctx, "a1"); let result = run_async(handle_check(&mut ctx, &json!({"id": "a1"}))).unwrap(); - assert_eq!(result["status"], "completed"); - assert_eq!(result["output"], "fake output"); + assert_eq!(result["status"], "finished"); + assert_eq!(result["id"], "a1"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("agent__collect") + ); assert_eq!( ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), - None + Some(true) ); + + let collected = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap(); + + assert_eq!(collected["status"], "completed"); + assert_eq!(collected["output"], "fake output"); } #[test] From fcc3756634163f98e10ff9d5e15b5bdc3cecda43 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 15:47:34 -0600 Subject: [PATCH 07/28] fix(function): floor tool-output truncation cut to a UTF-8 char boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When max_chars landed inside a multi-byte UTF-8 character of the serialized output, s.get(..max_chars) returned None and the code fell back to the FULL untruncated string while still prepending the truncation marker — the "truncated" output actually grew. The cut is now floored to the previous char boundary so the prefix is always a valid, genuinely truncated slice. --- src/function/mod.rs | 53 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 9 deletions(-) diff --git a/src/function/mod.rs b/src/function/mod.rs index 9d7d3d6..579c430 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -405,7 +405,11 @@ impl ToolResult { pub fn truncate_if_needed(mut self, max_chars: usize) -> Self { let s = self.output.to_string(); if s.len() > max_chars { - let prefix = s.get(..max_chars).unwrap_or(s.as_str()); + let mut cut = max_chars; + while !s.is_char_boundary(cut) { + cut -= 1; + } + let prefix = &s[..cut]; self.output = json!(format!( "[truncated: tool output exceeded {max_chars} chars]\n{prefix}" )); @@ -4179,23 +4183,54 @@ mod tests { assert!(!results[0].output.to_string().contains("[truncated")); } - /// Pins current behavior: when the char cap lands inside a multi-byte - /// UTF-8 character of the serialized output, no prefix can be taken, so - /// the truncation marker is prepended to the FULL original output and the - /// "truncated" result is longer than the input. + /// When the char cap lands inside a multi-byte UTF-8 character of the + /// serialized output, the cut is floored to the previous char boundary + /// so the output actually shrinks. #[test] - fn truncate_if_needed_utf8_boundary_returns_full_output_with_marker() { + fn truncate_if_needed_floors_cut_to_char_boundary() { let serialized = json!("aé").to_string(); let result = ToolResult::new(call("t", Some("id-1")), json!("aé")); let truncated = result.truncate_if_needed(3); let out = truncated.output.as_str().unwrap(); + assert_eq!(out, "[truncated: tool output exceeded 3 chars]\n\"a"); + assert!(out.len() < "[truncated: tool output exceeded 3 chars]\n".len() + serialized.len()); + } + + #[test] + fn truncate_if_needed_cap_on_char_boundary_truncates_normally() { + let result = ToolResult::new(call("t", Some("id-1")), json!("aé")); + + let truncated = result.truncate_if_needed(2); + assert_eq!( - out, - format!("[truncated: tool output exceeded 3 chars]\n{serialized}") + truncated.output.as_str().unwrap(), + "[truncated: tool output exceeded 2 chars]\n\"a" ); - assert!(out.len() > serialized.len()); + } + + #[test] + fn truncate_if_needed_cap_zero_yields_marker_and_empty_prefix() { + let result = ToolResult::new(call("t", Some("id-1")), json!("aé")); + + let truncated = result.truncate_if_needed(0); + + assert_eq!( + truncated.output.as_str().unwrap(), + "[truncated: tool output exceeded 0 chars]\n" + ); + } + + #[test] + fn truncate_if_needed_cap_at_or_above_length_leaves_output_unchanged() { + for cap in [5, 100] { + let result = ToolResult::new(call("t", Some("id-1")), json!("aé")); + + let truncated = result.truncate_if_needed(cap); + + assert_eq!(truncated.output, json!("aé")); + } } #[test] From cb025b7fffcc0e1e70ee5af69ad9b4c0c4dcaa84 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 16:54:44 -0600 Subject: [PATCH 08/28] feat: add background job runner, job__* handlers, and start gates Detached tokio::process runner with a frozen JobEnvSnapshot (env-derived bin dirs, vault-interpolated agent envs, COYOTE_TOOL_TIMEOUT resolved at start), process_group(0) with pgid-guarded SIGTERM/SIGKILL escalation, capture-only ring-buffer telemetry, and LLM_OUTPUT read after wait(). MCP jobs snapshot a single-entry McpRuntime holding only the validated server and render through the same free fn as the foreground path. job__start enforces its gates synchronously before any spawn: jobs_enabled, the backgroundable whitelist with directionality teaching errors, the per-request declared-names stash captured in before_chat_completion, then capacity (lazy supervisor get-or-init in plain sessions). job__check/list read the shared JobState cell without consuming; job__collect blocks with the escalation early-out and applies a tail-biased char-boundary cap plus optional tail_lines; job__cancel kills the group with a 5s grace. Job declarations are injected iff jobs are enabled at agent init, the plain-session function-init sites, and the exit_agent rebuild; job__ is carved out of enabled_tools filtering and excluded from concrete_tool_names so REPL toggles cannot grant or revoke it. --- src/config/agent.rs | 12 +- src/config/app_state.rs | 4 + src/config/input.rs | 15 +- src/config/mod.rs | 3 +- src/config/request_context.rs | 188 +++- src/function/jobs.rs | 1700 ++++++++++++++++++++++++++++++++- src/function/mod.rs | 45 + src/supervisor/mod.rs | 21 +- 8 files changed, 1965 insertions(+), 23 deletions(-) diff --git a/src/config/agent.rs b/src/config/agent.rs index acddb70..c139b71 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -3,7 +3,7 @@ use super::*; use crate::{ client::Model, config::memory, - function::{Functions, run_llm_function}, + function::{Functions, jobs::DEFAULT_MAX_CONCURRENT_JOBS, run_llm_function}, graph, rag, }; @@ -225,6 +225,16 @@ impl Agent { functions.append_supervisor_functions(); } + if app.function_calling_support + && agent_config + .max_concurrent_jobs + .or(app.max_concurrent_jobs) + .unwrap_or(DEFAULT_MAX_CONCURRENT_JOBS) + > 0 + { + functions.append_job_functions(); + } + functions.append_teammate_functions(); functions.append_user_interaction_functions(); diff --git a/src/config/app_state.rs b/src/config/app_state.rs index 7a6cb0e..d4c9079 100644 --- a/src/config/app_state.rs +++ b/src/config/app_state.rs @@ -1,6 +1,7 @@ use super::mcp_factory::{McpFactory, McpServerKey}; use super::rag_cache::RagCache; use crate::config::AppConfig; +use crate::config::jobs_enabled; use crate::function::Functions; use crate::mcp::{McpRegistry, McpServersConfig}; use crate::utils::AbortSignal; @@ -72,6 +73,9 @@ impl AppState { if !mcp_registry.is_empty() && config.mcp_server_support { functions.append_mcp_meta_functions(mcp_registry.server_features()); } + if jobs_enabled(None, &config) { + functions.append_job_functions(); + } let mcp_registry = if mcp_registry.is_empty() { None diff --git a/src/config/input.rs b/src/config/input.rs index 5d35064..85a6a68 100644 --- a/src/config/input.rs +++ b/src/config/input.rs @@ -9,7 +9,12 @@ use crate::utils::{AbortSignal, base64_encode, is_loader_protocol, sha256}; use anyhow::{Context, Result, bail}; use indexmap::IndexSet; -use std::{collections::HashMap, fs::File, io::Read, sync::Arc}; +use std::{ + collections::{HashMap, HashSet}, + fs::File, + io::Read, + sync::Arc, +}; use unicode_width::{UnicodeWidthChar, UnicodeWidthStr}; const IMAGE_EXTS: [&str; 5] = ["png", "jpeg", "jpg", "webp", "gif"]; @@ -158,6 +163,14 @@ impl Input { self.data_urls.clone() } + /// Names of the function declarations this request will send to the model. + pub fn declared_function_names(&self) -> HashSet { + self.functions + .as_ref() + .map(|functions| functions.iter().map(|f| f.name.clone()).collect()) + .unwrap_or_default() + } + pub fn tool_calls(&self) -> &Option { &self.tool_calls } diff --git a/src/config/mod.rs b/src/config/mod.rs index 6f070b4..e89caee 100644 --- a/src/config/mod.rs +++ b/src/config/mod.rs @@ -59,7 +59,8 @@ pub use self::skill_registry::SkillRegistry; #[cfg(test)] pub(crate) use self::tool_scope::test_fixtures; pub use self::tool_scope::{ - McpPromptCompletion, flatten_prompt_messages, resolve_prompt_args, sanitize_display_text, + McpPromptCompletion, McpRuntime, flatten_prompt_messages, resolve_prompt_args, + sanitize_display_text, }; pub use self::update::run_self_update; use crate::client::{ diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 3dc350c..00be93c 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -17,9 +17,13 @@ use super::{ use super::{MessageContentToolCalls, prompts}; use crate::client::{Model, ModelType, list_models}; use crate::function::{ - FunctionDeclaration, Functions, ToolCallTracker, ToolResult, memory::MEMORY_FUNCTION_PREFIX, - rag_query::RAG_FUNCTION_PREFIX, skill::SKILL_FUNCTION_PREFIX, - supervisor::SUPERVISOR_FUNCTION_PREFIX, todo::TODO_FUNCTION_PREFIX, + FunctionDeclaration, Functions, ToolCallTracker, ToolResult, + jobs::{DEFAULT_MAX_CONCURRENT_JOBS, JOB_FUNCTION_PREFIX}, + memory::MEMORY_FUNCTION_PREFIX, + rag_query::RAG_FUNCTION_PREFIX, + skill::SKILL_FUNCTION_PREFIX, + supervisor::SUPERVISOR_FUNCTION_PREFIX, + todo::TODO_FUNCTION_PREFIX, user_interaction::USER_FUNCTION_PREFIX, }; use crate::mcp::{ @@ -142,7 +146,7 @@ pub fn effective_max_concurrent_jobs(agent: Option<&Agent>, app: &AppConfig) -> agent .and_then(|a| a.max_concurrent_jobs()) .or(app.max_concurrent_jobs) - .unwrap_or(5) + .unwrap_or(DEFAULT_MAX_CONCURRENT_JOBS) } pub fn jobs_enabled(agent: Option<&Agent>, app: &AppConfig) -> bool { @@ -319,6 +323,8 @@ pub struct RequestContext { pub tool_scope: ToolScope, + pub declared_function_names: HashSet, + pub supervisor: Option>>, pub parent_supervisor: Option>>, pub self_agent_id: Option, @@ -352,6 +358,7 @@ impl RequestContext { agent: None, last_message: None, tool_scope: ToolScope::default(), + declared_function_names: Default::default(), supervisor: None, parent_supervisor: None, self_agent_id: None, @@ -411,6 +418,7 @@ impl RequestContext { mcp_runtime, tool_tracker: ToolCallTracker::default(), }, + declared_function_names: Default::default(), supervisor: None, parent_supervisor: None, self_agent_id: None, @@ -457,6 +465,7 @@ impl RequestContext { agent: self.agent.clone(), last_message: self.last_message.clone(), tool_scope: self.tool_scope.clone(), + declared_function_names: self.declared_function_names.clone(), supervisor: self.supervisor.clone(), parent_supervisor: self.parent_supervisor.clone(), self_agent_id: self.self_agent_id.clone(), @@ -501,6 +510,7 @@ impl RequestContext { mcp_runtime: McpRuntime::default(), tool_tracker: tool_call_tracker, }, + declared_function_names: Default::default(), supervisor: None, parent_supervisor: parent.supervisor.clone(), self_agent_id: Some(self_agent_id), @@ -905,6 +915,9 @@ impl RequestContext { } pub fn before_chat_completion(&mut self, input: &Input) -> Result<()> { + // The R11 gate in `job__start` validates against exactly what was + // declared to the model for THIS request; refresh it every time. + self.declared_function_names = input.declared_function_names(); self.last_message = Some(LastMessage::new(input.clone(), String::new())); Ok(()) } @@ -1313,6 +1326,7 @@ impl RequestContext { && !v.name.starts_with("memory__") && !v.name.starts_with("skill__") && !v.name.starts_with("rag__") + && !v.name.starts_with("job__") }) .map(|v| v.name.clone()) .collect() @@ -2130,7 +2144,8 @@ impl RequestContext { && v.name.starts_with(SKILL_FUNCTION_PREFIX)) || (self.auto_continue_config().enabled && v.name.starts_with(TODO_FUNCTION_PREFIX)) - || v.name.starts_with(RAG_FUNCTION_PREFIX)) + || v.name.starts_with(RAG_FUNCTION_PREFIX) + || v.name.starts_with(JOB_FUNCTION_PREFIX)) && !existing.contains(&v.name) }) .cloned() @@ -2157,6 +2172,7 @@ impl RequestContext { || v.name.starts_with(SUPERVISOR_FUNCTION_PREFIX) || v.name.starts_with(MEMORY_FUNCTION_PREFIX) || v.name.starts_with(RAG_FUNCTION_PREFIX) + || v.name.starts_with(JOB_FUNCTION_PREFIX) }); } @@ -3869,6 +3885,9 @@ impl RequestContext { { functions.append_rag_query_functions(); } + if self.agent.is_none() && jobs_enabled(None, app) { + functions.append_job_functions(); + } let tool_tracker = self.tool_scope.tool_tracker.clone(); self.tool_scope = ToolScope { @@ -4206,6 +4225,9 @@ impl RequestContext { if self.working_mode.is_repl() { functions.append_user_interaction_functions(); } + if jobs_enabled(None, app) { + functions.append_job_functions(); + } let tool_tracker = self.tool_scope.tool_tracker.clone(); self.tool_scope = ToolScope { functions, @@ -5713,6 +5735,126 @@ mod tests { assert!(ctx.select_functions(&role).is_none()); } + #[test] + fn select_functions_returns_job_functions_even_with_no_enabled_tools() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + let fns = ctx.select_functions(&Role::default()).unwrap(); + let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); + + assert_eq!( + names, + vec![ + "job__start", + "job__check", + "job__collect", + "job__cancel", + "job__list" + ] + ); + } + + #[test] + fn select_functions_preserves_job_tools_under_role_filter() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["foo".to_string()])); + + let fns = ctx.select_functions(&role).unwrap(); + assert!( + fns.iter().any(|f| f.name == "job__start"), + "job__ tools must survive a role tool filter" + ); + } + + #[test] + fn concrete_tool_names_excludes_job_functions() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + assert!(ctx.concrete_tool_names().is_empty()); + } + + #[test] + fn before_chat_completion_refreshes_declared_function_names() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + let input = Input::from_str(&ctx, "hello", None).unwrap(); + ctx.before_chat_completion(&input).unwrap(); + + assert_eq!(ctx.declared_function_names.len(), 5); + assert!(ctx.declared_function_names.contains("job__start")); + + ctx.tool_scope = ToolScope::default(); + let input = Input::from_str(&ctx, "hello again", None).unwrap(); + ctx.before_chat_completion(&input).unwrap(); + + assert!( + ctx.declared_function_names.is_empty(), + "stash must be refreshed on every request" + ); + } + + #[test] + #[serial] + fn rebuild_tool_scope_gates_job_functions_on_jobs_enabled() { + let _guard = TestConfigDirGuard::new(); + let app_state = app_state_with_mcp_config(false, &[]); + let mut ctx = RequestContext::new(app_state, WorkingMode::Cmd); + let app = ctx.app.config.clone(); + let abort = utils::create_abort_signal(); + + run_async(ctx.rebuild_tool_scope(&app, None, abort.clone())).unwrap(); + assert!(ctx.tool_scope.functions.contains("job__start")); + + let jobs_off = AppConfig { + max_concurrent_jobs: Some(0), + ..(*app).clone() + }; + run_async(ctx.rebuild_tool_scope(&jobs_off, None, abort.clone())).unwrap(); + assert!( + !ctx.tool_scope + .functions + .declarations() + .iter() + .any(|f| f.name.starts_with("job__")) + ); + + let fc_off = AppConfig { + function_calling_support: false, + ..(*app).clone() + }; + run_async(ctx.rebuild_tool_scope(&fc_off, None, abort)).unwrap(); + assert!( + !ctx.tool_scope + .functions + .declarations() + .iter() + .any(|f| f.name.starts_with("job__")) + ); + } + + #[test] + #[serial] + fn exit_agent_rebuild_retains_job_functions_when_enabled() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + + ctx.exit_agent(&app).unwrap(); + assert!(ctx.tool_scope.functions.contains("job__start")); + + let jobs_off = AppConfig { + max_concurrent_jobs: Some(0), + ..(*app).clone() + }; + ctx.exit_agent(&jobs_off).unwrap(); + assert!(!ctx.tool_scope.functions.contains("job__start")); + } + #[test] fn select_functions_all_enabled_tools_returns_all_non_mcp() { let mut ctx = create_test_ctx(); @@ -5879,6 +6021,42 @@ mod tests { ); } + #[test] + #[serial] + fn select_functions_preserves_job_tools_under_agent_filter() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + let agent_name = format!( + "test_job_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + let abort = utils::create_abort_signal(); + run_async(ctx.use_agent(&app, &agent_name, None, abort)).unwrap(); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["foo".to_string()])); + + let fns = ctx.select_functions(&role).unwrap(); + let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); + assert!( + names.contains(&"job__start"), + "job__ tools must survive an agent tool filter, got: {names:?}" + ); + assert!(names.contains(&"job__collect")); + } + #[test] fn fork_for_branch_clones_skill_registry() { let mut ctx = create_test_ctx(); diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 05ac697..14c66a1 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -1,9 +1,44 @@ -use crate::supervisor::Supervisor; +use super::memory::MEMORY_FUNCTION_PREFIX; +use super::rag_query::RAG_FUNCTION_PREFIX; +use super::skill::SKILL_FUNCTION_PREFIX; +use super::supervisor::SUPERVISOR_FUNCTION_PREFIX; +use super::todo::TODO_FUNCTION_PREFIX; +use super::user_interaction::USER_FUNCTION_PREFIX; +use super::{FunctionDeclaration, JsonSchema, PATH_SEP, mcp_error_display, render_tool_result}; +use crate::config::{ + McpRuntime, RequestContext, effective_max_concurrent_jobs, jobs_enabled, paths, +}; +use crate::graph; +use crate::mcp::{ + MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX, MCP_INVOKE_META_FUNCTION_NAME_PREFIX, + MCP_PROMPT_META_FUNCTION_NAME_PREFIX, MCP_READ_META_FUNCTION_NAME_PREFIX, + MCP_SEARCH_META_FUNCTION_NAME_PREFIX, +}; +use crate::supervisor::{JobHandle, JobResult, JobState, JobStatus, Supervisor}; +use crate::utils::{create_abort_signal, muted_warning_text, temp_file, wait_abort_signal}; -use parking_lot::RwLock; +use anyhow::{Context, Result, anyhow, bail}; +use indexmap::IndexMap; +use parking_lot::{Mutex, RwLock}; +use serde_json::{Value, json}; +use std::collections::HashMap; +use std::path::PathBuf; +use std::process::Stdio; use std::sync::Arc; +use std::time::{Duration, Instant}; +use std::{env, fs}; +use tokio::io::AsyncReadExt; +use tokio::time; +use uuid::Uuid; + +pub const JOB_FUNCTION_PREFIX: &str = "job__"; + +pub const DEFAULT_MAX_CONCURRENT_JOBS: usize = 5; + +const JOB_RESULT_TAIL_CAP_CHARS: usize = 50_000; + +const JOB_KILL_GRACE: Duration = Duration::from_secs(5); -#[allow(dead_code)] pub fn is_agent_task(supervisor: Option<&Arc>>, id: &str) -> bool { id.starts_with("agent_") || id.starts_with("graph_agent_") @@ -17,7 +52,6 @@ pub struct RingBuf { total_written: u64, } -#[allow(dead_code)] impl RingBuf { pub fn new(capacity: usize) -> Self { Self { @@ -69,9 +103,1019 @@ impl Default for RingBuf { } } +/// Everything a detached process job needs, frozen at `job__start`: config, +/// env, and PATH changes made afterwards do not affect a running job. +pub struct JobEnvSnapshot { + cmd_name: String, + display_name: String, + cmd_args: Vec, + envs: HashMap, + output_file: PathBuf, + timeout_secs: u64, +} + +/// The complete context an MCP job task owns. The runtime holds ONLY the +/// validated server's handle, so the detached task cannot reach any other +/// server even by bug. +pub struct JobCtx { + mcp_runtime: McpRuntime, + current_depth: usize, +} + +pub fn job_function_declarations() -> Vec { + vec![ + FunctionDeclaration { + name: format!("{JOB_FUNCTION_PREFIX}start"), + description: "Run a tool call as a background job and return immediately with a job id, so you can \ + keep working while it runs. `arguments` is the same object the tool takes when called \ + directly. Backgroundable tools: external command tools (e.g. execute_command) and \ + `mcp_invoke_*` calls; built-in `agent__`/`job__`/`user__`/`todo__`/`memory__`/`skill__` \ + tools cannot be backgrounded. The job runs against a snapshot of the current config and \ + environment; later changes do not affect it. Process jobs honor COYOTE_TOOL_TIMEOUT; MCP \ + jobs have NO timeout — cancel a hung one with `job__cancel`. Jobs do not survive coyote \ + exiting.".to_string(), + parameters: JsonSchema { + type_value: Some("object".to_string()), + properties: Some(IndexMap::from([ + ( + "tool".to_string(), + JsonSchema { + type_value: Some("string".to_string()), + description: Some("Name of the tool to run in the background, exactly as it appears in your tool catalog".into()), + ..Default::default() + }, + ), + ( + "arguments".to_string(), + JsonSchema { + type_value: Some("object".to_string()), + description: Some("The arguments object the tool takes when called directly".into()), + ..Default::default() + }, + ), + ])), + required: Some(vec!["tool".to_string()]), + ..Default::default() + }, + agent: false, + }, + FunctionDeclaration { + name: format!("{JOB_FUNCTION_PREFIX}check"), + description: "Non-blocking status probe for a background job. Returns status, elapsed time, and a tail \ + of the output captured so far; it NEVER consumes the result — use `job__collect` for \ + that. Call sparingly: if repeated checks show no change, do other work instead — you \ + will be notified when the job completes.".to_string(), + parameters: JsonSchema { + type_value: Some("object".to_string()), + properties: Some(IndexMap::from([( + "id".to_string(), + JsonSchema { + type_value: Some("string".to_string()), + description: Some("The job ID returned by job__start".into()), + ..Default::default() + }, + )])), + required: Some(vec!["id".to_string()]), + ..Default::default() + }, + agent: false, + }, + FunctionDeclaration { + name: format!("{JOB_FUNCTION_PREFIX}collect"), + description: "Block until the named background job finishes, then return its result and remove the \ + job. The result keeps the LAST 50,000 chars by default (failures land at the tail of \ + build logs); pass `tail_lines` to keep only the last N lines instead.".to_string(), + parameters: JsonSchema { + type_value: Some("object".to_string()), + properties: Some(IndexMap::from([ + ( + "id".to_string(), + JsonSchema { + type_value: Some("string".to_string()), + description: Some("The job ID returned by job__start".into()), + ..Default::default() + }, + ), + ( + "tail_lines".to_string(), + JsonSchema { + type_value: Some("number".to_string()), + description: Some("Keep only the last N lines of the result".into()), + ..Default::default() + }, + ), + ])), + required: Some(vec!["id".to_string()]), + ..Default::default() + }, + agent: false, + }, + FunctionDeclaration { + name: format!("{JOB_FUNCTION_PREFIX}cancel"), + description: "Cancel a background job: kills its process (SIGTERM, then SIGKILL after a 5s grace) and \ + discards the handle. Returns any partial output captured so far. The id cannot be used \ + afterwards.".to_string(), + parameters: JsonSchema { + type_value: Some("object".to_string()), + properties: Some(IndexMap::from([( + "id".to_string(), + JsonSchema { + type_value: Some("string".to_string()), + description: Some("The job ID returned by job__start".into()), + ..Default::default() + }, + )])), + required: Some(vec!["id".to_string()]), + ..Default::default() + }, + agent: false, + }, + FunctionDeclaration { + name: format!("{JOB_FUNCTION_PREFIX}list"), + description: "List all background jobs you have started that are still registered, with status, \ + elapsed time, and bytes of output captured.".to_string(), + parameters: JsonSchema { + type_value: Some("object".to_string()), + properties: Some(IndexMap::new()), + ..Default::default() + }, + agent: false, + }, + ] +} + +pub async fn handle_job_tool( + ctx: &mut RequestContext, + cmd_name: &str, + args: &Value, +) -> Result { + let action = cmd_name + .strip_prefix(JOB_FUNCTION_PREFIX) + .unwrap_or(cmd_name); + + match action { + "start" => handle_start(ctx, args).await, + "check" => handle_check(ctx, args), + "collect" => handle_collect(ctx, args).await, + "cancel" => handle_cancel(ctx, args).await, + "list" => handle_list(ctx), + _ => bail!("Unknown job action: {action}"), + } +} + +fn job_status_str(status: JobStatus) -> &'static str { + match status { + JobStatus::Running => "running", + JobStatus::Completed => "completed", + JobStatus::Failed => "failed", + } +} + +fn job_miss_error(supervisor: Option<&Arc>>, id: &str) -> Value { + if is_agent_task(supervisor, id) { + json!({ + "status": "error", + "message": format!( + "'{id}' is a spawned agent, not a background job — use agent__check / agent__collect / agent__cancel" + ), + }) + } else { + json!({ + "status": "error", + "message": format!( + "No job '{id}' is registered — it may have already been collected or cancelled. job__list shows active jobs." + ), + }) + } +} + +fn whitelist_rejection(tool: &str) -> Option { + let non_invoke_mcp_prefixes = [ + MCP_SEARCH_META_FUNCTION_NAME_PREFIX, + MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX, + MCP_READ_META_FUNCTION_NAME_PREFIX, + MCP_PROMPT_META_FUNCTION_NAME_PREFIX, + ]; + let reason = if tool.starts_with(SUPERVISOR_FUNCTION_PREFIX) + || tool.starts_with(JOB_FUNCTION_PREFIX) + { + Some(format!( + "'{tool}' is already asynchronous — call it directly. Agents may start jobs, but jobs never start agents or other jobs." + )) + } else if tool.starts_with(USER_FUNCTION_PREFIX) { + Some(format!( + "'{tool}' is interactive and must run in-turn — a background job cannot touch the terminal. Call it directly." + )) + } else if tool.starts_with(TODO_FUNCTION_PREFIX) + || tool.starts_with(MEMORY_FUNCTION_PREFIX) + || tool.starts_with(SKILL_FUNCTION_PREFIX) + || tool.starts_with(RAG_FUNCTION_PREFIX) + { + Some(format!( + "'{tool}' mutates agent/session state and must run in-turn. Call it directly." + )) + } else if non_invoke_mcp_prefixes + .iter() + .any(|prefix| tool.starts_with(prefix)) + { + Some(format!( + "'{tool}' is a sub-second call; invoke it directly." + )) + } else { + None + }; + + reason.map(|why| { + json!({ + "status": "error", + "message": format!( + "{why} Backgroundable tools: external command tools (e.g. execute_command) and mcp_invoke_* calls." + ), + }) + }) +} + +async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { + if !jobs_enabled(ctx.agent.as_ref(), &ctx.app.config) { + return Ok(json!({ + "status": "error", + "message": "Background jobs are disabled in this context (max_concurrent_jobs is 0).", + })); + } + + let tool = args + .get("tool") + .and_then(Value::as_str) + .map(str::trim) + .filter(|s| !s.is_empty()) + .ok_or_else(|| anyhow!("'tool' is required"))? + .to_string(); + let arguments = args.get("arguments").cloned().unwrap_or_else(|| json!({})); + + if let Some(rejection) = whitelist_rejection(&tool) { + return Ok(rejection); + } + + if !ctx.declared_function_names.contains(&tool) { + return Ok(json!({ + "status": "error", + "message": format!( + "'{tool}' is not enabled in this context — job__start can only background tools declared to you in this request. Use the exact name of a tool from your current catalog." + ), + })); + } + + let supervisor = match ctx.supervisor.as_ref() { + Some(sup) => Arc::clone(sup), + None => { + let max_jobs = effective_max_concurrent_jobs(ctx.agent.as_ref(), &ctx.app.config); + let sup = Arc::new(RwLock::new( + Supervisor::new(0, 0).with_max_concurrent_jobs(max_jobs), + )); + ctx.supervisor = Some(Arc::clone(&sup)); + sup + } + }; + + { + let sup = supervisor.read(); + if sup.active_job_count() >= sup.max_concurrent_jobs() { + return Ok(json!({ + "status": "error", + "message": format!( + "At capacity: {}/{} jobs running. Collect or cancel one first.", + sup.active_job_count(), + sup.max_concurrent_jobs() + ), + })); + } + } + + let short_uuid = &Uuid::new_v4().to_string()[..8]; + let job_id = format!("job_{short_uuid}"); + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + + let join_handle = if tool.starts_with(MCP_INVOKE_META_FUNCTION_NAME_PREFIX) { + let server = tool.replace(&format!("{MCP_INVOKE_META_FUNCTION_NAME_PREFIX}_"), ""); + let Some(server_handle) = ctx.tool_scope.mcp_runtime.get(&server) else { + return Ok(json!({ + "status": "error", + "message": format!("MCP server '{server}' is not connected in this context."), + })); + }; + let inner_tool = arguments + .get("tool") + .and_then(Value::as_str) + .ok_or_else(|| anyhow!("Missing 'tool' in arguments"))? + .to_string(); + let inner_args = arguments + .get("arguments") + .cloned() + .unwrap_or_else(|| json!({})); + let mut mcp_runtime = McpRuntime::new(); + mcp_runtime.insert(server.clone(), Arc::clone(server_handle)); + let job_ctx = JobCtx { + mcp_runtime, + current_depth: ctx.current_depth, + }; + let task_state = Arc::clone(&state); + tokio::spawn(async move { + let result = run_mcp_job(job_ctx, server, inner_tool, inner_args).await; + task_state.lock().status = match &result { + Ok(_) => JobStatus::Completed, + Err(_) => JobStatus::Failed, + }; + result + }) + } else { + let snapshot = build_env_snapshot(ctx, &tool, &arguments)?; + let task_state = Arc::clone(&state); + let task_buf = Arc::clone(&output_buf); + tokio::spawn(async move { + let result = run_process_job(snapshot, Arc::clone(&task_state), task_buf).await; + let mut job_state = task_state.lock(); + job_state.pgid = None; + job_state.status = match &result { + Ok(job_result) if job_result.exit_code == Some(0) => JobStatus::Completed, + _ => JobStatus::Failed, + }; + drop(job_state); + result + }) + }; + + let handle = JobHandle { + id: job_id.clone(), + tool: tool.clone(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state, + output_buf, + no_change_checks: 0, + }; + + // On a capacity race the handle is dropped here, which kills the process + // group and aborts the task. + if let Err(e) = supervisor.write().register(handle) { + return Ok(json!({ + "status": "error", + "message": format!("{e}"), + })); + } + + Ok(json!({ + "status": "ok", + "job_id": job_id, + "tool": tool, + "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting.", + })) +} + +fn handle_check(ctx: &RequestContext, args: &Value) -> Result { + let id = args + .get("id") + .and_then(Value::as_str) + .ok_or_else(|| anyhow!("'id' is required"))?; + + let Some(supervisor) = ctx.supervisor.as_ref() else { + return Ok(job_miss_error(None, id)); + }; + let sup = supervisor.read(); + let Some(job) = sup.job(id) else { + drop(sup); + return Ok(job_miss_error(ctx.supervisor.as_ref(), id)); + }; + + let status = job.state.lock().status; + let (tail, total_written) = { + let buf = job.output_buf.lock(); + (buf.tail(), buf.total_written()) + }; + let tail_truncated = (tail.len() as u64) < total_written; + let mut result = json!({ + "status": job_status_str(status), + "id": id, + "tool": job.tool, + "elapsed_secs": job.started_at.elapsed().as_secs(), + "output_tail": String::from_utf8_lossy(&tail).to_string(), + "output_bytes_captured": total_written, + "tail_truncated": tail_truncated, + }); + if matches!(status, JobStatus::Running) { + result["message"] = json!( + "Job is still running. Call job__collect to block for the result, or do other work — you will be notified on completion." + ); + } else { + result["message"] = json!(format!( + "Job finished — retrieve the result with job__collect --id {id}" + )); + } + + Ok(result) +} + +async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { + let id = args + .get("id") + .and_then(Value::as_str) + .ok_or_else(|| anyhow!("'id' is required"))?; + let tail_lines = args + .get("tail_lines") + .and_then(Value::as_u64) + .map(|n| n as usize); + + let Some(supervisor) = ctx.supervisor.as_ref().cloned() else { + return Ok(job_miss_error(None, id)); + }; + + let target_abort = { + let sup = supervisor.read(); + let Some(job) = sup.job(id) else { + drop(sup); + return Ok(job_miss_error(ctx.supervisor.as_ref(), id)); + }; + job.abort_signal.clone() + }; + + loop { + let is_finished = { + let sup = supervisor.read(); + sup.job(id).is_none_or(|job| job.join_handle.is_finished()) + }; + + if is_finished { + break; + } + + if let Some(queue) = ctx.root_escalation_queue() + && queue.has_pending() + { + let summary = queue.pending_summary(); + return Ok(json!({ + "status": "pending", + "id": id, + "message": format!("Job '{id}' is still running, but child agents have pending escalations that need your reply. Reply via agent__reply_escalation, then call job__collect again."), + "pending_escalations": summary, + })); + } + + if target_abort.aborted() { + let deadline = time::Instant::now() + Duration::from_secs(2); + while time::Instant::now() < deadline { + let is_finished = { + let sup = supervisor.read(); + sup.job(id).is_none_or(|job| job.join_handle.is_finished()) + }; + if is_finished { + break; + } + time::sleep(Duration::from_millis(50)).await; + } + break; + } + + tokio::select! { + _ = time::sleep(Duration::from_millis(200)) => {} + _ = wait_abort_signal(&target_abort) => {} + } + } + + let handle = { + let mut sup = supervisor.write(); + sup.take_job(id) + }; + + let Some(mut handle) = handle else { + return Ok(json!({ + "status": "error", + "message": format!("Job '{id}' completed but could not be collected. It may have been collected by another call."), + })); + }; + + let joined = (&mut handle.join_handle).await; + let tool = handle.tool.clone(); + let elapsed_secs = handle.started_at.elapsed().as_secs(); + let status = handle.state.lock().status; + let (tail, total_written) = { + let buf = handle.output_buf.lock(); + (buf.tail(), buf.total_written()) + }; + let output_tail = String::from_utf8_lossy(&tail).to_string(); + + let job_result = match joined { + Err(join_err) => { + return Ok(json!({ + "status": "failed", + "id": id, + "tool": tool, + "error": format!("Job task panicked: {join_err}"), + "output_tail": output_tail, + "output_bytes_captured": total_written, + })); + } + Ok(Err(e)) => { + return Ok(json!({ + "status": "failed", + "id": id, + "tool": tool, + "error": format!("{e}"), + "output_tail": output_tail, + "output_bytes_captured": total_written, + })); + } + Ok(Ok(job_result)) => job_result, + }; + + let (result_value, result_truncated) = cap_result(job_result.output, tail_lines); + let mut response = json!({ + "status": job_status_str(status), + "id": id, + "tool": tool, + "elapsed_secs": elapsed_secs, + "result": result_value, + "output_tail": output_tail, + "output_bytes_captured": job_result.output_bytes_captured, + }); + if let Some(exit_code) = job_result.exit_code { + response["exit_code"] = json!(exit_code); + } + if result_truncated { + response["result_truncated"] = json!(true); + } + + Ok(response) +} + +async fn handle_cancel(ctx: &RequestContext, args: &Value) -> Result { + let id = args + .get("id") + .and_then(Value::as_str) + .ok_or_else(|| anyhow!("'id' is required"))?; + + let Some(supervisor) = ctx.supervisor.as_ref() else { + return Ok(job_miss_error(None, id)); + }; + + let handle = { + let mut sup = supervisor.write(); + sup.take_job(id) + }; + + let Some(mut handle) = handle else { + return Ok(job_miss_error(ctx.supervisor.as_ref(), id)); + }; + + handle.abort_signal.set_ctrlc(); + kill_job_with_grace(&mut handle).await; + + let tool = handle.tool.clone(); + let (tail, total_written) = { + let buf = handle.output_buf.lock(); + (buf.tail(), buf.total_written()) + }; + + Ok(json!({ + "status": "cancelled", + "id": id, + "tool": tool, + "output_tail": String::from_utf8_lossy(&tail).to_string(), + "output_bytes_captured": total_written, + })) +} + +/// SIGTERM the process group, give it a grace period, then SIGKILL. Every +/// group kill is gated on `state.pgid` still being set: the job task clears +/// it right after `wait()` reaps the child, and killing after the reap could +/// signal an innocent recycled pid. MCP jobs (no pgid) fall through to a +/// plain task abort. +async fn kill_job_with_grace(handle: &mut JobHandle) { + #[cfg(unix)] + { + let pgid = handle.state.lock().pgid; + if let Some(pgid) = pgid { + unsafe { libc::killpg(pgid, libc::SIGTERM) }; + if time::timeout(JOB_KILL_GRACE, &mut handle.join_handle) + .await + .is_ok() + { + return; + } + if handle.state.lock().pgid.is_some() { + unsafe { libc::killpg(pgid, libc::SIGKILL) }; + } + } + } + handle.join_handle.abort(); + let _ = time::timeout(JOB_KILL_GRACE, &mut handle.join_handle).await; +} + +fn handle_list(ctx: &RequestContext) -> Result { + let Some(supervisor) = ctx.supervisor.as_ref() else { + return Ok(json!({ + "active_jobs": 0, + "max_concurrent_jobs": effective_max_concurrent_jobs(ctx.agent.as_ref(), &ctx.app.config), + "jobs": [], + })); + }; + let sup = supervisor.read(); + + let jobs: Vec = sup + .jobs() + .map(|job| { + let status = job.state.lock().status; + json!({ + "id": job.id, + "tool": job.tool, + "status": job_status_str(status), + "elapsed_secs": job.started_at.elapsed().as_secs(), + "output_bytes_captured": job.output_buf.lock().total_written(), + }) + }) + .collect(); + + Ok(json!({ + "active_jobs": sup.active_job_count(), + "max_concurrent_jobs": sup.max_concurrent_jobs(), + "jobs": jobs, + })) +} + +/// Mirrors the foreground `extract_call_config` + `run_llm_function` env +/// assembly, resolved eagerly so the detached task owns everything it needs. +fn build_env_snapshot( + ctx: &RequestContext, + tool: &str, + arguments: &Value, +) -> Result { + let agent = ctx.agent.as_ref(); + let (cmd_name, mut cmd_args, mut envs) = match agent { + Some(agent) => match agent.functions().find(tool) { + Some(declaration) if declaration.agent => ( + format!("{}-{tool}", agent.name()), + vec![tool.to_string()], + agent.variable_envs(), + ), + Some(_) => (tool.to_string(), vec![], agent.variable_envs()), + None => (tool.to_string(), vec![], HashMap::new()), + }, + None => (tool.to_string(), vec![], HashMap::new()), + }; + + let mut bin_dirs: Vec = vec![]; + if let Some(agent) = agent { + let dir = paths::agent_bin_dir(agent.name()); + if dir.exists() { + bin_dirs.push(dir); + } + if graph::agent_has_graph(agent.name()) { + envs.insert("AUTO_CONFIRM".into(), "true".into()); + } + } else { + bin_dirs.push(paths::functions_bin_dir()); + } + let current_path = env::var("PATH").context("No PATH environment variable")?; + let prepend_path = bin_dirs + .iter() + .map(|v| format!("{}{PATH_SEP}", v.display())) + .collect::>() + .join(""); + envs.insert("PATH".into(), format!("{prepend_path}{current_path}")); + + let output_file = temp_file("-job-", ""); + envs.insert("LLM_OUTPUT".into(), output_file.display().to_string()); + envs.insert("CLICOLOR_FORCE".into(), "1".into()); + envs.insert("FORCE_COLOR".into(), "1".into()); + + cmd_args.push(arguments.to_string()); + + #[cfg(windows)] + let cmd_name = super::polyfill_cmd_name(&cmd_name, &bin_dirs); + + #[cfg(windows)] + let cmd_args = { + let mut args = cmd_args; + if let Some(json_data) = args.pop() { + let tool_data_file = temp_file("-tool-data-", ".json"); + fs::write(&tool_data_file, &json_data)?; + envs.insert( + "LLM_TOOL_DATA_FILE".into(), + tool_data_file.display().to_string(), + ); + } + args + }; + + let timeout_secs = env::var("COYOTE_TOOL_TIMEOUT") + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(1800); + + Ok(JobEnvSnapshot { + cmd_name, + display_name: tool.to_string(), + cmd_args, + envs, + output_file, + timeout_secs, + }) +} + +async fn pump_into_ring(mut reader: impl AsyncReadExt + Unpin, output_buf: Arc>) { + let mut chunk = [0u8; 1024]; + loop { + match reader.read(&mut chunk).await { + Ok(0) | Err(_) => break, + Ok(n) => output_buf.lock().push(&chunk[..n]), + } + } +} + +async fn run_process_job( + snapshot: JobEnvSnapshot, + state: Arc>, + output_buf: Arc>, +) -> Result { + let mut command = tokio::process::Command::new(&snapshot.cmd_name); + command + .args(&snapshot.cmd_args) + .envs(&snapshot.envs) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .kill_on_drop(true); + #[cfg(unix)] + command.process_group(0); + + let mut child = command + .spawn() + .map_err(|err| anyhow!("Unable to run {}, {err}", snapshot.display_name))?; + + #[cfg(unix)] + if let Some(pid) = child.id() { + state.lock().pgid = Some(pid as i32); + } + + let stdout = child + .stdout + .take() + .ok_or_else(|| anyhow!("Failed to capture stdout"))?; + let stderr = child + .stderr + .take() + .ok_or_else(|| anyhow!("Failed to capture stderr"))?; + let stdout_pump = tokio::spawn(pump_into_ring(stdout, Arc::clone(&output_buf))); + let stderr_pump = tokio::spawn(pump_into_ring(stderr, Arc::clone(&output_buf))); + + let wait_result = if snapshot.timeout_secs > 0 { + match time::timeout(Duration::from_secs(snapshot.timeout_secs), child.wait()).await { + Ok(wait_result) => wait_result, + Err(_) => { + kill_expired_job(&mut child, &state).await; + state.lock().pgid = None; + let _ = stdout_pump.await; + let _ = stderr_pump.await; + let output_bytes_captured = output_buf.lock().total_written(); + let message = format!( + "Tool call '{}' timed out after {}s and was killed (set COYOTE_TOOL_TIMEOUT to adjust; 0 = unlimited)", + snapshot.display_name, snapshot.timeout_secs + ); + return Ok(JobResult { + output: json!({"tool_call_error": message}), + exit_code: None, + output_bytes_captured, + }); + } + } + } else { + child.wait().await + }; + let status = + wait_result.map_err(|err| anyhow!("Unable to run {}, {err}", snapshot.display_name))?; + // pid-reuse guard: the child is reaped, so a later group kill against + // this pgid could hit an innocent recycled pid. + state.lock().pgid = None; + let _ = stdout_pump.await; + let _ = stderr_pump.await; + let output_bytes_captured = output_buf.lock().total_written(); + + let exit_code = status.code(); + if exit_code.unwrap_or_default() != 0 { + let message = format!( + "Tool call '{}' exited with code {}", + snapshot.display_name, + exit_code.unwrap_or_default() + ); + let mut error_json = json!({"tool_call_error": message}); + if let Ok(contents) = fs::read_to_string(&snapshot.output_file) + && !contents.trim().is_empty() + { + error_json["output"] = json!(contents); + } + return Ok(JobResult { + output: error_json, + exit_code, + output_bytes_captured, + }); + } + + let mut output = Value::Null; + if snapshot.output_file.exists() { + let contents = fs::read_to_string(&snapshot.output_file) + .context("Failed to retrieve tool call output")?; + if !contents.is_empty() { + output = serde_json::from_str(&contents) + .ok() + .unwrap_or_else(|| json!({"output": contents})); + } + } + + Ok(JobResult { + output, + exit_code, + output_bytes_captured, + }) +} + +async fn kill_expired_job(child: &mut tokio::process::Child, state: &Arc>) { + #[cfg(unix)] + { + let pgid = state.lock().pgid; + if let Some(pgid) = pgid { + unsafe { libc::killpg(pgid, libc::SIGTERM) }; + if time::timeout(JOB_KILL_GRACE, child.wait()).await.is_err() { + unsafe { libc::killpg(pgid, libc::SIGKILL) }; + let _ = child.wait().await; + } + return; + } + } + #[cfg(not(unix))] + let _ = state; + let _ = child.start_kill(); + let _ = child.wait().await; +} + +async fn run_mcp_job( + job_ctx: JobCtx, + server: String, + tool: String, + arguments: Value, +) -> Result { + let raw = match job_ctx.mcp_runtime.invoke(&server, &tool, arguments).await { + Ok(raw) => raw, + Err(e) => { + if job_ctx.current_depth == 0 { + let error_msg = format!("MCP job invocation failed: {e}"); + eprintln!("{}", muted_warning_text(&mcp_error_display(&error_msg))); + } + return Err(e); + } + }; + let output = render_tool_result(serde_json::to_value(raw)?, &server)?; + Ok(JobResult { + output, + exit_code: None, + output_bytes_captured: 0, + }) +} + +/// Tail-biased result capping owned by the collect handler: keeps the LAST +/// `tail_lines`/50,000 chars (build failures land at the tail), always cutting +/// on a char boundary. +fn cap_result(output: Value, tail_lines: Option) -> (Value, bool) { + if output.is_null() { + return (json!("DONE"), false); + } + let mut text = match &output { + Value::String(s) => s.clone(), + other => other.to_string(), + }; + let mut truncated = false; + if let Some(n) = tail_lines { + let lines: Vec<&str> = text.lines().collect(); + if lines.len() > n { + text = lines[lines.len() - n..].join("\n"); + truncated = true; + } + } + if let Some(capped) = tail_chars(&text, JOB_RESULT_TAIL_CAP_CHARS) { + text = capped; + truncated = true; + } + if truncated { + (json!(text), true) + } else { + (output, false) + } +} + +fn tail_chars(text: &str, max_chars: usize) -> Option { + let total = text.chars().count(); + if total <= max_chars { + return None; + } + let cut = text + .char_indices() + .nth(total - max_chars) + .map(|(i, _)| i) + .unwrap_or(0); + Some(format!( + "[truncated: kept last {max_chars} of {total} chars]\n{}", + &text[cut..] + )) +} + #[cfg(test)] mod tests { use super::*; + use crate::config::{AppConfig, AppState, WorkingMode}; + use crate::function::supervisor::{ + GuardrailAction, check_pending_agents_guardrail, handle_supervisor_tool, + }; + use crate::supervisor::mailbox::Inbox; + use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult}; + use std::future::Future; + + fn default_app_state() -> Arc { + Arc::new(AppState::test_default()) + } + + fn app_state_with_config(update: impl FnOnce(&mut AppConfig)) -> Arc { + let mut state = AppState::test_default(); + let mut config = (*state.config).clone(); + update(&mut config); + state.config = Arc::new(config); + Arc::new(state) + } + + fn plain_ctx() -> RequestContext { + RequestContext::new(default_app_state(), WorkingMode::Cmd) + } + + fn ctx_with_job_supervisor(max_jobs: usize) -> RequestContext { + let mut ctx = plain_ctx(); + ctx.supervisor = Some(Arc::new(RwLock::new( + Supervisor::new(0, 3).with_max_concurrent_jobs(max_jobs), + ))); + ctx + } + + fn make_running_job(id: &str) -> JobHandle { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let join_handle = rt.spawn(async { + Ok(JobResult { + output: Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + JobHandle { + id: id.to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + } + } + + fn run_async(f: F) -> F::Output { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(f) + } + + #[cfg(unix)] + fn test_snapshot(cmd: &str, args: &[&str], timeout_secs: u64) -> JobEnvSnapshot { + let output_file = temp_file("-job-test-", ""); + let mut envs = HashMap::new(); + envs.insert("PATH".to_string(), env::var("PATH").unwrap()); + envs.insert("LLM_OUTPUT".to_string(), output_file.display().to_string()); + JobEnvSnapshot { + cmd_name: cmd.to_string(), + display_name: cmd.to_string(), + cmd_args: args.iter().map(|s| s.to_string()).collect(), + envs, + output_file, + timeout_secs, + } + } #[test] fn ring_buf_returns_contents_below_capacity() { @@ -125,10 +1169,6 @@ mod tests { #[test] fn is_agent_task_matches_registered_agents() { - use crate::supervisor::mailbox::Inbox; - use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult}; - use crate::utils::create_abort_signal; - let rt = tokio::runtime::Builder::new_current_thread() .enable_all() .build() @@ -158,4 +1198,648 @@ mod tests { assert!(is_agent_task(Some(&sup), "a1")); assert!(!is_agent_task(Some(&sup), "missing")); } + + #[test] + fn job_function_declarations_cover_all_five_actions() { + let names: Vec = job_function_declarations() + .into_iter() + .map(|d| d.name) + .collect(); + assert_eq!( + names, + vec![ + "job__start", + "job__check", + "job__collect", + "job__cancel", + "job__list" + ] + ); + } + + #[test] + fn whitelist_rejects_state_mutating_tools() { + for tool in ["memory__write", "todo__add", "skill__load", "rag__query"] { + let rejection = whitelist_rejection(tool).unwrap(); + let message = rejection["message"].as_str().unwrap(); + assert!( + message.contains("mutates agent/session state"), + "unexpected message for {tool}: {message}" + ); + } + } + + #[test] + fn whitelist_rejects_async_and_interactive_tools() { + for tool in ["agent__spawn", "job__check"] { + let message = whitelist_rejection(tool).unwrap()["message"] + .as_str() + .unwrap() + .to_string(); + assert!( + message.contains("already asynchronous"), + "unexpected message for {tool}: {message}" + ); + } + let message = whitelist_rejection("user__select").unwrap()["message"] + .as_str() + .unwrap() + .to_string(); + assert!(message.contains("interactive")); + } + + #[test] + fn whitelist_rejects_fast_mcp_meta_tools() { + for tool in [ + "mcp_search_github", + "mcp_describe_github", + "mcp_read_github", + "mcp_prompt_github", + ] { + let message = whitelist_rejection(tool).unwrap()["message"] + .as_str() + .unwrap() + .to_string(); + assert!( + message.contains("sub-second"), + "unexpected message for {tool}: {message}" + ); + } + } + + #[test] + fn whitelist_allows_external_and_mcp_invoke_tools() { + assert!(whitelist_rejection("execute_command").is_none()); + assert!(whitelist_rejection("mcp_invoke_github").is_none()); + } + + #[test] + fn handle_start_rejects_non_whitelisted_tool_without_spawn() { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("memory__write".into()); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "memory__write", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("mutates agent/session state") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[test] + fn handle_start_rejects_undeclared_tool_without_spawn() { + let mut ctx = plain_ctx(); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "execute_command", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[test] + fn handle_start_rejects_when_jobs_disabled() { + let app_state = app_state_with_config(|config| config.max_concurrent_jobs = Some(0)); + let mut ctx = RequestContext::new(app_state, WorkingMode::Cmd); + ctx.declared_function_names.insert("execute_command".into()); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "execute_command", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!(result["message"].as_str().unwrap().contains("disabled")); + assert!(ctx.supervisor.is_none()); + } + + #[test] + fn handle_start_rejects_at_capacity_without_spawn() { + let mut ctx = ctx_with_job_supervisor(1); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("j1")) + .unwrap(); + ctx.declared_function_names.insert("execute_command".into()); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "execute_command", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("At capacity: 1/1") + ); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().active_job_count(), + 1 + ); + } + + #[test] + fn handle_start_rejects_unconnected_mcp_server() { + let mut ctx = plain_ctx(); + ctx.declared_function_names + .insert("mcp_invoke_github".into()); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "mcp_invoke_github", "arguments": {"tool": "search"}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not connected") + ); + assert_eq!( + ctx.supervisor.as_ref().unwrap().read().active_job_count(), + 0 + ); + } + + #[cfg(unix)] + #[test] + fn handle_start_lazy_inits_supervisor_and_collect_returns_result() { + run_async(async { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("echo".into()); + + let started = handle_start(&mut ctx, &json!({"tool": "echo", "arguments": {}})) + .await + .unwrap(); + + assert_eq!(started["status"], "ok"); + let job_id = started["job_id"].as_str().unwrap().to_string(); + assert!(job_id.starts_with("job_")); + assert!( + ctx.supervisor.is_some(), + "plain sessions lazily init a supervisor" + ); + + let collected = handle_collect(&ctx, &json!({"id": job_id})).await.unwrap(); + + assert_eq!(collected["status"], "completed"); + assert_eq!(collected["result"], "DONE"); + assert_eq!(collected["exit_code"], 0); + assert!(collected["output_tail"].as_str().unwrap().contains("{}")); + assert!(!ctx.supervisor.as_ref().unwrap().read().has_job(&job_id)); + }); + } + + #[test] + fn handle_check_unknown_id_teaches_job_list() { + let ctx = ctx_with_job_supervisor(4); + let result = handle_check(&ctx, &json!({"id": "job_x"})).unwrap(); + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("No job 'job_x' is registered") + ); + } + + #[test] + fn job_handlers_teach_cross_kind_for_agent_ids() { + let ctx = ctx_with_job_supervisor(4); + for result in [ + handle_check(&ctx, &json!({"id": "agent_explore_1"})).unwrap(), + run_async(handle_collect(&ctx, &json!({"id": "agent_explore_1"}))).unwrap(), + run_async(handle_cancel(&ctx, &json!({"id": "agent_explore_1"}))).unwrap(), + ] { + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("is a spawned agent, not a background job") + ); + } + } + + #[test] + fn job_handlers_miss_without_supervisor() { + let ctx = plain_ctx(); + let result = handle_check(&ctx, &json!({"id": "job_x"})).unwrap(); + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("No job 'job_x'") + ); + } + + #[test] + fn handle_check_reports_running_job_tail() { + let ctx = ctx_with_job_supervisor(4); + let job = make_running_job("j1"); + job.output_buf.lock().push(b"hello"); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(job) + .unwrap(); + + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + + assert_eq!(result["status"], "running"); + assert_eq!(result["tool"], "execute_command"); + assert_eq!(result["output_tail"], "hello"); + assert_eq!(result["output_bytes_captured"], 5); + assert_eq!(result["tail_truncated"], false); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("still running") + ); + } + + #[test] + fn handle_check_finished_job_points_at_collect_without_consuming() { + let ctx = ctx_with_job_supervisor(4); + let job = make_running_job("j1"); + job.state.lock().status = JobStatus::Completed; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(job) + .unwrap(); + + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + + assert_eq!(result["status"], "completed"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("job__collect --id j1") + ); + assert!(ctx.supervisor.as_ref().unwrap().read().has_job("j1")); + } + + #[test] + fn handle_collect_applies_tail_lines() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let join_handle = tokio::spawn(async { + Ok(JobResult { + output: json!("l1\nl2\nl3"), + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + let handle = JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Completed, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_collect(&ctx, &json!({"id": "j1", "tail_lines": 2})) + .await + .unwrap(); + + assert_eq!(result["status"], "completed"); + assert_eq!(result["result"], "l2\nl3"); + assert_eq!(result["result_truncated"], true); + }); + } + + #[test] + fn handle_collect_maps_panic_to_failed_with_ring_content() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let join_handle: tokio::task::JoinHandle> = + tokio::spawn(async { panic!("boom") }); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + output_buf.lock().push(b"partial"); + let handle = JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Failed, + pgid: None, + })), + output_buf, + no_change_checks: 0, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_collect(&ctx, &json!({"id": "j1"})).await.unwrap(); + + assert_eq!(result["status"], "failed"); + assert!(result["error"].as_str().unwrap().contains("panicked")); + assert_eq!(result["output_tail"], "partial"); + }); + } + + #[cfg(unix)] + #[test] + fn handle_cancel_kills_running_process_job() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("sleep", &["30"], 0); + let task_state = Arc::clone(&state); + let task_buf = Arc::clone(&output_buf); + let join_handle = + tokio::spawn(async move { run_process_job(snapshot, task_state, task_buf).await }); + + let deadline = time::Instant::now() + Duration::from_secs(2); + while state.lock().pgid.is_none() && time::Instant::now() < deadline { + time::sleep(Duration::from_millis(10)).await; + } + assert!(state.lock().pgid.is_some(), "runner must record the pgid"); + + let handle = JobHandle { + id: "j1".to_string(), + tool: "sleep".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::clone(&state), + output_buf, + no_change_checks: 0, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_cancel(&ctx, &json!({"id": "j1"})).await.unwrap(); + + assert_eq!(result["status"], "cancelled"); + assert_eq!(result["tool"], "sleep"); + assert!(!ctx.supervisor.as_ref().unwrap().read().has_job("j1")); + }); + } + + #[test] + fn handle_list_reports_jobs_and_capacity() { + let ctx = ctx_with_job_supervisor(4); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("j1")) + .unwrap(); + + let result = handle_list(&ctx).unwrap(); + + assert_eq!(result["active_jobs"], 1); + assert_eq!(result["max_concurrent_jobs"], 4); + assert_eq!(result["jobs"][0]["id"], "j1"); + assert_eq!(result["jobs"][0]["status"], "running"); + } + + #[test] + fn handle_list_without_supervisor_reports_empty() { + let ctx = plain_ctx(); + let result = handle_list(&ctx).unwrap(); + assert_eq!(result["active_jobs"], 0); + assert_eq!(result["max_concurrent_jobs"], 5); + assert_eq!(result["jobs"].as_array().unwrap().len(), 0); + } + + #[test] + fn cap_result_normalizes_null_to_done() { + let (value, truncated) = cap_result(Value::Null, None); + assert_eq!(value, json!("DONE")); + assert!(!truncated); + } + + #[test] + fn cap_result_preserves_small_values() { + let (value, truncated) = cap_result(json!({"a": 1}), None); + assert_eq!(value, json!({"a": 1})); + assert!(!truncated); + } + + #[test] + fn cap_result_keeps_last_chars_on_char_boundary() { + let text = "é".repeat(JOB_RESULT_TAIL_CAP_CHARS + 10); + let (value, truncated) = cap_result(json!(text), None); + assert!(truncated); + let capped = value.as_str().unwrap(); + assert!(capped.starts_with(&format!( + "[truncated: kept last {} of {} chars]", + JOB_RESULT_TAIL_CAP_CHARS, + JOB_RESULT_TAIL_CAP_CHARS + 10 + ))); + } + + #[test] + fn tail_chars_floors_to_char_boundary() { + let capped = tail_chars("aébc", 2).unwrap(); + assert!(capped.ends_with("bc")); + assert!(capped.starts_with("[truncated: kept last 2 of 4 chars]")); + assert!(tail_chars("abc", 3).is_none()); + } + + #[cfg(unix)] + #[test] + fn run_process_job_times_out_and_clears_pgid() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("sleep", &["30"], 1); + + let result = run_process_job(snapshot, Arc::clone(&state), output_buf) + .await + .unwrap(); + + assert_eq!(result.exit_code, None); + assert!( + result.output["tool_call_error"] + .as_str() + .unwrap() + .contains("timed out after 1s") + ); + assert!( + state.lock().pgid.is_none(), + "pid-reuse guard must clear pgid" + ); + }); + } + + #[cfg(unix)] + #[test] + fn run_process_job_reads_llm_output_and_captures_ring() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot( + "sh", + &["-c", "printf hi > \"$LLM_OUTPUT\"; echo captured"], + 0, + ); + + let result = run_process_job(snapshot, state, Arc::clone(&output_buf)) + .await + .unwrap(); + + assert_eq!(result.exit_code, Some(0)); + assert_eq!(result.output, json!({"output": "hi"})); + let tail = String::from_utf8_lossy(&output_buf.lock().tail()).to_string(); + assert!(tail.contains("captured")); + assert_eq!(result.output_bytes_captured, 9); + }); + } + + #[cfg(unix)] + #[test] + fn run_process_job_reports_nonzero_exit_with_partial_output() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = + test_snapshot("sh", &["-c", "printf partial > \"$LLM_OUTPUT\"; exit 3"], 0); + + let result = run_process_job(snapshot, state, output_buf).await.unwrap(); + + assert_eq!(result.exit_code, Some(3)); + assert!( + result.output["tool_call_error"] + .as_str() + .unwrap() + .contains("exited with code 3") + ); + assert_eq!(result.output["output"], "partial"); + }); + } + + #[test] + fn jobs_only_supervisor_rejects_agent_spawn_at_capacity_zero() { + let mut ctx = ctx_with_job_supervisor(5); + + let result = run_async(handle_supervisor_tool( + &mut ctx, + "agent__spawn", + &json!({"agent": "explore", "prompt": "x"}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("At capacity: 0/0") + ); + } + + #[test] + fn jobs_only_supervisor_guardrail_takes_no_action() { + let mut ctx = ctx_with_job_supervisor(5); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("j1")) + .unwrap(); + + assert!(matches!( + check_pending_agents_guardrail(&mut ctx), + GuardrailAction::NoAction + )); + } + + #[test] + fn jobs_only_supervisor_agent_surfaces_stay_functional() { + let mut ctx = ctx_with_job_supervisor(5); + + let listed = run_async(handle_supervisor_tool( + &mut ctx, + "agent__list_running", + &json!({}), + )) + .unwrap(); + assert_eq!(listed["active_count"], 0); + assert_eq!(listed["max_concurrent"], 0); + + let created = run_async(handle_supervisor_tool( + &mut ctx, + "agent__task_create", + &json!({"subject": "research"}), + )) + .unwrap(); + assert_eq!(created["status"], "ok"); + + let tasks = run_async(handle_supervisor_tool( + &mut ctx, + "agent__task_list", + &json!({}), + )) + .unwrap(); + assert_eq!(tasks["tasks"].as_array().unwrap().len(), 1); + } } diff --git a/src/function/mod.rs b/src/function/mod.rs index 579c430..78ee969 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -28,6 +28,7 @@ use anyhow::{Context, Result, anyhow, bail}; use futures_util::future; use indexmap::IndexMap; use indoc::formatdoc; +use jobs::JOB_FUNCTION_PREFIX; use memory::MEMORY_FUNCTION_PREFIX; use rag_query::RAG_FUNCTION_PREFIX; use rust_embed::Embed; @@ -625,6 +626,10 @@ impl Functions { .extend(supervisor::escalation_function_declarations()); } + pub fn append_job_functions(&mut self) { + self.declarations.extend(jobs::job_function_declarations()); + } + pub fn append_teammate_functions(&mut self) { self.declarations .extend(supervisor::teammate_function_declarations()); @@ -1548,6 +1553,15 @@ impl ToolCall { json!({"tool_call_error": error_msg}) }) } + _ if cmd_name.starts_with(JOB_FUNCTION_PREFIX) => { + jobs::handle_job_tool(ctx, &cmd_name, &json_data) + .await + .unwrap_or_else(|e| { + let error_msg = format!("Job tool failed: {e}"); + eprintln!("{}", muted_warning_text(&format!("⚠️ {error_msg} ⚠️"))); + json!({"tool_call_error": error_msg}) + }) + } _ => match run_llm_function(cmd_name, cmd_args, envs, agent_name) { Ok(Some(contents)) => serde_json::from_str(&contents) .ok() @@ -2858,6 +2872,37 @@ mod tests { assert!(f.contains("agent__reply_escalation")); } + #[test] + fn functions_append_job_adds_declarations() { + let mut f = Functions::default(); + f.append_job_functions(); + assert!(f.contains("job__start")); + assert!(f.contains("job__check")); + assert!(f.contains("job__collect")); + assert!(f.contains("job__cancel")); + assert!(f.contains("job__list")); + } + + #[test] + fn eval_routes_declared_job_calls_to_job_handlers() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.tool_scope.functions.append_job_functions(); + let calls = vec![call("job__list", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + assert_eq!(results.len(), 1); + assert_eq!(results[0].output["active_jobs"], 0); + assert_eq!(results[0].output["jobs"], json!([])); + } + + #[test] + fn eval_soft_fails_job_calls_when_jobs_not_declared() { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let calls = vec![call("job__start", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + let err = results[0].output["tool_call_error"].as_str().unwrap(); + assert!(err.contains("Unexpected call")); + } + #[test] fn functions_append_teammate_adds_declarations() { let mut f = Functions::default(); diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index dfa072d..ac143b5 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -40,7 +40,6 @@ pub struct AgentHandle { pub child_supervisor: Option>>, } -#[allow(dead_code)] #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum JobStatus { Running, @@ -49,12 +48,10 @@ pub enum JobStatus { } pub struct JobState { - #[allow(dead_code)] pub status: JobStatus, pub pgid: Option, } -#[allow(dead_code)] pub struct JobResult { pub output: Value, pub exit_code: Option, @@ -63,14 +60,11 @@ pub struct JobResult { pub struct JobHandle { pub id: String, - #[allow(dead_code)] pub tool: String, - #[allow(dead_code)] pub started_at: Instant, pub join_handle: JoinHandle>, pub abort_signal: AbortSignal, pub state: Arc>, - #[allow(dead_code)] pub output_buf: Arc>, #[allow(dead_code)] pub no_change_checks: u32, @@ -157,6 +151,20 @@ impl Supervisor { }) } + pub fn job(&self, id: &str) -> Option<&JobHandle> { + match self.handles.get(id) { + Some(TaskHandle::Job(handle)) => Some(handle), + _ => None, + } + } + + pub fn jobs(&self) -> impl Iterator { + self.handles.values().filter_map(|handle| match handle { + TaskHandle::Job(handle) => Some(handle), + TaskHandle::Agent(_) => None, + }) + } + pub fn active_count(&self) -> usize { self.agents().count() } @@ -182,7 +190,6 @@ impl Supervisor { self.max_depth } - #[allow(dead_code)] pub fn max_concurrent_jobs(&self) -> usize { self.max_concurrent_jobs } From 4025b8dacd8b9fe4b54163585f39b7a6536185d8 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 16:56:10 -0600 Subject: [PATCH 09/28] feat: inject background-jobs prompt guidance when jobs are enabled Agents whose function pool includes job__* declarations get a Background Jobs section teaching start/check/collect/cancel discipline and the snapshot/no-persistence semantics. Presence of the declarations doubles as the jobs_enabled predicate, so a context with function calling off or max_concurrent_jobs 0 sees no job prompt text. --- src/config/agent.rs | 21 ++++++++++++++++++--- src/config/prompts.rs | 10 ++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/src/config/agent.rs b/src/config/agent.rs index c139b71..1bafbe3 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -3,15 +3,19 @@ use super::*; use crate::{ client::Model, config::memory, - function::{Functions, jobs::DEFAULT_MAX_CONCURRENT_JOBS, run_llm_function}, + function::{ + Functions, + jobs::{DEFAULT_MAX_CONCURRENT_JOBS, JOB_FUNCTION_PREFIX}, + run_llm_function, + }, graph, rag, }; use super::rag_cache::RagKey; use crate::config::paths; use crate::config::prompts::{ - DEFAULT_SPAWN_INSTRUCTIONS, DEFAULT_TEAMMATE_INSTRUCTIONS, DEFAULT_TODO_INSTRUCTIONS, - DEFAULT_USER_INTERACTION_INSTRUCTIONS, + DEFAULT_JOB_INSTRUCTIONS, DEFAULT_SPAWN_INSTRUCTIONS, DEFAULT_TEAMMATE_INSTRUCTIONS, + DEFAULT_TODO_INSTRUCTIONS, DEFAULT_USER_INTERACTION_INSTRUCTIONS, }; use crate::graph::types::RagNode; use crate::graph::{Graph, GraphParser, NodeType}; @@ -450,6 +454,17 @@ impl Agent { output.push_str(DEFAULT_SPAWN_INSTRUCTIONS); } + // Job declarations are appended at init iff jobs are enabled for this + // agent, so their presence doubles as the jobs_enabled predicate. + if self + .functions + .declarations() + .iter() + .any(|f| f.name.starts_with(JOB_FUNCTION_PREFIX)) + { + output.push_str(DEFAULT_JOB_INSTRUCTIONS); + } + output.push_str(DEFAULT_TEAMMATE_INSTRUCTIONS); output.push_str(DEFAULT_USER_INTERACTION_INSTRUCTIONS); diff --git a/src/config/prompts.rs b/src/config/prompts.rs index cb2e163..e08afec 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -190,6 +190,16 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {" 4. **Respond promptly**; the child agent is blocked and waiting (5-minute timeout). "}; +pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" + ## Background Jobs + + For long-running tool calls (builds, test suites, slow commands), call `job__start` and keep + working instead of blocking. Check progress with `job__check` (sparingly), block on the result + with `job__collect`, cancel with `job__cancel`, and list jobs with `job__list`. Collect or + cancel every job you started before ending your turn. Jobs run against a snapshot of the + current config/environment and do not survive coyote exiting." +}; + pub(in crate::config) const DEFAULT_TEAMMATE_INSTRUCTIONS: &str = indoc! {" ## Teammate Messaging From 177d61cf94c57be02042fea4129555f33566999a Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 17:28:04 -0600 Subject: [PATCH 10/28] fix: harden job runner lifecycle and whitelist conformance Reject fast built-in file tools (fs_* / ast_grep) in job__start per the backgroundable-tools whitelist; clean up env-snapshot temp files on every exit of run_process_job via a drop guard; bound the output-pump awaits and abort them on the failure path; treat signal death (no exit code) as a failure with a teaching message; bound job__collect's post-drain join with a SIGKILL escalation so a TERM-ignoring process cannot hang collect after a Ctrl-C teardown; document the unguarded SIGTERM pid-reuse window; give the injected Background Jobs prompt section a fresh line on both sides; extract the MCP server name with strip_prefix instead of replace. Capacity-0 audit for jobs-disabled contexts: REPL displays have no supervisor consumers (only Ctrl-C/exit cancel_recursive at repl/mod.rs:460,473, kind-agnostic); session save/load does not persist supervisor state (src/config/session.rs has no supervisor references) -- nothing to test for either. --- src/config/agent.rs | 3 + src/config/prompts.rs | 3 +- src/config/request_context.rs | 4 +- src/function/jobs.rs | 265 ++++++++++++++++++++++++++++++++-- 4 files changed, 256 insertions(+), 19 deletions(-) diff --git a/src/config/agent.rs b/src/config/agent.rs index 1bafbe3..8ff23d8 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -462,6 +462,9 @@ impl Agent { .iter() .any(|f| f.name.starts_with(JOB_FUNCTION_PREFIX)) { + if !output.ends_with('\n') { + output.push('\n'); + } output.push_str(DEFAULT_JOB_INSTRUCTIONS); } diff --git a/src/config/prompts.rs b/src/config/prompts.rs index e08afec..556bc26 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -197,7 +197,8 @@ pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" working instead of blocking. Check progress with `job__check` (sparingly), block on the result with `job__collect`, cancel with `job__cancel`, and list jobs with `job__list`. Collect or cancel every job you started before ending your turn. Jobs run against a snapshot of the - current config/environment and do not survive coyote exiting." + current config/environment and do not survive coyote exiting. +" }; pub(in crate::config) const DEFAULT_TEAMMATE_INSTRUCTIONS: &str = indoc! {" diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 00be93c..b5b26bb 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -915,8 +915,8 @@ impl RequestContext { } pub fn before_chat_completion(&mut self, input: &Input) -> Result<()> { - // The R11 gate in `job__start` validates against exactly what was - // declared to the model for THIS request; refresh it every time. + // `job__start` validates against exactly what was declared to the + // model for THIS request; refresh it every time. self.declared_function_names = input.declared_function_names(); self.last_message = Some(LastMessage::new(input.clone(), String::new())); Ok(()) diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 14c66a1..bc36f0e 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -39,6 +39,8 @@ const JOB_RESULT_TAIL_CAP_CHARS: usize = 50_000; const JOB_KILL_GRACE: Duration = Duration::from_secs(5); +const JOB_PUMP_DRAIN_GRACE: Duration = Duration::from_secs(2); + pub fn is_agent_task(supervisor: Option<&Arc>>, id: &str) -> bool { id.starts_with("agent_") || id.starts_with("graph_agent_") @@ -314,6 +316,10 @@ fn whitelist_rejection(tool: &str) -> Option { Some(format!( "'{tool}' mutates agent/session state and must run in-turn. Call it directly." )) + } else if tool.starts_with("fs_") || tool == "ast_grep" { + Some(format!( + "'{tool}' is fast — invoke it directly instead of backgrounding it." + )) } else if non_invoke_mcp_prefixes .iter() .any(|prefix| tool.starts_with(prefix)) @@ -400,7 +406,10 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { let output_buf = Arc::new(Mutex::new(RingBuf::default())); let join_handle = if tool.starts_with(MCP_INVOKE_META_FUNCTION_NAME_PREFIX) { - let server = tool.replace(&format!("{MCP_INVOKE_META_FUNCTION_NAME_PREFIX}_"), ""); + let server = tool + .strip_prefix(&format!("{MCP_INVOKE_META_FUNCTION_NAME_PREFIX}_")) + .ok_or_else(|| anyhow!("Malformed MCP invoke function name: {tool}"))? + .to_string(); let Some(server_handle) = ctx.tool_scope.mcp_runtime.get(&server) else { return Ok(json!({ "status": "error", @@ -597,7 +606,25 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { })); }; - let joined = (&mut handle.join_handle).await; + // Ctrl-C/exit teardown SIGTERMs the group without escalating, so a + // TERM-ignoring process would hang this join forever. Bound it and + // escalate to a group SIGKILL, gated on the pgid still being set. + let joined = match time::timeout(JOB_KILL_GRACE, &mut handle.join_handle).await { + Ok(joined) => joined, + Err(_) => { + #[cfg(unix)] + if let Some(pgid) = handle.state.lock().pgid { + unsafe { libc::killpg(pgid, libc::SIGKILL) }; + } + match time::timeout(JOB_KILL_GRACE, &mut handle.join_handle).await { + Ok(joined) => joined, + Err(_) => { + handle.join_handle.abort(); + (&mut handle.join_handle).await + } + } + } + }; let tool = handle.tool.clone(); let elapsed_secs = handle.started_at.elapsed().as_secs(); let status = handle.state.lock().status; @@ -692,7 +719,9 @@ async fn handle_cancel(ctx: &RequestContext, args: &Value) -> Result { /// group kill is gated on `state.pgid` still being set: the job task clears /// it right after `wait()` reaps the child, and killing after the reap could /// signal an innocent recycled pid. MCP jobs (no pgid) fall through to a -/// plain task abort. +/// plain task abort. Only the SIGKILL is re-gated; the SIGTERM fires after +/// the pgid read drops the lock, so a reap in that window could still hit a +/// recycled pid — accepted residual risk. async fn kill_job_with_grace(handle: &mut JobHandle) { #[cfg(unix)] { @@ -835,11 +864,40 @@ async fn pump_into_ring(mut reader: impl AsyncReadExt + Unpin, output_buf: Arc); + +impl Drop for TempFileGuard { + fn drop(&mut self) { + for path in &self.0 { + let _ = fs::remove_file(path); + } + } +} + +/// A grandchild that inherits the pipes keeps them open past the child's +/// exit; don't let that hold the job task past its own timeout. +async fn drain_pump(mut pump: tokio::task::JoinHandle<()>) { + if time::timeout(JOB_PUMP_DRAIN_GRACE, &mut pump) + .await + .is_err() + { + pump.abort(); + } +} + async fn run_process_job( snapshot: JobEnvSnapshot, state: Arc>, output_buf: Arc>, ) -> Result { + let mut temp_files = vec![snapshot.output_file.clone()]; + if let Some(tool_data_file) = snapshot.envs.get("LLM_TOOL_DATA_FILE") { + temp_files.push(PathBuf::from(tool_data_file)); + } + let _temp_guard = TempFileGuard(temp_files); + let mut command = tokio::process::Command::new(&snapshot.cmd_name); command .args(&snapshot.cmd_args) @@ -877,8 +935,8 @@ async fn run_process_job( Err(_) => { kill_expired_job(&mut child, &state).await; state.lock().pgid = None; - let _ = stdout_pump.await; - let _ = stderr_pump.await; + drain_pump(stdout_pump).await; + drain_pump(stderr_pump).await; let output_bytes_captured = output_buf.lock().total_written(); let message = format!( "Tool call '{}' timed out after {}s and was killed (set COYOTE_TOOL_TIMEOUT to adjust; 0 = unlimited)", @@ -894,22 +952,33 @@ async fn run_process_job( } else { child.wait().await }; - let status = - wait_result.map_err(|err| anyhow!("Unable to run {}, {err}", snapshot.display_name))?; + let status = match wait_result { + Ok(status) => status, + Err(err) => { + stdout_pump.abort(); + stderr_pump.abort(); + bail!("Unable to run {}, {err}", snapshot.display_name); + } + }; // pid-reuse guard: the child is reaped, so a later group kill against // this pgid could hit an innocent recycled pid. state.lock().pgid = None; - let _ = stdout_pump.await; - let _ = stderr_pump.await; + drain_pump(stdout_pump).await; + drain_pump(stderr_pump).await; let output_bytes_captured = output_buf.lock().total_written(); let exit_code = status.code(); - if exit_code.unwrap_or_default() != 0 { - let message = format!( - "Tool call '{}' exited with code {}", - snapshot.display_name, - exit_code.unwrap_or_default() - ); + if exit_code != Some(0) { + let message = match exit_code { + Some(code) => format!( + "Tool call '{}' exited with code {code}", + snapshot.display_name + ), + None => format!( + "Tool call '{}' was terminated by a signal", + snapshot.display_name + ), + }; let mut error_json = json!({"tool_call_error": message}); if let Ok(contents) = fs::read_to_string(&snapshot.output_file) && !contents.trim().is_empty() @@ -941,6 +1010,9 @@ async fn run_process_job( }) } +/// Same kill discipline as `kill_job_with_grace`, driven through the owned +/// `Child`. The SIGTERM here fires after the pgid read drops the lock and is +/// not re-gated — the same accepted pid-reuse window. async fn kill_expired_job(child: &mut tokio::process::Child, state: &Arc>) { #[cfg(unix)] { @@ -1267,12 +1339,45 @@ mod tests { } } + #[test] + fn whitelist_rejects_fast_file_builtins() { + for tool in ["fs_read", "fs_cat", "fs_grep", "ast_grep"] { + let message = whitelist_rejection(tool).unwrap()["message"] + .as_str() + .unwrap() + .to_string(); + assert!( + message.contains("is fast"), + "unexpected message for {tool}: {message}" + ); + } + } + #[test] fn whitelist_allows_external_and_mcp_invoke_tools() { assert!(whitelist_rejection("execute_command").is_none()); assert!(whitelist_rejection("mcp_invoke_github").is_none()); } + #[test] + fn handle_start_rejects_fast_builtins_without_spawn() { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("fs_read".into()); + ctx.declared_function_names.insert("ast_grep".into()); + + for tool in ["fs_read", "ast_grep"] { + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": tool, "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!(result["message"].as_str().unwrap().contains("is fast")); + } + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + #[test] fn handle_start_rejects_non_whitelisted_tool_without_spawn() { let mut ctx = plain_ctx(); @@ -1607,7 +1712,7 @@ mod tests { while state.lock().pgid.is_none() && time::Instant::now() < deadline { time::sleep(Duration::from_millis(10)).await; } - assert!(state.lock().pgid.is_some(), "runner must record the pgid"); + let pgid = state.lock().pgid.expect("runner must record the pgid"); let handle = JobHandle { id: "j1".to_string(), @@ -1631,6 +1736,15 @@ mod tests { assert_eq!(result["status"], "cancelled"); assert_eq!(result["tool"], "sleep"); assert!(!ctx.supervisor.as_ref().unwrap().read().has_job("j1")); + assert_eq!( + unsafe { libc::killpg(pgid, 0) }, + -1, + "process group must be dead after cancel" + ); + assert_eq!( + std::io::Error::last_os_error().raw_os_error(), + Some(libc::ESRCH) + ); }); } @@ -1777,6 +1891,125 @@ mod tests { }); } + #[cfg(unix)] + #[test] + fn run_process_job_reports_signal_death_as_error() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("sh", &["-c", "kill -KILL $$"], 0); + + let result = run_process_job(snapshot, state, output_buf).await.unwrap(); + + assert_eq!(result.exit_code, None); + assert!( + result.output["tool_call_error"] + .as_str() + .unwrap() + .contains("terminated by a signal") + ); + }); + } + + #[cfg(unix)] + #[test] + fn run_process_job_removes_output_temp_file() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("sh", &["-c", "printf hi > \"$LLM_OUTPUT\""], 0); + let output_file = snapshot.output_file.clone(); + + let result = run_process_job(snapshot, state, output_buf).await.unwrap(); + + assert_eq!(result.exit_code, Some(0)); + assert_eq!(result.output, json!({"output": "hi"})); + assert!(!output_file.exists(), "temp file must be removed"); + }); + } + + #[cfg(unix)] + #[test] + fn handle_collect_returns_after_term_ignoring_job_is_aborted() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = + test_snapshot("sh", &["-c", "trap '' TERM; while :; do sleep 1; done"], 0); + let task_state = Arc::clone(&state); + let task_buf = Arc::clone(&output_buf); + let join_handle = tokio::spawn(async move { + let result = run_process_job(snapshot, Arc::clone(&task_state), task_buf).await; + let mut job_state = task_state.lock(); + job_state.pgid = None; + job_state.status = match &result { + Ok(job_result) if job_result.exit_code == Some(0) => JobStatus::Completed, + _ => JobStatus::Failed, + }; + drop(job_state); + result + }); + + let deadline = time::Instant::now() + Duration::from_secs(2); + while state.lock().pgid.is_none() && time::Instant::now() < deadline { + time::sleep(Duration::from_millis(10)).await; + } + let pgid = state.lock().pgid.expect("runner must record the pgid"); + + let abort_signal = create_abort_signal(); + let handle = JobHandle { + id: "j1".to_string(), + tool: "sh".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: abort_signal.clone(), + state: Arc::clone(&state), + output_buf, + no_change_checks: 0, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + // Mimic Ctrl-C teardown: SIGTERM the group and flag the abort + // signal, leaving the handle registered. + abort_signal.set_ctrlc(); + unsafe { libc::killpg(pgid, libc::SIGTERM) }; + + let result = time::timeout( + Duration::from_secs(20), + handle_collect(&ctx, &json!({"id": "j1"})), + ) + .await + .expect("collect must not hang on a TERM-ignoring job") + .unwrap(); + + assert_eq!(result["status"], "failed"); + // killpg(pgid, 0) is unreliable here: the orphaned sleep + // grandchild lingers as an unreaped zombie and keeps the group + // id alive. Signal death proves the escalated SIGKILL landed. + assert!( + result["result"]["tool_call_error"] + .as_str() + .unwrap() + .contains("terminated by a signal") + ); + }); + } + #[test] fn jobs_only_supervisor_rejects_agent_spawn_at_capacity_zero() { let mut ctx = ctx_with_job_supervisor(5); From 6a694d10db09af51037d69421d5a231601d1405b Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 17:38:57 -0600 Subject: [PATCH 11/28] test(jobs): assert kind-aware guardrail surfaces running jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconciles the T2 guardrail delta with the T3 jobs-only-supervisor test at merge time, per plans/background-jobs-design.md §11 merge order. --- src/function/jobs.rs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/function/jobs.rs b/src/function/jobs.rs index bc36f0e..480c46e 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -2031,7 +2031,7 @@ mod tests { } #[test] - fn jobs_only_supervisor_guardrail_takes_no_action() { + fn jobs_only_supervisor_guardrail_surfaces_running_job() { let mut ctx = ctx_with_job_supervisor(5); ctx.supervisor .as_ref() @@ -2040,8 +2040,18 @@ mod tests { .register(make_running_job("j1")) .unwrap(); + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::Inject(prompt) => { + assert!(prompt.contains("j1")); + assert!(prompt.contains("job__collect")); + } + _ => panic!("expected Inject for a running job"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 1); + + let empty_ctx = &mut ctx_with_job_supervisor(5); assert!(matches!( - check_pending_agents_guardrail(&mut ctx), + check_pending_agents_guardrail(empty_ctx), GuardrailAction::NoAction )); } From 2d874f1d7cfb3db630c1aa97315ee5545d905517 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 17:50:28 -0600 Subject: [PATCH 12/28] feat(jobs): push background-job completion notifications via per-context queue - add NotificationQueue/SystemNotification: every context owns a fresh queue (children never inherit the parent's, avoiding first-drainer-wins races between transcripts) - job tasks push job_completed/job_failed events on completion, failure, and timeout; a panic skips the push and is surfaced by the guardrail's finished-handle enumeration and collect's JoinError mapping instead - events for jobs already collected or cancelled are dropped at drain time by filtering against live supervisor registration - replace inject_escalation_notification with single-pass merge_system_channel: pending_escalations (root-only) ordered before system_notifications (any depth) on the last tool result of a batch; byte-identical output when notifications are empty, proven by the unmodified pre-merger characterization tests --- src/config/request_context.rs | 34 ++++ src/function/jobs.rs | 151 +++++++++++++++++- src/function/mod.rs | 284 ++++++++++++++++++++++++++++++--- src/supervisor/mod.rs | 1 + src/supervisor/notification.rs | 128 +++++++++++++++ 5 files changed, 571 insertions(+), 27 deletions(-) create mode 100644 src/supervisor/notification.rs diff --git a/src/config/request_context.rs b/src/config/request_context.rs index b5b26bb..431177f 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -34,6 +34,7 @@ use crate::rag::Rag; use crate::supervisor::Supervisor; use crate::supervisor::escalation::EscalationQueue; use crate::supervisor::mailbox::Inbox; +use crate::supervisor::notification::NotificationQueue; use crate::utils::{ AbortSignal, abortable_run_with_spinner, edit_file, fuzzy_filter, get_env_name, list_file_names, now, render_prompt, temp_file, @@ -330,6 +331,7 @@ pub struct RequestContext { pub self_agent_id: Option, pub inbox: Option>, pub escalation_queue: Option>, + pub notification_queue: Arc, pub current_depth: usize, pub auto_continue_count: usize, pub pending_agents_guardrail_count: u32, @@ -364,6 +366,7 @@ impl RequestContext { self_agent_id: None, inbox: None, escalation_queue: None, + notification_queue: Arc::new(NotificationQueue::new()), current_depth: 0, auto_continue_count: 0, pending_agents_guardrail_count: 0, @@ -424,6 +427,7 @@ impl RequestContext { self_agent_id: None, inbox: None, escalation_queue: None, + notification_queue: Arc::new(NotificationQueue::new()), current_depth: 0, auto_continue_count: 0, pending_agents_guardrail_count: 0, @@ -471,6 +475,7 @@ impl RequestContext { self_agent_id: self.self_agent_id.clone(), inbox: self.inbox.clone(), escalation_queue: self.escalation_queue.clone(), + notification_queue: self.notification_queue.clone(), current_depth: self.current_depth, auto_continue_count: 0, pending_agents_guardrail_count: 0, @@ -516,6 +521,7 @@ impl RequestContext { self_agent_id: Some(self_agent_id), inbox: Some(inbox), escalation_queue: parent.escalation_queue.clone(), + notification_queue: Arc::new(NotificationQueue::new()), current_depth, auto_continue_count: 0, pending_agents_guardrail_count: 0, @@ -4202,6 +4208,7 @@ impl RequestContext { self.supervisor = supervisor; self.inbox = None; self.escalation_queue = None; + self.notification_queue = Arc::new(NotificationQueue::new()); self.self_agent_id = None; self.parent_supervisor = None; self.current_depth = 0; @@ -4244,6 +4251,7 @@ impl RequestContext { self.self_agent_id = None; self.inbox = None; self.escalation_queue = None; + self.notification_queue = Arc::new(NotificationQueue::new()); self.current_depth = 0; self.auto_continue_count = 0; self.pending_agents_guardrail_count = 0; @@ -5463,6 +5471,32 @@ mod tests { assert!(ctx.root_escalation_queue().is_none()); } + #[test] + fn new_for_child_gets_fresh_notification_queue() { + let parent = create_test_ctx(); + let child = RequestContext::new_for_child( + Arc::clone(&parent.app), + &parent, + 1, + Arc::new(Inbox::new()), + "agent_test_1".to_string(), + ); + assert!( + !Arc::ptr_eq(&parent.notification_queue, &child.notification_queue), + "each child owns its notifications; a shared queue would race drains" + ); + } + + #[test] + fn fork_for_branch_shares_notification_queue() { + let ctx = create_test_ctx(); + let branch = ctx.fork_for_branch(); + assert!(Arc::ptr_eq( + &ctx.notification_queue, + &branch.notification_queue + )); + } + fn app_state_with_mcp_config(mcp_server_support: bool, server_names: &[&str]) -> Arc { app_state_with_mcp_command(mcp_server_support, server_names, "echo") } diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 480c46e..b2fd2f5 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -14,6 +14,7 @@ use crate::mcp::{ MCP_PROMPT_META_FUNCTION_NAME_PREFIX, MCP_READ_META_FUNCTION_NAME_PREFIX, MCP_SEARCH_META_FUNCTION_NAME_PREFIX, }; +use crate::supervisor::notification::job_notification; use crate::supervisor::{JobHandle, JobResult, JobState, JobStatus, Supervisor}; use crate::utils::{create_abort_signal, muted_warning_text, temp_file, wait_abort_signal}; @@ -432,27 +433,39 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { current_depth: ctx.current_depth, }; let task_state = Arc::clone(&state); + let task_notifications = Arc::clone(&ctx.notification_queue); + let notify_id = job_id.clone(); + let notify_tool = tool.clone(); tokio::spawn(async move { let result = run_mcp_job(job_ctx, server, inner_tool, inner_args).await; - task_state.lock().status = match &result { - Ok(_) => JobStatus::Completed, - Err(_) => JobStatus::Failed, + let success = result.is_ok(); + task_state.lock().status = if success { + JobStatus::Completed + } else { + JobStatus::Failed }; + task_notifications.push(job_notification(¬ify_id, ¬ify_tool, success)); result }) } else { let snapshot = build_env_snapshot(ctx, &tool, &arguments)?; let task_state = Arc::clone(&state); let task_buf = Arc::clone(&output_buf); + let task_notifications = Arc::clone(&ctx.notification_queue); + let notify_id = job_id.clone(); + let notify_tool = tool.clone(); tokio::spawn(async move { let result = run_process_job(snapshot, Arc::clone(&task_state), task_buf).await; + let success = matches!(&result, Ok(job_result) if job_result.exit_code == Some(0)); let mut job_state = task_state.lock(); job_state.pgid = None; - job_state.status = match &result { - Ok(job_result) if job_result.exit_code == Some(0) => JobStatus::Completed, - _ => JobStatus::Failed, + job_state.status = if success { + JobStatus::Completed + } else { + JobStatus::Failed }; drop(job_state); + task_notifications.push(job_notification(¬ify_id, ¬ify_tool, success)); result }) }; @@ -1533,6 +1546,132 @@ mod tests { ); } + #[cfg(unix)] + #[test] + fn job_completion_pushes_notification_for_own_context() { + run_async(async { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("echo".into()); + + let started = handle_start(&mut ctx, &json!({"tool": "echo", "arguments": {}})) + .await + .unwrap(); + let job_id = started["job_id"].as_str().unwrap().to_string(); + + let supervisor = ctx.supervisor.clone().unwrap(); + let deadline = time::Instant::now() + Duration::from_secs(5); + while time::Instant::now() < deadline { + let finished = supervisor + .read() + .job(&job_id) + .is_none_or(|job| job.join_handle.is_finished()); + if finished { + break; + } + time::sleep(Duration::from_millis(10)).await; + } + + let events = ctx.notification_queue.drain(); + assert_eq!(events.len(), 1); + assert_eq!(events[0].event, "job_completed"); + assert_eq!(events[0].id, job_id); + assert_eq!(events[0].tool_or_agent, "echo"); + assert_eq!(events[0].status, "success"); + assert_eq!( + events[0].next_action, + format!("job__collect --id {job_id} for output") + ); + }); + } + + #[cfg(unix)] + #[test] + fn job_failure_pushes_failed_notification() { + run_async(async { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("false".into()); + + let started = handle_start(&mut ctx, &json!({"tool": "false", "arguments": {}})) + .await + .unwrap(); + let job_id = started["job_id"].as_str().unwrap().to_string(); + + let supervisor = ctx.supervisor.clone().unwrap(); + let deadline = time::Instant::now() + Duration::from_secs(5); + while time::Instant::now() < deadline { + let finished = supervisor + .read() + .job(&job_id) + .is_none_or(|job| job.join_handle.is_finished()); + if finished { + break; + } + time::sleep(Duration::from_millis(10)).await; + } + + let events = ctx.notification_queue.drain(); + assert_eq!(events.len(), 1); + assert_eq!(events[0].event, "job_failed"); + assert_eq!(events[0].status, "failed"); + }); + } + + #[test] + fn cancelled_job_notification_is_suppressed_at_drain() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let join_handle = tokio::spawn(async { + time::sleep(Duration::from_secs(30)).await; + Ok(JobResult { + output: Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + let handle = JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + ctx.notification_queue + .push(job_notification("j1", "execute_command", false)); + + let result = handle_cancel(&ctx, &json!({"id": "j1"})).await.unwrap(); + + assert_eq!(result["status"], "cancelled"); + assert!( + super::super::drain_live_notifications(&ctx).is_empty(), + "events for a cancelled job must never reach the model" + ); + }); + } + + #[test] + fn already_collected_job_notification_is_suppressed_at_drain() { + let ctx = ctx_with_job_supervisor(4); + ctx.notification_queue + .push(job_notification("j1", "execute_command", true)); + + assert!( + super::super::drain_live_notifications(&ctx).is_empty(), + "events for an already-collected job must be dropped" + ); + } + #[test] fn job_handlers_teach_cross_kind_for_agent_ids() { let ctx = ctx_with_job_supervisor(4); diff --git a/src/function/mod.rs b/src/function/mod.rs index 78ee969..4555dca 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -341,12 +341,17 @@ pub async fn eval_tool_calls( } } - if ctx.current_depth == 0 - && let Some(queue) = ctx.root_escalation_queue() - && queue.has_pending() - && let Some(last) = output.last_mut() - { - inject_escalation_notification(last, queue.pending_summary()); + if let Some(last) = output.last_mut() { + let escalations = if ctx.current_depth == 0 { + ctx.root_escalation_queue() + .filter(|queue| queue.has_pending()) + .map(|queue| queue.pending_summary()) + .unwrap_or_default() + } else { + vec![] + }; + let notifications = drain_live_notifications(ctx); + merge_system_channel(last, escalations, notifications); } Ok(output) @@ -365,22 +370,75 @@ fn normalize_tool_result(result: Value) -> Value { } } -fn inject_escalation_notification(last: &mut ToolResult, summary: Vec) { - let instruction = "Child agents are BLOCKED waiting for your reply. \ - Call agent__reply_escalation for each pending escalation to unblock them."; - match &mut last.output { - Value::Object(map) => { - map.insert("pending_escalations".into(), json!(summary)); - map.insert("escalation_instruction".into(), json!(instruction)); - } - other => { - *other = json!({ - "output": other.take(), - "pending_escalations": summary, - "escalation_instruction": instruction, - }); - } +/// Drains this context's own notification queue and drops events whose +/// handle is no longer registered with the supervisor (already collected or +/// cancelled), so the model is never pointed at a dead id. +fn drain_live_notifications(ctx: &RequestContext) -> Vec { + let events = ctx.notification_queue.drain(); + if events.is_empty() { + return vec![]; } + let Some(supervisor) = ctx.supervisor.as_ref() else { + return vec![]; + }; + let sup = supervisor.read(); + events + .into_iter() + .filter(|event| sup.has_job(&event.id) || sup.has_agent(&event.id)) + .map(|event| event.to_value()) + .collect() +} + +/// Single-pass merge of both system channels onto the last tool result of a +/// batch: pending escalations (children are blocked; listed first) and +/// background-task completion notifications. A single pass is mandatory — +/// two independent mergers would each apply the non-object wrap and nest the +/// output twice. With both channels empty this is a no-op, and with only +/// escalations it produces exactly the pre-notification output shape. +fn merge_system_channel(last: &mut ToolResult, escalations: Vec, notifications: Vec) { + if escalations.is_empty() && notifications.is_empty() { + return; + } + let escalation_instruction = "Child agents are BLOCKED waiting for your reply. \ + Call agent__reply_escalation for each pending escalation to unblock them."; + let notification_instruction = + "Background tasks have finished; collect each result with its next_action command."; + + let map = match &mut last.output { + Value::Object(map) => map, + other => { + let mut map = serde_json::Map::new(); + map.insert("output".into(), other.take()); + *other = Value::Object(map); + match other { + Value::Object(map) => map, + _ => unreachable!(), + } + } + }; + if !escalations.is_empty() { + map.insert("pending_escalations".into(), json!(escalations)); + map.insert( + "escalation_instruction".into(), + json!(escalation_instruction), + ); + } + if !notifications.is_empty() { + map.insert("system_notifications".into(), json!(notifications)); + map.insert( + "notification_instruction".into(), + json!(notification_instruction), + ); + } +} + +/// Escalation-only entry point retained so the characterization tests that +/// pinned the pre-merger output shape keep proving, unmodified, that +/// `merge_system_channel` with no notifications is byte-identical to the +/// injection behavior they were written against. +#[cfg(test)] +fn inject_escalation_notification(last: &mut ToolResult, summary: Vec) { + merge_system_channel(last, summary, vec![]); } #[derive(Debug, Clone, Deserialize, Serialize)] @@ -2495,6 +2553,7 @@ mod tests { }; use crate::config::{Agent, AgentConfig, AppConfig, AppState, WorkingMode}; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; + use crate::supervisor::notification::job_notification; use base64::Engine; use base64::engine::general_purpose::STANDARD; use rmcp::model::{CallToolResult, ContentBlock}; @@ -2579,6 +2638,189 @@ mod tests { assert!(result.output["escalation_instruction"].is_string()); } + fn ctx_with_registered_job(id: &str) -> RequestContext { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let join_handle = rt.spawn(async { + Ok(crate::supervisor::JobResult { + output: Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + let handle = crate::supervisor::JobHandle { + id: id.to_string(), + tool: "execute_command".to_string(), + started_at: std::time::Instant::now(), + join_handle, + abort_signal: crate::utils::create_abort_signal(), + state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { + status: crate::supervisor::JobStatus::Completed, + pgid: None, + })), + output_buf: Arc::new(parking_lot::Mutex::new(jobs::RingBuf::default())), + no_change_checks: 0, + }; + let mut sup = crate::supervisor::Supervisor::new(0, 3).with_max_concurrent_jobs(4); + sup.register(handle).unwrap(); + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); + ctx + } + + #[test] + fn merge_system_channel_noop_when_both_channels_empty() { + let mut object_result = ToolResult::new(call("t", Some("id-1")), json!({"status": "ok"})); + merge_system_channel(&mut object_result, vec![], vec![]); + assert_eq!(object_result.output, json!({"status": "ok"})); + + let mut plain_result = ToolResult::new(call("t", Some("id-1")), json!("DONE")); + merge_system_channel(&mut plain_result, vec![], vec![]); + assert_eq!(plain_result.output, json!("DONE")); + } + + #[test] + fn merge_system_channel_escalations_only_matches_legacy_wrap_bytes() { + let summary = vec![json!({"escalation_id": "esc_1"})]; + let expected = json!({ + "output": "DONE", + "pending_escalations": summary, + "escalation_instruction": "Child agents are BLOCKED waiting for your reply. \ + Call agent__reply_escalation for each pending escalation to unblock them.", + }); + + let mut result = ToolResult::new(call("t", Some("id-1")), json!("DONE")); + merge_system_channel(&mut result, summary, vec![]); + + assert_eq!( + serde_json::to_string(&result.output).unwrap(), + serde_json::to_string(&expected).unwrap() + ); + } + + #[test] + fn merge_system_channel_adds_notifications_without_escalation_keys() { + let mut result = ToolResult::new(call("t", Some("id-1")), json!({"status": "ok"})); + merge_system_channel(&mut result, vec![], vec![json!({"id": "job_1"})]); + assert_eq!(result.output["status"], "ok"); + assert_eq!( + result.output["system_notifications"], + json!([{"id": "job_1"}]) + ); + assert!( + result.output["notification_instruction"] + .as_str() + .unwrap() + .contains("next_action") + ); + assert!(result.output.get("pending_escalations").is_none()); + assert!(result.output.get("escalation_instruction").is_none()); + } + + #[test] + fn merge_system_channel_wraps_non_object_once_with_both_channels() { + let mut result = ToolResult::new(call("t", Some("id-1")), json!("DONE")); + merge_system_channel( + &mut result, + vec![json!({"escalation_id": "esc_1"})], + vec![json!({"id": "job_1"})], + ); + assert_eq!(result.output["output"], json!("DONE")); + assert_eq!( + result.output["pending_escalations"][0]["escalation_id"], + "esc_1" + ); + assert_eq!(result.output["system_notifications"][0]["id"], "job_1"); + let keys: Vec<&str> = result + .output + .as_object() + .unwrap() + .keys() + .map(|k| k.as_str()) + .collect(); + assert_eq!( + keys, + vec![ + "output", + "pending_escalations", + "escalation_instruction", + "system_notifications", + "notification_instruction" + ] + ); + } + + #[test] + fn drain_live_notifications_drops_unregistered_ids() { + let ctx = ctx_with_registered_job("job_live"); + ctx.notification_queue + .push(job_notification("job_live", "execute_command", true)); + ctx.notification_queue + .push(job_notification("job_gone", "execute_command", true)); + + let live = drain_live_notifications(&ctx); + + assert_eq!(live.len(), 1); + assert_eq!(live[0]["id"], "job_live"); + assert!( + ctx.notification_queue.drain().is_empty(), + "drain must consume the queue" + ); + } + + #[test] + fn drain_live_notifications_without_supervisor_drops_everything() { + let ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.notification_queue + .push(job_notification("job_x", "execute_command", true)); + assert!(drain_live_notifications(&ctx).is_empty()); + } + + #[test] + fn eval_tool_calls_merges_notifications_at_depth_without_escalations() { + let mut ctx = ctx_with_registered_job("job_n1"); + ctx.current_depth = 1; + let queue = ctx.ensure_root_escalation_queue(); + submit_escalation(&queue, "esc_1"); + ctx.notification_queue + .push(job_notification("job_n1", "execute_command", true)); + + let calls = vec![call("unknown_tool", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + let out = &results[0].output; + assert_eq!(out["system_notifications"][0]["id"], "job_n1"); + assert_eq!(out["system_notifications"][0]["event"], "job_completed"); + assert!(out["notification_instruction"].is_string()); + assert!( + out.get("pending_escalations").is_none(), + "escalations are root-only" + ); + } + + #[test] + fn eval_tool_calls_merges_both_channels_onto_last_result() { + let mut ctx = ctx_with_registered_job("job_n1"); + let queue = ctx.ensure_root_escalation_queue(); + submit_escalation(&queue, "esc_1"); + ctx.notification_queue + .push(job_notification("job_n1", "execute_command", false)); + + let calls = vec![call("unknown_tool", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + + let out = &results[0].output; + assert_eq!(out["pending_escalations"][0]["escalation_id"], "esc_1"); + assert_eq!(out["system_notifications"][0]["event"], "job_failed"); + assert!( + out.get("output").is_none(), + "object outputs are extended in place, never wrapped" + ); + } + #[test] fn eval_tool_calls_soft_fails_unknown_tool() { let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index ac143b5..05ececb 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -1,5 +1,6 @@ pub mod escalation; pub mod mailbox; +pub mod notification; pub mod taskqueue; use crate::function::jobs::RingBuf; diff --git a/src/supervisor/notification.rs b/src/supervisor/notification.rs new file mode 100644 index 0000000..7f9d830 --- /dev/null +++ b/src/supervisor/notification.rs @@ -0,0 +1,128 @@ +use fmt::{Debug, Formatter}; +use serde_json::{Value, json}; +use std::fmt; + +/// One background-task completion event, delivered to the context that +/// started the task by merging a `system_notifications` entry onto the last +/// tool result of a batch. +#[derive(Clone)] +pub struct SystemNotification { + pub event: &'static str, + pub id: String, + pub tool_or_agent: String, + pub status: &'static str, + pub next_action: String, +} + +impl SystemNotification { + pub fn to_value(&self) -> Value { + json!({ + "event": self.event, + "id": self.id, + "tool_or_agent": self.tool_or_agent, + "status": self.status, + "next_action": self.next_action, + }) + } +} + +pub fn job_notification(id: &str, tool: &str, success: bool) -> SystemNotification { + SystemNotification { + event: if success { + "job_completed" + } else { + "job_failed" + }, + id: id.to_string(), + tool_or_agent: tool.to_string(), + status: if success { "success" } else { "failed" }, + next_action: format!("job__collect --id {id} for output"), + } +} + +/// Completion events for background work started by ONE context. Unlike the +/// escalation queue (shared, root-owned), every context owns a fresh queue: +/// a queue shared between parent and child would race their drains and +/// deliver one context's events into the other's transcript. +pub struct NotificationQueue { + pending: parking_lot::Mutex>, +} + +impl NotificationQueue { + pub fn new() -> Self { + Self { + pending: parking_lot::Mutex::new(Vec::new()), + } + } + + pub fn push(&self, notification: SystemNotification) { + self.pending.lock().push(notification); + } + + pub fn drain(&self) -> Vec { + std::mem::take(&mut *self.pending.lock()) + } +} + +impl Default for NotificationQueue { + fn default() -> Self { + Self::new() + } +} + +impl Debug for NotificationQueue { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + let count = self.pending.lock().len(); + f.debug_struct("NotificationQueue") + .field("pending_count", &count) + .finish() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn job_notification_success_shape() { + let event = job_notification("job_a1b2", "execute_command", true); + assert_eq!( + event.to_value(), + json!({ + "event": "job_completed", + "id": "job_a1b2", + "tool_or_agent": "execute_command", + "status": "success", + "next_action": "job__collect --id job_a1b2 for output", + }) + ); + } + + #[test] + fn job_notification_failure_shape() { + let event = job_notification("job_a1b2", "execute_command", false); + assert_eq!(event.event, "job_failed"); + assert_eq!(event.status, "failed"); + assert_eq!(event.next_action, "job__collect --id job_a1b2 for output"); + } + + #[test] + fn drain_empties_queue_and_preserves_order() { + let queue = NotificationQueue::new(); + queue.push(job_notification("job_1", "execute_command", true)); + queue.push(job_notification("job_2", "execute_command", false)); + + let drained = queue.drain(); + + assert_eq!(drained.len(), 2); + assert_eq!(drained[0].id, "job_1"); + assert_eq!(drained[1].id, "job_2"); + assert!(queue.drain().is_empty()); + } + + #[test] + fn drain_on_empty_queue_is_a_noop() { + let queue = NotificationQueue::default(); + assert!(queue.drain().is_empty()); + } +} From caabf41b65f46ec5d7dea07cbacf1609bb703206 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 17:52:19 -0600 Subject: [PATCH 13/28] feat(supervisor): push agent completion notifications to the spawning context The spawned-agent task now pushes an agent_completed/agent_failed event into the spawning context's notification queue before returning, so a parent that keeps working learns mid-turn that a child finished instead of discovering it only at the turn-end guardrail. Cancelled or already-collected agents are suppressed by the existing drain-time registration filter. This delivery applies regardless of whether background jobs are enabled. --- src/function/mod.rs | 64 +++++++++++++++++++++++++++++++++- src/function/supervisor.rs | 21 +++++++---- src/supervisor/notification.rs | 40 +++++++++++++++++++++ 3 files changed, 118 insertions(+), 7 deletions(-) diff --git a/src/function/mod.rs b/src/function/mod.rs index 4555dca..5b1b614 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -2553,7 +2553,7 @@ mod tests { }; use crate::config::{Agent, AgentConfig, AppConfig, AppState, WorkingMode}; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; - use crate::supervisor::notification::job_notification; + use crate::supervisor::notification::{agent_notification, job_notification}; use base64::Engine; use base64::engine::general_purpose::STANDARD; use rmcp::model::{CallToolResult, ContentBlock}; @@ -2779,6 +2779,68 @@ mod tests { assert!(drain_live_notifications(&ctx).is_empty()); } + fn ctx_with_registered_agent(id: &str) -> RequestContext { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let agent_id = id.to_string(); + let join_handle = rt.spawn(async move { + Ok(crate::supervisor::AgentResult { + id: agent_id, + agent_name: "explore".into(), + output: String::new(), + exit_status: crate::supervisor::AgentExitStatus::Completed, + }) + }); + std::mem::forget(rt); + let handle = crate::supervisor::AgentHandle { + id: id.to_string(), + agent_name: "explore".to_string(), + depth: 1, + inbox: Arc::new(crate::supervisor::mailbox::Inbox::new()), + abort_signal: crate::utils::create_abort_signal(), + join_handle, + child_supervisor: None, + }; + let mut sup = crate::supervisor::Supervisor::new(4, 3); + sup.register(handle).unwrap(); + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); + ctx + } + + #[test] + fn drain_live_notifications_keeps_registered_agent_events() { + let ctx = ctx_with_registered_agent("agent_explore_1"); + ctx.notification_queue + .push(agent_notification("agent_explore_1", "explore", true)); + + let live = drain_live_notifications(&ctx); + + assert_eq!(live.len(), 1); + assert_eq!(live[0]["event"], "agent_completed"); + assert_eq!( + live[0]["next_action"], + "agent__collect --id agent_explore_1 for output" + ); + } + + #[test] + fn drain_live_notifications_drops_collected_agent_events() { + let ctx = ctx_with_registered_agent("agent_explore_1"); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .take("agent_explore_1") + .unwrap(); + ctx.notification_queue + .push(agent_notification("agent_explore_1", "explore", true)); + + assert!(drain_live_notifications(&ctx).is_empty()); + } + #[test] fn eval_tool_calls_merges_notifications_at_depth_without_escalations() { let mut ctx = ctx_with_registered_job("job_n1"); diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index 2e766e2..fee6e3e 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -5,6 +5,7 @@ use crate::config::{ jobs_enabled, list_agents_with_descriptions, }; use crate::supervisor::mailbox::{Envelope, EnvelopePayload, Inbox}; +use crate::supervisor::notification::agent_notification; use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult, Supervisor, TaskKind}; use crate::utils::{AbortSignal, create_abort_signal, wait_abort_signal}; @@ -866,25 +867,33 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { let spawn_agent_id = agent_id.clone(); let spawn_agent_name = agent_name.clone(); let spawn_abort = child_abort.clone(); + let spawn_notifications = Arc::clone(&ctx.notification_queue); let child_supervisor = child_ctx.supervisor.clone(); let join_handle = tokio::spawn(async move { let result = run_child_agent(child_ctx, input, spawn_abort).await; - match result { - Ok(output) => Ok(AgentResult { + let agent_result = match result { + Ok(output) => AgentResult { id: spawn_agent_id, agent_name: spawn_agent_name, output, exit_status: AgentExitStatus::Completed, - }), - Err(e) => Ok(AgentResult { + }, + Err(e) => AgentResult { id: spawn_agent_id, agent_name: spawn_agent_name, output: String::new(), exit_status: AgentExitStatus::Failed(e.to_string()), - }), - } + }, + }; + let success = agent_result.exit_status == AgentExitStatus::Completed; + spawn_notifications.push(agent_notification( + &agent_result.id, + &agent_result.agent_name, + success, + )); + Ok(agent_result) }); let handle = AgentHandle { diff --git a/src/supervisor/notification.rs b/src/supervisor/notification.rs index 7f9d830..4f9261a 100644 --- a/src/supervisor/notification.rs +++ b/src/supervisor/notification.rs @@ -40,6 +40,20 @@ pub fn job_notification(id: &str, tool: &str, success: bool) -> SystemNotificati } } +pub fn agent_notification(id: &str, agent_name: &str, success: bool) -> SystemNotification { + SystemNotification { + event: if success { + "agent_completed" + } else { + "agent_failed" + }, + id: id.to_string(), + tool_or_agent: agent_name.to_string(), + status: if success { "success" } else { "failed" }, + next_action: format!("agent__collect --id {id} for output"), + } +} + /// Completion events for background work started by ONE context. Unlike the /// escalation queue (shared, root-owned), every context owns a fresh queue: /// a queue shared between parent and child would race their drains and @@ -106,6 +120,32 @@ mod tests { assert_eq!(event.next_action, "job__collect --id job_a1b2 for output"); } + #[test] + fn agent_notification_success_shape() { + let event = agent_notification("agent_explore_a1b2", "explore", true); + assert_eq!( + event.to_value(), + json!({ + "event": "agent_completed", + "id": "agent_explore_a1b2", + "tool_or_agent": "explore", + "status": "success", + "next_action": "agent__collect --id agent_explore_a1b2 for output", + }) + ); + } + + #[test] + fn agent_notification_failure_shape() { + let event = agent_notification("agent_explore_a1b2", "explore", false); + assert_eq!(event.event, "agent_failed"); + assert_eq!(event.status, "failed"); + assert_eq!( + event.next_action, + "agent__collect --id agent_explore_a1b2 for output" + ); + } + #[test] fn drain_empties_queue_and_preserves_order() { let queue = NotificationQueue::new(); From 24ed6749528da7838873ca5bf86d6a63a7122138 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 18:17:15 -0600 Subject: [PATCH 14/28] feat(jobs): exempt polling tools from loop tracker and hint on unchanged checks --- src/function/jobs.rs | 103 ++++++++++++++++++++++++++++++++++++- src/function/mod.rs | 60 +++++++++++++++++++++ src/function/supervisor.rs | 1 + src/supervisor/mod.rs | 10 +++- 4 files changed, 171 insertions(+), 3 deletions(-) diff --git a/src/function/jobs.rs b/src/function/jobs.rs index b2fd2f5..f2d6c92 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -479,6 +479,7 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { state, output_buf, no_change_checks: 0, + last_check_state: None, }; // On a capacity race the handle is dropped here, which kills the process @@ -507,8 +508,8 @@ fn handle_check(ctx: &RequestContext, args: &Value) -> Result { let Some(supervisor) = ctx.supervisor.as_ref() else { return Ok(job_miss_error(None, id)); }; - let sup = supervisor.read(); - let Some(job) = sup.job(id) else { + let mut sup = supervisor.write(); + let Some(job) = sup.job_mut(id) else { drop(sup); return Ok(job_miss_error(ctx.supervisor.as_ref(), id)); }; @@ -518,6 +519,13 @@ fn handle_check(ctx: &RequestContext, args: &Value) -> Result { let buf = job.output_buf.lock(); (buf.tail(), buf.total_written()) }; + let check_state = (status, total_written); + if job.last_check_state == Some(check_state) { + job.no_change_checks += 1; + } else { + job.no_change_checks = 0; + job.last_check_state = Some(check_state); + } let tail_truncated = (tail.len() as u64) < total_written; let mut result = json!({ "status": job_status_str(status), @@ -532,6 +540,11 @@ fn handle_check(ctx: &RequestContext, args: &Value) -> Result { result["message"] = json!( "Job is still running. Call job__collect to block for the result, or do other work — you will be notified on completion." ); + if job.no_change_checks >= 3 { + result["hint"] = json!( + "No change across repeated checks — still running; call job__collect to block, or do other work — a system notification will fire on completion." + ); + } } else { result["message"] = json!(format!( "Job finished — retrieve the result with job__collect --id {id}" @@ -1175,6 +1188,7 @@ mod tests { })), output_buf: Arc::new(Mutex::new(RingBuf::default())), no_change_checks: 0, + last_check_state: None, } } @@ -1640,6 +1654,7 @@ mod tests { })), output_buf: Arc::new(Mutex::new(RingBuf::default())), no_change_checks: 0, + last_check_state: None, }; ctx.supervisor .as_ref() @@ -1754,6 +1769,86 @@ mod tests { assert!(ctx.supervisor.as_ref().unwrap().read().has_job("j1")); } + #[test] + fn handle_check_hints_after_repeated_unchanged_checks() { + let ctx = ctx_with_job_supervisor(4); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("j1")) + .unwrap(); + + for _ in 0..3 { + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + assert!(result.get("hint").is_none()); + } + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + assert_eq!( + result["hint"], + "No change across repeated checks — still running; call job__collect to block, or do other work — a system notification will fire on completion." + ); + } + + #[test] + fn handle_check_no_change_counter_resets_on_output_change() { + let ctx = ctx_with_job_supervisor(4); + let job = make_running_job("j1"); + let output_buf = Arc::clone(&job.output_buf); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(job) + .unwrap(); + + for _ in 0..3 { + handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + } + assert!( + handle_check(&ctx, &json!({"id": "j1"})) + .unwrap() + .get("hint") + .is_some() + ); + + output_buf.lock().push(b"more"); + for _ in 0..3 { + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + assert!(result.get("hint").is_none()); + } + assert!( + handle_check(&ctx, &json!({"id": "j1"})) + .unwrap() + .get("hint") + .is_some() + ); + } + + #[test] + fn handle_check_finished_job_never_hints() { + let ctx = ctx_with_job_supervisor(4); + let job = make_running_job("j1"); + job.state.lock().status = JobStatus::Completed; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(job) + .unwrap(); + + for _ in 0..5 { + let result = handle_check(&ctx, &json!({"id": "j1"})).unwrap(); + assert!(result.get("hint").is_none()); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("job__collect --id j1") + ); + } + } + #[test] fn handle_collect_applies_tail_lines() { run_async(async { @@ -1777,6 +1872,7 @@ mod tests { })), output_buf: Arc::new(Mutex::new(RingBuf::default())), no_change_checks: 0, + last_check_state: None, }; ctx.supervisor .as_ref() @@ -1815,6 +1911,7 @@ mod tests { })), output_buf, no_change_checks: 0, + last_check_state: None, }; ctx.supervisor .as_ref() @@ -1862,6 +1959,7 @@ mod tests { state: Arc::clone(&state), output_buf, no_change_checks: 0, + last_check_state: None, }; ctx.supervisor .as_ref() @@ -2115,6 +2213,7 @@ mod tests { state: Arc::clone(&state), output_buf, no_change_checks: 0, + last_check_state: None, }; ctx.supervisor .as_ref() diff --git a/src/function/mod.rs b/src/function/mod.rs index 5b1b614..e116dd0 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -2409,6 +2409,19 @@ fn polyfill_cmd_name>(cmd_name: &str, bin_dir: &[T]) -> String { cmd_name } +// Polling tools are expected to repeat; recording them would also let them +// break up detection of a real loop in the calls they interleave with. +const LOOP_TRACKER_EXEMPT_TOOLS: [&str; 4] = [ + "job__check", + "job__list", + "agent__check", + "agent__list_running", +]; + +fn is_loop_tracker_exempt(name: &str) -> bool { + LOOP_TRACKER_EXEMPT_TOOLS.contains(&name) +} + #[derive(Debug, Clone)] pub struct ToolCallTracker { last_calls: VecDeque, @@ -2430,6 +2443,9 @@ impl ToolCallTracker { } pub fn check_loop(&self, new_call: &ToolCall) -> Option { + if is_loop_tracker_exempt(&new_call.name) { + return None; + } if self.last_calls.len() < self.max_repeats { return None; } @@ -2496,6 +2512,9 @@ impl ToolCallTracker { } pub fn record_call(&mut self, call: ToolCall) { + if is_loop_tracker_exempt(&call.name) { + return; + } if self.last_calls.len() >= self.chain_len * self.max_repeats { self.last_calls.pop_front(); } @@ -2663,6 +2682,7 @@ mod tests { })), output_buf: Arc::new(parking_lot::Mutex::new(jobs::RingBuf::default())), no_change_checks: 0, + last_check_state: None, }; let mut sup = crate::supervisor::Supervisor::new(0, 3).with_max_concurrent_jobs(4); sup.register(handle).unwrap(); @@ -3105,6 +3125,46 @@ mod tests { assert!(msg.contains("repeat_tool")); } + #[test] + fn tracker_exempt_tools_never_trip() { + for name in LOOP_TRACKER_EXEMPT_TOOLS { + let mut tracker = ToolCallTracker::default(); + let exempt = call_with_args(name, json!({"id": "j1"})); + tracker.record_call(exempt.clone()); + tracker.record_call(exempt.clone()); + assert!(tracker.check_loop(&exempt).is_none()); + + let other = call_with_args("execute_command", json!({"command": "ls"})); + tracker.record_call(other.clone()); + assert!( + tracker.check_loop(&other).is_none(), + "exempt calls must not count toward the repeat threshold" + ); + tracker.record_call(other.clone()); + assert!(tracker.check_loop(&other).is_some()); + } + } + + #[test] + fn tracker_exempt_interleave_does_not_mask_real_loop() { + let mut tracker = ToolCallTracker::default(); + let x = call_with_args("execute_command", json!({"command": "ls"})); + tracker.record_call(call_with_args("job__check", json!({"id": "j1"}))); + tracker.record_call(x.clone()); + tracker.record_call(call_with_args("job__check", json!({"id": "j1"}))); + tracker.record_call(x.clone()); + assert!(tracker.check_loop(&x).is_some()); + } + + #[test] + fn tracker_non_exempt_behavior_unchanged() { + let mut tracker = ToolCallTracker::default(); + let c = call_with_args("fs_cat", json!({"path": "a.txt"})); + tracker.record_call(c.clone()); + tracker.record_call(c.clone()); + assert!(tracker.check_loop(&c).is_some()); + } + #[test] fn prefix_constants_are_correct() { assert_eq!(TODO_FUNCTION_PREFIX, "todo__"); diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index fee6e3e..e1586ab 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -1635,6 +1635,7 @@ mod tests { })), output_buf: Arc::new(Mutex::new(RingBuf::default())), no_change_checks: 0, + last_check_state: None, } } diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index 05ececb..ef8e2fc 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -67,8 +67,8 @@ pub struct JobHandle { pub abort_signal: AbortSignal, pub state: Arc>, pub output_buf: Arc>, - #[allow(dead_code)] pub no_change_checks: u32, + pub last_check_state: Option<(JobStatus, u64)>, } impl JobHandle { @@ -159,6 +159,13 @@ impl Supervisor { } } + pub fn job_mut(&mut self, id: &str) -> Option<&mut JobHandle> { + match self.handles.get_mut(id) { + Some(TaskHandle::Job(handle)) => Some(handle), + _ => None, + } + } + pub fn jobs(&self) -> impl Iterator { self.handles.values().filter_map(|handle| match handle { TaskHandle::Job(handle) => Some(handle), @@ -392,6 +399,7 @@ mod tests { })), output_buf: Arc::new(Mutex::new(RingBuf::default())), no_change_checks: 0, + last_check_state: None, } } From 6256b5fcfa45d4bf76534b8c05f502855fb1d2ef Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 18:16:31 -0600 Subject: [PATCH 15/28] docs: document background jobs across prompts, config example, and README - Extend the injected Background Jobs prompt guidance: system_notifications push on completion, collect-only-when-idle wait protocol, and the graph LLM-node collect-before-final-turn rule - Mention the system_notifications push in the agent spawning guidance and in the sisyphus/architect wait-protocol text (agent completions push notifications too) - config.example.yaml: max_concurrent_jobs (default 5, 0 = disabled) - README: features-list entry pointing at the Background-Jobs wiki page --- README.md | 1 + assets/agents/architect/config.yaml | 2 +- assets/agents/sisyphus/config.yaml | 2 +- config.example.yaml | 1 + src/config/prompts.rs | 19 ++++++++++++------- 5 files changed, 16 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 0d32e56..dda5ec7 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g * [Skills](https://github.com/Dark-Alex-17/coyote/wiki/Skills): Modular knowledge or capability packs the LLM can load and unload mid-conversation. Multiple skills compose; instructions stack, tools and MCPs union. * [Agents](https://github.com/Dark-Alex-17/coyote/wiki/Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools. * [Graph Agents](https://github.com/Dark-Alex-17/coyote/wiki/Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns). +* [Background Jobs](https://github.com/Dark-Alex-17/coyote/wiki/Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working — completion arrives as a push notification. * [Todo System](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System): Built-in task tracking for improved LLM reliability with smaller models. * [Environment Variables](https://github.com/Dark-Alex-17/coyote/wiki/Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables. * [Client Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Clients): Configuration instructions for various LLM providers. diff --git a/assets/agents/architect/config.yaml b/assets/agents/architect/config.yaml index b4b365d..3cc1fab 100644 --- a/assets/agents/architect/config.yaml +++ b/assets/agents/architect/config.yaml @@ -261,7 +261,7 @@ instructions: | 3. **Wait for Sisyphus.** Do not poll `agent__collect` on a running agent — do non-overlapping work (e.g. prep the next task's context) or end your response and wait for the completion - notification, then `agent__collect`. + notification (a `system_notifications` entry on your next tool result), then `agent__collect`. 4. **Verify against the plan (divergence check).** When Sisyphus returns, do NOT trust its self-report — get an INDEPENDENT conformance verdict: diff --git a/assets/agents/sisyphus/config.yaml b/assets/agents/sisyphus/config.yaml index dbbd2b7..05dc267 100644 --- a/assets/agents/sisyphus/config.yaml +++ b/assets/agents/sisyphus/config.yaml @@ -233,7 +233,7 @@ instructions: | 1. Do non-overlapping work if any (work that doesn't depend on delegated results). 2. If none → **end your response.** Do not call `agent__collect` immediately. - 3. The system notifies you on completion. + 3. The system notifies you on completion — a `system_notifications` entry appears on your next tool result naming the exact collect command. 4. On notification, call `agent__collect` to retrieve results. ### Anti-duplication rule (BLOCKING) diff --git a/config.example.yaml b/config.example.yaml index 247333a..40ab1c5 100644 --- a/config.example.yaml +++ b/config.example.yaml @@ -204,6 +204,7 @@ summary_context_prompt: > # The text prompt used for including the summar 'This is a summary of the chat history as a recap: ' compression_keep_last: 0 # Number of most-recent messages to keep visible after compression (0 = compress all messages) max_tool_result_chars: null # Cap on tool result characters forwarded to the model per call (null = no cap) +max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once per context (default: 5; 0 disables background jobs entirely) # ---- Memory ---- # See the [Memory documentation](https://github.com/Dark-Alex-17/coyote/wiki/Memory) for more information. diff --git a/src/config/prompts.rs b/src/config/prompts.rs index 556bc26..58ac50e 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -112,9 +112,10 @@ pub(in crate::config) const DEFAULT_SPAWN_INSTRUCTIONS: &str = indoc! {" ### CRITICAL: Never end your turn with pending agents - Spawned agents do NOT report back on their own. They run in the background until you - actively reclaim them with `agent__collect` (to get their output) or `agent__cancel` - (to discard them). If you spawn agents and then emit a final message without reclaiming + Spawned agents do NOT deliver their results on their own. When one finishes, a + `system_notifications` entry appears on your next tool result naming the exact collect + command — but the output is only retrieved when you actively reclaim it with `agent__collect` + (or discard it with `agent__cancel`). If you spawn agents and then emit a final message without reclaiming them, the system will detect the unreclaimed agents and reject the turn-end, injecting a reminder forcing you to handle them. After several such reminders, the system will auto-cancel them and warn you that work was lost. @@ -194,10 +195,14 @@ pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" ## Background Jobs For long-running tool calls (builds, test suites, slow commands), call `job__start` and keep - working instead of blocking. Check progress with `job__check` (sparingly), block on the result - with `job__collect`, cancel with `job__cancel`, and list jobs with `job__list`. Collect or - cancel every job you started before ending your turn. Jobs run against a snapshot of the - current config/environment and do not survive coyote exiting. + working instead of blocking — completion arrives as a `system_notifications` entry on your + next tool result. Check progress with `job__check` (sparingly), block on the result with + `job__collect` (only when you have nothing else to do), cancel with `job__cancel`, and list + jobs with `job__list`. Collect or cancel every job you started before ending your turn. In + graph LLM nodes, collect or cancel your jobs before ending your final node turn — an + uncollected job at node turn-end burns node iterations via the guardrail and can fail the + node. Jobs run against a snapshot of the current config/environment and do not survive + coyote exiting. " }; From 28018f33c9b350b27bd57a3d9ef0b7c031b2a40e Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 20:33:10 -0600 Subject: [PATCH 16/28] test(jobs): add feature, hardening, and surface test matrix for background jobs Covers the plan's T7 matrix: zero-diff invariants when jobs are off (byte-identical tool lists and prompts, None-vs-Some select_functions), validation hardening (shell/path-shaped/PATH-resolvable names, undeclared MCP servers, non-whitelisted and context-filtered tools, mapping-tool aliases, mid-batch tool-scope freshness), process lifecycle (grandchild process-group kill, pgid clear after normal completion, panic skips the completion notification), guardrail behavior (finished-job discard on force-terminate, bounded inject-then-terminate iteration burn), surface conformance (concrete_tool_names exclusion, toggle rejection, tools_info listing, infra preservation under empty filters), supervisor swaps (use_agent/exit_agent kill running jobs, child contexts cannot reach parent job ids), and graph-node job lifecycle with deferred notification drain. --- src/config/agent.rs | 47 +++++ src/config/request_context.rs | 250 +++++++++++++++++++++++++ src/function/jobs.rs | 340 ++++++++++++++++++++++++++++++++++ src/function/mod.rs | 61 ++++++ src/function/supervisor.rs | 82 ++++++++ src/graph/executor.rs | 82 ++++++++ 6 files changed, 862 insertions(+) diff --git a/src/config/agent.rs b/src/config/agent.rs index 8ff23d8..2be8b31 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -1515,4 +1515,51 @@ nodes: {} assert_eq!(config.top_k, Some(7)); assert_eq!(config.embedding_model.as_deref(), Some("some:model")); } + + #[test] + fn interpolated_instructions_without_job_declarations_is_byte_identical_across_job_settings() { + let agent = |max_concurrent_jobs| { + Agent::test_new(AgentConfig { + instructions: "hi".to_string(), + max_concurrent_jobs, + ..AgentConfig::default() + }) + }; + + let baseline = agent(None).interpolated_instructions(); + assert!( + !baseline.contains(DEFAULT_JOB_INSTRUCTIONS), + "no job guidance may be injected without job__ declarations" + ); + assert_eq!(baseline, agent(Some(0)).interpolated_instructions()); + assert_eq!(baseline, agent(Some(7)).interpolated_instructions()); + + let mut with_unrelated = agent(None); + with_unrelated.functions.append_todo_functions(); + assert_eq!( + baseline, + with_unrelated.interpolated_instructions(), + "job guidance injection must key strictly on the job__ prefix" + ); + } + + #[test] + fn interpolated_instructions_with_job_declarations_appends_job_guidance() { + let config = AgentConfig { + instructions: "hi".to_string(), + ..AgentConfig::default() + }; + let baseline = Agent::test_new(config.clone()).interpolated_instructions(); + + let mut agent = Agent::test_new(config); + agent.functions.append_job_functions(); + let output = agent.interpolated_instructions(); + + assert!(output.contains(DEFAULT_JOB_INSTRUCTIONS)); + let expected = format!( + "hi\n{DEFAULT_JOB_INSTRUCTIONS}{}", + baseline.strip_prefix("hi").unwrap() + ); + assert_eq!(output, expected); + } } diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 431177f..25f9204 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -7555,4 +7555,254 @@ mod tests { "global config" ); } + + #[test] + fn select_functions_preserves_job_tools_under_empty_role_filter() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec![])); + + let fns = ctx.select_functions(&role).unwrap(); + let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); + assert_eq!( + names, + vec![ + "job__start", + "job__check", + "job__collect", + "job__cancel", + "job__list" + ], + "job__ tools must survive an empty role tool filter" + ); + } + + #[test] + #[serial] + fn select_functions_preserves_job_tools_under_empty_agent_filter() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + let agent_name = format!( + "test_job_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + let abort = utils::create_abort_signal(); + run_async(ctx.use_agent(&app, &agent_name, None, abort)).unwrap(); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec![])); + + let fns = ctx.select_functions(&role).unwrap(); + let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); + assert!( + names.contains(&"job__start"), + "job__ tools must survive an empty agent tool filter, got: {names:?}" + ); + assert!(names.contains(&"job__collect")); + } + + #[test] + #[serial] + fn select_functions_when_jobs_disabled_is_byte_identical_to_no_jobs_baseline() { + let _guard = TestConfigDirGuard::new(); + let app_state = app_state_with_mcp_config(false, &[]); + let mut ctx = RequestContext::new(app_state, WorkingMode::Repl); + let app = ctx.app.config.clone(); + let abort = utils::create_abort_signal(); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["all".to_string()])); + + let jobs_off = AppConfig { + max_concurrent_jobs: Some(0), + ..(*app).clone() + }; + run_async(ctx.rebuild_tool_scope(&jobs_off, None, abort.clone())).unwrap(); + let without_jobs = serde_json::to_string(&ctx.select_functions(&role)).unwrap(); + assert!( + !without_jobs.contains("job__"), + "no job__ declarations may leak when jobs are disabled, got: {without_jobs}" + ); + + run_async(ctx.rebuild_tool_scope(&app, None, abort)).unwrap(); + let with_jobs = ctx.select_functions(&role).unwrap(); + assert!(with_jobs.iter().any(|f| f.name.starts_with("job__"))); + let stripped: Vec = with_jobs + .into_iter() + .filter(|f| !f.name.starts_with("job__")) + .collect(); + + assert_eq!( + without_jobs, + serde_json::to_string(&Some(stripped)).unwrap(), + "jobs-disabled tool list must be byte-identical to the jobs-enabled list minus job__ declarations" + ); + } + + #[test] + fn select_functions_returns_none_when_no_tools_enabled_and_jobs_disabled() { + let app_state = { + let config = AppConfig { + max_concurrent_jobs: Some(0), + ..AppConfig::default() + }; + Arc::new(AppState { + config: Arc::new(config), + vault: Arc::new(Vault::default()), + mcp_factory: Arc::new(McpFactory::default()), + rag_cache: Arc::new(RagCache::default()), + mcp_config: None, + mcp_log_path: None, + mcp_registry: None, + functions: Functions::default(), + }) + }; + let ctx = RequestContext::new(app_state, WorkingMode::Cmd); + assert!(ctx.select_functions(&Role::default()).is_none()); + } + + #[test] + fn tools_info_lists_job_tools_when_enabled() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + let info = ctx.tools_info().unwrap(); + + for name in [ + "job__start", + "job__check", + "job__collect", + "job__cancel", + "job__list", + ] { + assert!( + info.contains(name), + "expected {name} in output, got: {info}" + ); + } + } + + fn make_running_job(abort_signal: utils::AbortSignal) -> crate::supervisor::JobHandle { + // Leak the runtime so the spawned task is never polled and the job + // stays running for the duration of the test. + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let join_handle = rt.spawn(async { + Ok(crate::supervisor::JobResult { + output: serde_json::Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + crate::supervisor::JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: std::time::Instant::now(), + join_handle, + abort_signal, + state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { + status: crate::supervisor::JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(parking_lot::Mutex::new( + crate::function::jobs::RingBuf::default(), + )), + no_change_checks: 0, + last_check_state: None, + } + } + + #[test] + #[serial] + fn use_agent_cancels_running_jobs_of_previous_supervisor() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + let agent_name = format!( + "test_agent_{}", + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() + ); + let agent_dir = paths::agent_data_dir(&agent_name); + create_dir_all(&agent_dir).unwrap(); + write( + agent_dir.join("config.yaml"), + format!("name: {agent_name}\ninstructions: hi\n"), + ) + .unwrap(); + + let job_sig = utils::create_abort_signal(); + let old_sup = Arc::new(RwLock::new( + Supervisor::new(4, 3).with_max_concurrent_jobs(1), + )); + old_sup + .write() + .register(make_running_job(job_sig.clone())) + .unwrap(); + ctx.supervisor = Some(old_sup); + + run_async(ctx.use_agent(&app, &agent_name, None, utils::create_abort_signal())).unwrap(); + + assert!( + job_sig.aborted(), + "running jobs of the previous supervisor must be cancelled" + ); + assert!(ctx.supervisor.is_some()); + } + + #[test] + #[serial] + fn exit_agent_cancels_running_jobs() { + let _guard = TestConfigDirGuard::new(); + let mut ctx = create_test_ctx(); + let app = ctx.app.config.clone(); + + let job_sig = utils::create_abort_signal(); + let sup = Arc::new(RwLock::new( + Supervisor::new(4, 3).with_max_concurrent_jobs(1), + )); + sup.write() + .register(make_running_job(job_sig.clone())) + .unwrap(); + ctx.agent = Some(Agent::test_new(AgentConfig::default())); + ctx.supervisor = Some(sup); + + ctx.exit_agent(&app).unwrap(); + + assert!(job_sig.aborted(), "exit_agent must cancel running jobs"); + assert!(ctx.supervisor.is_none()); + } + + #[test] + fn toggle_tool_rejects_job_tools_as_unknown() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + + for action in ["enable", "disable"] { + let err = ctx.toggle_tool(action, "job__start").unwrap_err(); + assert!( + err.to_string().contains("Unknown tool 'job__start'"), + "expected job__start to be rejected on {action}, got: {err}" + ); + } + } } diff --git a/src/function/jobs.rs b/src/function/jobs.rs index f2d6c92..7d140a4 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -1928,6 +1928,49 @@ mod tests { }); } + /// The completion notification is pushed from inside the job task, after + /// the run — a panic unwinds past the push, and neither the supervisor nor + /// collect synthesizes a notification for a panicked job. + #[test] + fn panicked_job_task_skips_the_completion_notification() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let join_handle: tokio::task::JoinHandle> = + tokio::spawn(async { panic!("boom") }); + let handle = JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + // A panic never reaches the status update — the cell + // stays Running, which is the real post-panic state. + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + last_check_state: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_collect(&ctx, &json!({"id": "j1"})).await.unwrap(); + + assert_eq!(result["status"], "failed"); + assert!(result["error"].as_str().unwrap().contains("panicked")); + assert!( + ctx.notification_queue.drain().is_empty(), + "a panicked job must never produce a completion notification" + ); + }); + } + #[cfg(unix)] #[test] fn handle_cancel_kills_running_process_job() { @@ -2323,4 +2366,301 @@ mod tests { .unwrap(); assert_eq!(tasks["tasks"].as_array().unwrap().len(), 1); } + + #[cfg(unix)] + #[test] + fn handle_cancel_kills_grandchild_process() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("sh", &["-c", "sleep 30 & echo CHILD:$!; wait"], 0); + let task_state = Arc::clone(&state); + let task_buf = Arc::clone(&output_buf); + let join_handle = + tokio::spawn(async move { run_process_job(snapshot, task_state, task_buf).await }); + + let deadline = time::Instant::now() + Duration::from_secs(5); + let grandchild_pid = loop { + let tail = String::from_utf8_lossy(&output_buf.lock().tail()).to_string(); + if let Some(rest) = tail.split("CHILD:").nth(1) + && let Some(line_end) = rest.find('\n') + { + break rest[..line_end].trim().parse::().unwrap(); + } + assert!( + time::Instant::now() < deadline, + "grandchild pid never appeared in the ring buffer" + ); + time::sleep(Duration::from_millis(10)).await; + }; + + let handle = JobHandle { + id: "j1".to_string(), + tool: "sh".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::clone(&state), + output_buf, + no_change_checks: 0, + last_check_state: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_cancel(&ctx, &json!({"id": "j1"})).await.unwrap(); + + assert_eq!(result["status"], "cancelled"); + // kill(pid, 0) alone can't observe the death: the orphaned + // grandchild lingers as an unreaped zombie under init/launchd, + // so a Z state also proves the group kill landed. + fn grandchild_is_dead(pid: i32) -> bool { + let esrch = unsafe { libc::kill(pid, 0) } == -1 + && std::io::Error::last_os_error().raw_os_error() == Some(libc::ESRCH); + if esrch { + return true; + } + let stat = std::process::Command::new("ps") + .args(["-o", "stat=", "-p", &pid.to_string()]) + .output() + .map(|out| String::from_utf8_lossy(&out.stdout).trim().to_string()) + .unwrap_or_default(); + stat.is_empty() || stat.starts_with('Z') + } + let deadline = time::Instant::now() + Duration::from_secs(5); + while !grandchild_is_dead(grandchild_pid) { + assert!( + time::Instant::now() < deadline, + "grandchild must die with the process group" + ); + time::sleep(Duration::from_millis(10)).await; + } + }); + } + + #[cfg(unix)] + #[test] + fn run_process_job_clears_pgid_after_normal_completion() { + run_async(async { + let state = Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })); + let output_buf = Arc::new(Mutex::new(RingBuf::default())); + let snapshot = test_snapshot("echo", &["done"], 0); + + let result = run_process_job(snapshot, Arc::clone(&state), output_buf) + .await + .unwrap(); + + assert_eq!(result.exit_code, Some(0)); + assert!( + state.lock().pgid.is_none(), + "pid-reuse guard must clear pgid" + ); + }); + } + + #[test] + fn handle_start_rejects_shell_and_path_shaped_names_without_spawn() { + let mut ctx = plain_ctx(); + + for tool in ["bash", "./script.sh", "/usr/bin/env", "ls"] { + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": tool, "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + } + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[cfg(unix)] + #[test] + fn handle_start_rejects_context_filtered_tool_and_accepts_in_filter() { + run_async(async { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("echo".into()); + + let rejected = handle_start(&mut ctx, &json!({"tool": "git_command", "arguments": {}})) + .await + .unwrap(); + + assert_eq!(rejected["status"], "error"); + assert!( + rejected["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + + let started = handle_start(&mut ctx, &json!({"tool": "echo", "arguments": {}})) + .await + .unwrap(); + + assert_eq!(started["status"], "ok"); + let job_id = started["job_id"].as_str().unwrap().to_string(); + let collected = handle_collect(&ctx, &json!({"id": job_id})).await.unwrap(); + assert_eq!(collected["status"], "completed"); + }); + } + + #[test] + fn handle_start_rejects_undeclared_mcp_invoke_without_spawn() { + let mut ctx = plain_ctx(); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "mcp_invoke_someserver", "arguments": {"tool": "search"}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[test] + fn handle_start_rejects_declared_but_non_whitelisted_tools_without_spawn() { + let mut ctx = plain_ctx(); + for tool in ["memory__write", "fs_read", "agent__spawn", "user__select"] { + ctx.declared_function_names.insert(tool.into()); + } + + for (tool, category) in [ + ("memory__write", "mutates agent/session state"), + ("fs_read", "is fast"), + ("agent__spawn", "already asynchronous"), + ("user__select", "interactive"), + ] { + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": tool, "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!(result["message"].as_str().unwrap().contains(category)); + } + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[test] + fn handle_start_rejects_mapping_tools_alias() { + let app_state = app_state_with_config(|config| { + config + .mapping_tools + .insert("shell".into(), "execute_command".into()); + }); + let mut ctx = RequestContext::new(app_state, WorkingMode::Cmd); + ctx.declared_function_names.insert("execute_command".into()); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "shell", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } + + #[test] + fn child_context_cannot_reach_parent_job_ids() { + run_async(async { + let parent = ctx_with_job_supervisor(4); + parent + .supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("job_p1")) + .unwrap(); + let child = RequestContext::new_for_child( + default_app_state(), + &parent, + 1, + Arc::new(Inbox::new()), + "c1".into(), + ); + assert!(child.supervisor.is_none()); + + let checked = handle_check(&child, &json!({"id": "job_p1"})).unwrap(); + let collected = handle_collect(&child, &json!({"id": "job_p1"})) + .await + .unwrap(); + let cancelled = handle_cancel(&child, &json!({"id": "job_p1"})) + .await + .unwrap(); + + for result in [checked, collected, cancelled] { + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("No job 'job_p1' is registered") + ); + } + assert!(parent.supervisor.as_ref().unwrap().read().has_job("job_p1")); + }); + } + + #[test] + fn handle_start_ignores_mid_batch_tool_scope_additions() { + let mut ctx = plain_ctx(); + ctx.declared_function_names.insert("job__start".into()); + ctx.tool_scope + .functions + .declarations + .push(FunctionDeclaration { + name: "late_external_tool".into(), + description: String::new(), + parameters: JsonSchema::default(), + agent: false, + }); + + let result = run_async(handle_start( + &mut ctx, + &json!({"tool": "late_external_tool", "arguments": {}}), + )) + .unwrap(); + + assert_eq!(result["status"], "error"); + assert!( + result["message"] + .as_str() + .unwrap() + .contains("not enabled in this context") + ); + assert!(ctx.supervisor.is_none(), "no job may be spawned"); + } } diff --git a/src/function/mod.rs b/src/function/mod.rs index e116dd0..7a96ec4 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -2940,6 +2940,51 @@ mod tests { ); } + #[cfg(unix)] + #[test] + fn job_finished_earlier_drains_notification_on_later_batch() { + run_async(async { + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + ctx.declared_function_names.insert("echo".into()); + + let started = jobs::handle_job_tool( + &mut ctx, + "job__start", + &json!({"tool": "echo", "arguments": {}}), + ) + .await + .unwrap(); + assert_eq!(started["status"], "ok"); + let job_id = started["job_id"].as_str().unwrap().to_string(); + + let supervisor = ctx.supervisor.clone().unwrap(); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); + while tokio::time::Instant::now() < deadline { + let finished = supervisor + .read() + .job(&job_id) + .is_none_or(|job| job.join_handle.is_finished()); + if finished { + break; + } + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + + let calls = vec![call("unknown_tool", Some("id-2"))]; + let results = eval_tool_calls(&mut ctx, calls).await.unwrap(); + + let out = &results.last().unwrap().output; + assert_eq!(out["system_notifications"][0]["id"], job_id); + assert_eq!(out["system_notifications"][0]["event"], "job_completed"); + assert!( + out["notification_instruction"] + .as_str() + .unwrap() + .contains("next_action") + ); + }); + } + #[test] fn normalize_tool_result_preserves_non_null_values() { assert_eq!( @@ -3125,6 +3170,22 @@ mod tests { assert!(msg.contains("repeat_tool")); } + #[test] + fn loop_tracker_exempt_list_is_exactly_the_polling_tools() { + let actual: std::collections::HashSet<&str> = + LOOP_TRACKER_EXEMPT_TOOLS.iter().copied().collect(); + let expected: std::collections::HashSet<&str> = [ + "job__check", + "job__list", + "agent__check", + "agent__list_running", + ] + .into_iter() + .collect(); + assert_eq!(LOOP_TRACKER_EXEMPT_TOOLS.len(), 4); + assert_eq!(actual, expected); + } + #[test] fn tracker_exempt_tools_never_trip() { for name in LOOP_TRACKER_EXEMPT_TOOLS { diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index e1586ab..e276c47 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -2911,4 +2911,86 @@ mod tests { assert_job_teaching_error(&result, "bg_p"); }); } + + #[test] + fn guardrail_burns_bounded_injects_then_force_terminates_running_job() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + rt.block_on(async { + let mut ctx = ctx_with_job_capable_supervisor(); + let abort = create_abort_signal(); + let join_handle = tokio::spawn(async { + time::sleep(Duration::from_secs(60)).await; + Ok(JobResult { + output: json!(null), + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + let handle = JobHandle { + id: "job_1".to_string(), + tool: "execute_command".to_string(), + started_at: std::time::Instant::now(), + join_handle, + abort_signal: abort.clone(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Running, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + last_check_state: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + for expected_count in 1..=PENDING_AGENTS_GUARDRAIL_MAX { + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::Inject(prompt) => assert!(prompt.contains("job_1")), + _ => panic!("expected Inject below max"), + } + assert_eq!(ctx.pending_agents_guardrail_count, expected_count); + } + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::ForceTerminate(ids) => { + assert_eq!(ids, vec!["job_1".to_string()]); + } + _ => panic!("expected ForceTerminate at max"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert!(abort.aborted()); + }); + } + + #[test] + fn guardrail_force_terminate_discards_finished_uncollected_job() { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "job_1"); + let deadline = std::time::Instant::now() + Duration::from_secs(5); + while !pending_tasks(&ctx).iter().any(|t| t.finished) { + assert!( + std::time::Instant::now() < deadline, + "job 'job_1' never finished" + ); + std::thread::sleep(Duration::from_millis(10)); + } + ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + + match check_pending_agents_guardrail(&mut ctx) { + GuardrailAction::ForceTerminate(ids) => { + assert_eq!(ids, vec!["job_1".to_string()]); + } + _ => panic!("expected ForceTerminate action"), + } + assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert!(!ctx.supervisor.as_ref().unwrap().read().has_job("job_1")); + } } diff --git a/src/graph/executor.rs b/src/graph/executor.rs index a53441d..70ec480 100644 --- a/src/graph/executor.rs +++ b/src/graph/executor.rs @@ -856,4 +856,86 @@ nodes: ); assert!(err.contains("sleeper"), "error should name frontier: {err}"); } + + #[cfg(unix)] + #[tokio::test] + async fn background_job_survives_graph_node_execution() { + if !cmd_available("bash") { + eprintln!("skipping: bash not available"); + return; + } + let ws = TestWorkspace::new(); + ws.write_script("noop.sh", "#!/bin/bash\necho '{}'\n"); + + let yaml = r#" +name: background_job_survival_test +start: noop +nodes: + noop: + type: script + script: noop.sh + state_updates: {} + next: done + done: + type: end + output: "done" +"#; + let graph: Graph = serde_yaml::from_str(yaml).unwrap(); + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + let join_handle = rt.spawn(async { + Ok(crate::supervisor::JobResult { + output: Value::Null, + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + std::mem::forget(rt); + let handle = crate::supervisor::JobHandle { + id: "job_bg".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { + status: crate::supervisor::JobStatus::Completed, + pgid: None, + })), + output_buf: Arc::new(parking_lot::Mutex::new( + crate::function::jobs::RingBuf::default(), + )), + no_change_checks: 0, + last_check_state: None, + }; + let mut sup = crate::supervisor::Supervisor::new(0, 3).with_max_concurrent_jobs(4); + sup.register(handle).unwrap(); + + let mut ctx = make_ctx(); + ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); + ctx.notification_queue + .push(crate::supervisor::notification::job_notification( + "job_bg", + "execute_command", + true, + )); + + let abort = create_abort_signal(); + let result = GraphExecutor::new(graph, &ws.dir) + .execute(&mut ctx, abort) + .await + .unwrap_or_else(|e| panic!("executor failed: {e:#}")); + assert_eq!(result, "done"); + + assert!( + ctx.supervisor.as_ref().unwrap().read().has_job("job_bg"), + "graph execution must not touch registered job handles" + ); + let events = ctx.notification_queue.drain(); + assert_eq!(events.len(), 1, "queued notification must survive the run"); + assert_eq!(events[0].id, "job_bg"); + assert_eq!(events[0].event, "job_completed"); + } } From 2eb63cfc0d6c3d8c83d24e1c5261ff2c977fdbd9 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 20:47:34 -0600 Subject: [PATCH 17/28] test: pin memory config on in eval-routing test for environment-independent CI The eval_routes_memory_prefix_to_memory_handler characterization test inherited the host machine's memory configuration: on runners without a memory setup, should_register_memory_tools() gates the handler off and the error message differs. Force memory = Some(true) at ctx construction so the test asserts the same handler path everywhere. --- src/function/mod.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/function/mod.rs b/src/function/mod.rs index 7a96ec4..8151857 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -4730,7 +4730,11 @@ mod tests { #[test] fn eval_routes_memory_prefix_to_memory_handler() { - let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); + let mut app_state = AppState::test_default(); + let mut app_config = (*app_state.config).clone(); + app_config.memory = Some(true); + app_state.config = Arc::new(app_config); + let mut ctx = RequestContext::new(Arc::new(app_state), WorkingMode::Cmd); ctx.tool_scope.functions.append_memory_functions(); let out = run_async(call_with_args("memory__read", json!({})).eval(&mut ctx)).unwrap(); From 5a9f8c42b96bc14af87ddb151ea741265271f849 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Tue, 25 Aug 2026 20:52:31 -0600 Subject: [PATCH 18/28] test: assert memory routing without depending on host memory files memory_config() only reports enabled when a global memory index or a workspace memory store exists on disk, so asserting the handler's 'name is required' detail was environment-dependent even with the memory pref forced on. The routing test now accepts either memory-handler-owned message: the 'Memory tool failed' prefix alone proves the memory__ prefix reached the memory handler. --- src/function/mod.rs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/src/function/mod.rs b/src/function/mod.rs index 8151857..f05e9fc 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -4730,18 +4730,17 @@ mod tests { #[test] fn eval_routes_memory_prefix_to_memory_handler() { - let mut app_state = AppState::test_default(); - let mut app_config = (*app_state.config).clone(); - app_config.memory = Some(true); - app_state.config = Arc::new(app_config); - let mut ctx = RequestContext::new(Arc::new(app_state), WorkingMode::Cmd); + let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); ctx.tool_scope.functions.append_memory_functions(); let out = run_async(call_with_args("memory__read", json!({})).eval(&mut ctx)).unwrap(); let err = out["tool_call_error"].as_str().unwrap(); assert!(err.starts_with("Memory tool failed"), "{err}"); - assert!(err.contains("name is required"), "{err}"); + assert!( + err.contains("name is required") || err.contains("Memory tools are disabled"), + "expected a memory-handler-owned error regardless of host memory files: {err}" + ); } #[test] From 4e50b4ff4a3644cc9ded691b03a778db3d795073 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 12:43:51 -0600 Subject: [PATCH 19/28] refactor: Modified the naming of several generalized supervisor values --- README.md | 2 +- plans/background-jobs-design.md | 1200 ------------------------------- src/acp/server.rs | 4 +- src/config/agent.rs | 2 - src/config/app_config.rs | 26 +- src/config/app_state.rs | 1 + src/config/input.rs | 1 - src/config/request_context.rs | 60 +- src/function/jobs.rs | 64 +- src/function/mod.rs | 70 +- src/function/supervisor.rs | 122 ++-- src/graph/executor.rs | 31 +- src/graph/llm.rs | 4 +- src/main.rs | 4 +- src/repl/mod.rs | 6 +- src/supervisor/mod.rs | 11 +- 16 files changed, 253 insertions(+), 1355 deletions(-) delete mode 100644 plans/background-jobs-design.md diff --git a/README.md b/README.md index dda5ec7..2b0b79a 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g * [Skills](https://github.com/Dark-Alex-17/coyote/wiki/Skills): Modular knowledge or capability packs the LLM can load and unload mid-conversation. Multiple skills compose; instructions stack, tools and MCPs union. * [Agents](https://github.com/Dark-Alex-17/coyote/wiki/Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools. * [Graph Agents](https://github.com/Dark-Alex-17/coyote/wiki/Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns). -* [Background Jobs](https://github.com/Dark-Alex-17/coyote/wiki/Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working — completion arrives as a push notification. +* [Background Jobs](https://github.com/Dark-Alex-17/coyote/wiki/Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working, and completion arrives as a push notification. * [Todo System](https://github.com/Dark-Alex-17/coyote/wiki/TODO-System): Built-in task tracking for improved LLM reliability with smaller models. * [Environment Variables](https://github.com/Dark-Alex-17/coyote/wiki/Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables. * [Client Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Clients): Configuration instructions for various LLM providers. diff --git a/plans/background-jobs-design.md b/plans/background-jobs-design.md deleted file mode 100644 index 52bb456..0000000 --- a/plans/background-jobs-design.md +++ /dev/null @@ -1,1200 +0,0 @@ -# Design: Background Tool Jobs & Completion Push Notifications - -Status: DRAFT v1.6 — grounded in src/function/mod.rs (eval_tool_calls, escalation -injection), src/supervisor/mod.rs (Supervisor/AgentHandle), src/function/ -supervisor.rs (agent__* handlers, turn-end guardrail), and Oracle rulings -(agent_oracle_10936145, persisted as memory `coyote-bg-jobs-notifications-rulings`). -User rulings R6/R7 added 2026-08-21; R7 amended + R8/R9 and §9 (regression -parity) added same day after injection-gating and test-coverage exploration. -§13 items ALL RESOLVED (final code audits 2026-08-21: JobCtx, JobEnvSnapshot, -sbx process model, truncation). Gate audit 2026-08-21 (LEAKY, 8 findings, -11/11 receipts verified) remediated here. Oracle plan review 2026-08-21 -(agent_oracle_f0d3932c): architecture + T0–T7 decomposition APPROVED; -findings folded in: shared JobState cell w/ pgid-clear (pid-reuse guard), -use_agent reset-site correction (there is NO full-context /clear), explicit -shutdown mechanism, capacity-0 supervisor consumer audit, panic-notification -semantics, tail-side UTF-8 flooring, lock discipline, parallelization map. -Re-gate on v1.5 (all prior findings verified fixed, receipts 100%): 3 -residual gaps fixed in v1.6 — lazy plain-session supervisor init (R9/§6), -T2∥T3 function/mod.rs caveat (§11), collect-side cap ownership (§3/§6). - -## 0. Decision record - -Rulings already made (Oracle 2026-08-21, user same day) — do not relitigate -during implementation: - -- **R1 (registry)**: One unified `Supervisor` whose `handles` map holds - `TaskHandle = enum { Agent(AgentHandle), Job(JobHandle) }`. REJECTED: a - kind-flag field on a generalized struct (Option-soup; a Job with a - `child_supervisor` becomes representable) and a parallel `JobRegistry` - (guardrail, `cancel_recursive`, and all 5 turn-end sites already traverse - `ctx.supervisor`; a second registry duplicates traversal and creates two - cleanup paths that drift). -- **R2 (invocation shape)**: Meta-tool `job__start { tool, arguments }`, - mirroring the `mcp_invoke_*` wrapper shape the model already sees in its own - transcripts. REJECTED: injecting a `background: true` param into every tool - schema — coyote doesn't own MCP passthrough or argc-generated schemas, the - param must be stripped before dispatch, and strict parsers - (`additionalProperties: false` MCP servers, argc bins) hard-fail on leaks. -- **R3 (push mechanism)**: Notifications are delivered by (a) merging a - `system_notifications` key into the LAST real ToolResult of a batch (the - proven `pending_escalations` channel) and (b) the turn-end guardrail user - message. REJECTED: a mid-turn synthetic `[SYSTEM NOTIFICATION]` user - message — attribution confusion (model reads it as user intent and breaks - off its tool chain), no clean seam (`merge_tool_results` at 5 sites vs one - function in `eval_tool_calls`), fake "user said" artifacts persisting in - saved sessions, and ZERO coverage gain: the only moment with no ToolResult - to merge into is an empty batch, which IS turn-end, which the guardrail - owns. This is the same transcript-integrity principle as the - `__escalation_notification` phantom-call fix (see memory - `coyote-escalation-notification-bug`): never fabricate transcript content - the model didn't produce. -- **R4 (agents too)**: Completion notifications are ALWAYS-ON for both jobs - and spawned agents. REJECTED: per-spawn opt-in (asymmetry doubles prompt - guidance; the model polls anyway; volume is one terse event per - completion). A `notify: false` opt-out may be added later if fanout spam - materializes — not in v1. NOTE: this is one of the four deliberate - behavior deltas that apply even when jobs are disabled (§9.1). -- **R5 (persistence)**: Jobs die with the coyote process. No pid files, no - output spooling, no reattach-after-restart in v1. Half-building persistence - is worse than not having it. -- **R6 (check semantics — user-ruled 2026-08-21)**: `job__check` AND - `agent__check` are pure status probes; they NEVER consume the handle. - `collect` is the single retrieval+reclaim verb. REJECTED: today's - agent__check finished→collect delegation (supervisor.rs:844) — it returns - an unbounded payload the model didn't opt into (job results can be huge), - makes check's return shape bimodal, breaks the documented check-then-collect - pattern (follow-up collect errors on the consumed id), and silently defeats - the H7 guardrail backstop (once consumed, an overlooked result is - unrecoverable). With push notifications the round-trip saving is marginal: - completion usually arrives as a notification naming the collect command, - not via polling. Finished JOB checks return status + output_tail preview + - the exact collect command; finished AGENT checks return status + the exact - collect command only (agents have no ring buffer — nothing to preview). - Deliberate delta, see §9.1. -- **R7 (concurrency knob — user-ruled 2026-08-21, AMENDED same day)**: - `max_concurrent_jobs` is BOTH a global `Config` field (concrete, default - **5**) AND an `Option` override on `AgentConfig`, resolved - agent-override-first exactly like `max_tool_result_chars` - (src/function/mod.rs:332-337 pattern: - `agent.field().or(global)`). Rationale for the amendment: plain (non-agent) - sessions have NO `AgentConfig` at all (§2), and jobs — unlike agent - spawning — ARE available in plain sessions (R9), so the global field is - the only knob there; a serde default alone is not user-tweakable. - Role/session-runtime overrides remain DEFERRED (parity with - `max_concurrent_agents`). Implementation gotcha: `AppConfig` duplicates - `Config` fields across FOUR touch points (struct ~app_config.rs:66, - Default ~:151, From ~:237, env override ~:573) — miss one and the - field silently stays default. -- **R8 (feature-off gating — user-ruled 2026-08-21; amended 2026-08-24)**: - the job feature is OFF for a context when EITHER (a) the effective - `max_concurrent_jobs` is 0, OR (b) `function_calling_support` is false - (config mod.rs:232 / AppConfig app_config.rs:36, env-overridable - app_config.rs:523). When OFF, the `job__*` tool declarations are NOT - injected, the job prompt instructions are NOT injected, and no job - capacity exists — mirroring exactly how `can_spawn_agents: false` omits - the agent__* family today (declarations gated at Agent::init - agent.rs:224-226; prompt text at agent.rs:439-441; supervisor creation at - rc.rs:4139-4143). A model that has never seen a job__ declaration cannot - call one; `max_concurrent_jobs: 0` = the feature does not exist for that - context. The function-calling gate mirrors how skill/memory/rag function - injection already checks `app.function_calling_support` at Agent::init - (agent.rs:231, 238, 252) — NOT how agent__* does it (agent__* injects - unconditionally and only refuses at runtime, supervisor.rs:526/710; - job__* must gate at INJECTION so neither declarations nor instructions - ever reach the prompt). The model-level `supports_function_calling()` - strip at input.rs:260 remains the generic per-request safety net but is - NOT the gate — it strips declarations only, never injected instructions. -- **R9 (plain sessions)**: jobs ARE available outside agent contexts - (plain REPL/role sessions), governed by the global config value alone. - This requires creating the Supervisor outside `use_agent`: init condition - becomes `can_spawn_agents || jobs_enabled` — where `jobs_enabled` is - `function_calling_support && effective_max_concurrent_jobs > 0` (R8) — with - agent capacity 0 when `can_spawn_agents` is false (Supervisor::register - already rejects at capacity — tested behavior). `exit_agent`'s Functions - rebuild (rc.rs:4177-4186) must retain job declarations when jobs are - enabled. Creation sites (gate-ruled 2026-08-21): EAGER in `use_agent` - with the amended condition (existing site rc.rs:4139-4143); LAZY in plain - sessions — `job__start` get-or-inits `ctx.supervisor` (agent capacity 0) - on first use (job handlers run on the sequential `&mut ctx` path, so they - can set it); `exit_agent`'s unconditional `self.supervisor = None` - (rc.rs:4192) STAYS as-is — `cancel_recursive` already ran (rc.rs:4190) - and the next `job__start` lazily recreates. A plain session that never - starts a job keeps `supervisor: None` — bit-identical to today (§9.2.3). -- **R10 (REPL surfaces — user-ruled 2026-08-24)**: `job__*` are BUILT-IN - functions — always on when `jobs_enabled`, never individually - toggleable: - - `.info tools` MUST list them when enabled. This is FREE: `tools_info` - (rc.rs:700) renders the `select_functions` output, so it falls out of - §3 injection + carve-outs — no dedicated code. - - `.list tools`, `.tool enable/disable` validation, AND the `.tool` - tab completions MUST all exclude them. ONE change covers all three - surfaces: add `job__` to the built-in prefix exclusion list in - `concrete_tool_names()` (rc.rs:1283-1308; today user__/mcp_/todo__/ - agent__/memory__/skill__/rag__) — `.list tools` (rc.rs:2732), - `toggle_tool` validation (rc.rs:1356), and `repl_complete`'s `.tool` - arm (rc.rs:3458) all draw from that one pool. This also keeps `job__` - out of `toggle_tool`'s disable-path pool materialization, so job - tools can never leak into a persisted `enabled_tools` list. - - Consequently `.tool enable job__start` errors with the existing - "Unknown tool" teaching error — identical to agent__/todo__ today. -- **R11 (subagents & tool-filtered contexts — user-ruled 2026-08-24)**: - - An explicit `enabled_tools` list (role, session, agent, or graph LLM - node `tools:` — node lists arrive as role-level enabled_tools, see the - comment at rc.rs:2043-2046) can neither GRANT nor REVOKE `job__*`; - presence is governed solely by `jobs_enabled` (R8). Mechanism: - `JOB_FUNCTION_PREFIX` joins BOTH infra carve-outs in - `select_enabled_functions` (the §3 bullet: agent-path retain - ~rc.rs:2139-2150, non-agent builtin re-add ~rc.rs:2109-2128); extend - `select_functions_preserves_infra_tools_under_agent_filter` - (rc.rs:5642). - - A context WITH a tool list CAN use `job__*` (user-confirmed) — but - `job__start` additionally validates that the requested tool is - AVAILABLE in the calling context (§3), so a narrowed node can only - background tools it could call in the foreground; `job__start` must - not be an `enabled_tools` bypass. - - A context with NO enabled tools still SEES `job__*` — consistent with - how agent__/todo__/user__ infra prefixes survive an empty filter - today. Harmless: every `job__start` there rejects with the - context-availability teaching error, since nothing is startable. - NAMED CONSEQUENCE (Oracle N2, accepted): this flips previously - TOOLLESS contexts from `tools: None` to `Some([five job__* decls])` - request shape — `select_functions` returns None only when the merged - set is empty (rc.rs:2277-2287; pinned by - `select_functions_returns_none_when_no_tools_enabled`, rc.rs:5493, - which needs a jobs-off guard). E.g. a pure-text graph node with - `tools: []` now sends a tools array. T7 pins both states. - - Subagents: NO special machinery. Each spawned agent runs Agent::init - → R8 gating with its own effective `max_concurrent_jobs` (agent - override → global); its jobs register in its OWN supervisor; parent - `cancel_recursive` already cascades through child supervisors (T1). - -Explicit non-goals: job persistence across restarts (R5); backgrounding -internal state-mutating tools (§4 whitelist); a general mid-turn user-message -injection mechanism (R3); job-to-job messaging (jobs have no inbox — they are -processes/futures, not conversants). - -## 1. Summary - -Today every tool call blocks the turn: `eval_tool_calls` (src/function/ -mod.rs:262) is awaited inline from `call_chat_completions` (src/client/ -common.rs:526), so a 10-minute build or a slow MCP call pins the model until -it finishes. Coyote already solved this exact problem for *agents* -(`agent__spawn`/`check`/`collect`/`cancel` on tokio tasks) — this design -generalizes that machinery to *tool invocations*: - -1. A **`job__*` tool family** — `job__start`, `job__check`, `job__collect`, - `job__cancel`, `job__list` — that runs a whitelisted tool call as a - background tokio task registered in the existing `Supervisor`. -2. A **per-context `NotificationQueue`** so job and agent completions are - *pushed* to the model (merged into the next batch's last ToolResult, and - enumerated at turn-end by the guardrail) instead of relying on the model - remembering to poll. -3. A **guardrail predicate fix** so finished-but-uncollected handles block - turn-end with exact collect commands — which also fixes a latent bug where - a finished-but-uncollected *agent's* output is silently abandoned today. - -## 2. What exists today (verified) - -Turn loop & dispatch: -- `ask()` (src/repl/mod.rs:1416) recurses: `call_chat_completions[_streaming]` - → `eval_tool_calls` → non-empty `tool_results` re-enter `ask()` via - `input.merge_tool_results` (repl/mod.rs:1469-1476). Empty results = turn - end. Same loop duplicated in src/main.rs:581, src/acp/server.rs:214, - src/graph/llm.rs:258-302, and the child loop `run_child_agent` - (src/function/supervisor.rs:435-504). -- `eval_tool_calls` (mod.rs:262) partitions a batch on `is_mcp_meta_function` - (mod.rs:291-293): calls carrying any of the FIVE MCP meta-prefixes - (`mcp_invoke_/mcp_search_/mcp_describe_/mcp_read_/mcp_prompt_` — the - catalog grew read/prompt with the 2026-08-25 MCP resources/prompts merge, - src/mcp/mod.rs:36-48) run CONCURRENTLY via - `future::join_all` with `&RequestContext` (`eval_mcp` takes `&ctx`, - mod.rs:296-300); everything else runs SEQUENTIALLY because `ToolCall::eval` - (mod.rs:1420) takes `&mut RequestContext`. Results re-sorted by original - index; per-call errors soft-fail as `{"tool_call_error": ...}`; nulls - normalize to `"DONE"` (mod.rs:358-364) because empty results end the turn. -- Post-processing order matters: dedup/loop check (mod.rs:271-284) → - `max_tool_result_chars` truncation (mod.rs:332-337) → escalation injection - into the last result at depth 0 (mod.rs:343-347). - -Supervisor & agents: -- `agent__spawn` (`handle_spawn`, supervisor.rs:649) builds a child ctx - (`RequestContext::new_for_child`, request_context.rs:465 — note: - `supervisor: None` fresh per child, but `escalation_queue` CLONE-INHERITED - from parent at rc.rs:497) and `tokio::spawn`s `run_child_agent` - (supervisor.rs:778-795). The result lives in the - `JoinHandle>` inside `AgentHandle { id, agent_name, - depth, inbox, abort_signal, join_handle, child_supervisor }` - (src/supervisor/mod.rs:30-38), registered in `Supervisor { handles: - HashMap }` (mod.rs:40-45). -- `agent__check` polls `is_finished()` (supervisor.rs:827); on a finished - agent it silently DELEGATES to `handle_collect` (supervisor.rs:844), i.e. - check can consume the handle. `agent__collect` = 200ms poll loop that - breaks out early with `status: "pending"` if escalations are pending - (supervisor.rs:894-937), then `Supervisor::take` + await, optional LLM - summarization above a threshold (supervisor.rs:1401-1451). `agent__cancel` - = take + `cancel_recursive()` on child supervisor + abort signal + ≤5s - join wait (supervisor.rs:1044-1067). -- Turn-end guardrail `check_pending_agents_guardrail` (supervisor.rs:71-99): - called at all 5 turn-end sites; injects a `[SYSTEM GUARDRAIL]` user message - via `Input::from_str` and recurses; after `PENDING_AGENTS_GUARDRAIL_MAX = 3` - reminders, `ForceTerminate` + `cancel_recursive()`. CRITICAL: - `pending_agent_ids` (supervisor.rs:41-53) filters - `is_finished == Some(false)` — finished-but-uncollected handles pass the - guardrail silently and their results are dropped. -- `run_child_agent` ends with `supervisor.read().cancel_recursive()` - (supervisor.rs:498-500) — child-owned handles get cleanup for free IFF job - cancellation is wired into `cancel_recursive`. -- Context-reset sites (Oracle-corrected 2026-08-21): there is NO - full-context `/clear` command — the REPL has only `.clear todo` - (repl/mod.rs:1320-1335). The real supervisor-reset sites are `use_agent` - (rc.rs:4155 — REPLACES `self.supervisor` WITHOUT calling - `cancel_recursive` today) and `exit_agent` (rc.rs:4190 — calls it). Also - relevant: `process::exit` bypasses destructors at main.rs:672 (shell - execute), main.rs:273, logs.rs:18, config/mod.rs:783. - -Tool-injection gating & config layering (verified 2026-08-21): -- agent__* declarations exist ONLY inside an agent's `Functions`: appended by - `Functions::append_supervisor_functions()` (mod.rs:616-621), called at - `Agent::init` iff `can_spawn_agents` (agent.rs:224-226). `spawnable_agents` - and depth limits do NOT gate injection — enforced at spawn time - (supervisor.rs:663-670, Supervisor depth check). -- Per-request tool list: `RequestContext::select_enabled_functions` - (rc.rs:2039-2172); agent path copies agent declarations (:2098-2109) with - a `SUPERVISOR_FUNCTION_PREFIX` carve-out surviving role `enabled_tools` - filters (:2111-2121); non-agent path has a builtin-prefix filter - (:2077-2096). - A job__ prefix needs the same carve-outs. -- Prompt gating: `interpolated_instructions()` pushes - `DEFAULT_SPAWN_INSTRUCTIONS` (prompts.rs:74) iff - `can_spawn_agents && inject_spawn_instructions` (agent.rs:439-441). -- Supervisor exists ONLY when `use_agent` runs with `can_spawn_agents` - (rc.rs:4139-4143); plain REPL/role sessions have `supervisor: None`, no - agent__* declarations, and CANNOT spawn agents. `exit_agent` - (rc.rs:4175-4192) rebuilds `Functions::init` + user-interaction functions - only (:4113-4116). -- Global-vs-agent fallback precedent: `Option` on AgentConfig + concrete - global Config field + `agent.field().or(global)` at the resolution site - (max_tool_result_chars: mod.rs:332-337; compression_threshold: - session.rs:569-575). `max_concurrent_agents` does NOT follow it (agent-only, - serde default 4) — R7 makes the jobs knob follow the Option-fallback - pattern instead. - -Safe injection channels (the only two): -- Key-merge into the last real ToolResult: `inject_escalation_notification` - (mod.rs:366-381) adds `pending_escalations` + `escalation_instruction` keys - to the output object, wrapping non-objects as `{"output": old, ...}`. Runs - AFTER truncation. Gated `ctx.current_depth == 0` on the SHARED root - escalation queue. -- Synthetic user message at turn-end only (guardrail, auto-continue at - repl/mod.rs:1477-1524, `.recover`). - -Loop detection: root `ToolCallTracker::default()` = `new(2,3)` (mod.rs: -2336-2353) — TWO identical consecutive calls trip it. Unmitigated, a model -calling `job__check {id}` twice in a row is flagged as a loop. - -External argc tools: results are returned via the `LLM_OUTPUT` temp file, not -stdout (run_llm_function protocol; see memory -`coyote-execute-command-swallowed-output` for the hardened error semantics). -Stdout/stderr are human-facing side output. - -Test coverage today (inventoried 2026-08-21; all inline `#[cfg(test)]`, no -tests/ dir): ~153 tests across the area. FULL coverage: escalation.rs (11), -mailbox.rs (11), taskqueue.rs (17), Supervisor register/take/capacity/depth -unit tests (11), single-call eval_tool_calls behaviors (soft-fail, DONE -normalization, escalation injection incl. non-object wrap), tracker + dedup -unit tests, declaration/registry tests (function/mod.rs ≈55, -function/supervisor.rs 48). ZERO coverage (ranked by regression risk — the -basis for T0 in §11): handle_collect (poll loop, escalation early-out, -take+await, summarize threshold); guardrail ForceTerminate arm + counter -reset; multi-call eval_tool_calls (MCP/sequential partition, index re-sort, -loop-alert-in-batch); handle_spawn (capacity/depth/allow-list through the -handler); run_child_agent; turn-loop call sites + merge_tool_results; -max_tool_result_chars truncation; handle_check finished→collect delegation; -handle_cancel of a RUNNING agent / cancel_recursive; ToolCall::eval prefix -routing. - -## 3. The `job__*` tool family - -New prefix `JOB_FUNCTION_PREFIX = "job__"` routed in `ToolCall::eval`'s -prefix chain (mod.rs:1420) before the external-command fallthrough, handled -in src/function/jobs.rs (new), mirroring src/function/supervisor.rs. - -### Availability gating (R8, R9) - -Effective value: `agent.max_concurrent_jobs().or(global).unwrap_or(5)`. -`jobs_enabled = function_calling_support && effective_max_concurrent_jobs > 0` -(R8); when `jobs_enabled` is false the feature is OFF for that context: -- No `job__*` declarations appended (`append_job_functions()` called - conditionally at Agent::init, mirroring agent.rs:224-226, AND at the - non-agent `Functions::init` sites — rc.rs:3836, rc.rs:4177, - app_state.rs:71 — so plain sessions get jobs per R9). Every one of these - call sites checks the full `jobs_enabled` predicate, including - `function_calling_support` (pattern: agent.rs:231/238/252, where - skill/memory/rag functions already gate on it). -- No job prompt instructions in `interpolated_instructions()` (mirror - agent.rs:439-441, but on the full `jobs_enabled` predicate — with - function calling off, NO job instructions appear anywhere, since a model - that cannot emit tool calls must not be taught tool syntax). Plain - sessions get NO injected job prompt text at - all — guidance there comes solely from the job__* tool descriptions - (no plain-session analog of interpolated_instructions exists; this is - the simplest R8-consistent ruling). -- Prefix carve-outs added at rc.rs:2139-2150 (agent path role filter) and - rc.rs:2109-2128 (non-agent builtin filter) apply only when enabled. - These carve-outs are what make `.info tools` show job__* and what makes - job__* survive role/session/agent/graph-node `enabled_tools` filters - (R11); extend `select_functions_preserves_infra_tools_under_agent_filter` - (rc.rs:5642) accordingly. -- REPL surfaces (R10): add `job__` to the built-in prefix exclusion list - in `concrete_tool_names()` (rc.rs:1283-1308) — that ONE pool feeds - `.list tools` (rc.rs:2732), `.tool enable/disable` validation - (toggle_tool rc.rs:1356, incl. its disable-path pool materialization), - and the `.tool` tab completions (repl_complete rc.rs:3458). `.info - tools` (tools_info rc.rs:700) needs NO dedicated code — it renders - `select_functions` output. -- Supervisor init condition becomes - `can_spawn_agents || jobs_enabled`; when only jobs - are enabled, agent capacity is 0 (register-at-capacity rejection is - existing tested behavior). - -### `job__start { tool: string, arguments: object }` -- Description mirrors `mcp_invoke_*`: "`arguments` is the same object the - tool takes when called directly." `arguments` schema is free-form object. -- Validates the tool against the backgroundable whitelist (§4). Rejection is - a TEACHING error: names the tool, why it can't background (mutates agent - state / interactive), and the whitelist categories — the model WILL try - `job__start {tool: "memory__write"}`. -- Validates the tool is AVAILABLE in the calling context (R11): the - requested tool must appear in the PER-REQUEST DECLARED-NAMES STASH — - the set of function names actually sent to the model for the current - request, captured in `before_chat_completion` (see Validation hardening - rule 1 for the mechanism and why recomputing via `extract_role` is - WRONG). This inherently applies role/session/agent/LLM-node - `enabled_tools` filters and enabled-MCP-server filters, because the - stash IS the filtered output. Whitelisted-but-filtered-out → - TEACHING error: "'X' is not enabled in this context". Without this - check, `job__start` would be an `enabled_tools` bypass for narrowed - contexts (e.g. graph LLM nodes with a `tools:` list). -- Runs any pre-execution gate SYNCHRONOUSLY before returning — a detached - task cannot touch the tty (H3). AUDITED (§13.2): NO gates exist on the - external-tool path today (no approval prompts/denylists; the only - pre-spawn checks are dedup/loop/unknown-tool inside eval_tool_calls), so - this rule is future-proofing, not migration work. -- Returns immediately: `{ status: "ok", job_id: "job_", tool, message: - "Running in background. Check with job__check, block with job__collect, - cancel with job__cancel. You will receive a system_notifications entry on - completion. Jobs do not survive coyote exiting." }` (The - system_notifications sentence is transiently aspirational on a T3-only - tree — notifications land in T4; both merge in the same PR, accepted - window.) -- Capacity: separate per-context `max_concurrent_jobs` budget (R7: agent - override → global config → default 5) — jobs must not consume the agent - `max_concurrent` budget. Jobs skip the agent depth check (no depth - semantics). - -### `job__check { id }` -- Returns `{ status: running|completed|failed, id, tool, elapsed_secs, - output_tail, output_bytes_captured, tail_truncated }`. `status` is read - from the JobHandle's shared JobState cell (§6) — the JoinHandle result is - never peeked (it can't be without consuming it). `cancelled` is - deliberately absent from the enum: job__cancel removes the handle from - the registry, so a later check on that id hits the unknown-id error - below. -- Unknown id → teaching error: "No job 'X' is registered — it may have - already been collected or cancelled. job__list shows active jobs." -- `output_tail` = tail of the live ring buffer (§5) — progress telemetry, - NOT the result. Tail extraction uses `String::from_utf8_lossy` (the ring - can split a multibyte char at the wrap point). -- NEVER consumes the handle (R6). `agent__check` is ALIGNED to the same - pure-probe semantics in T2 (today it delegates finished→collect at - supervisor.rs:844); a finished job check returns status + tail preview + - the exact collect command; a finished agent check returns status + the - exact collect command (no preview — no ring buffer). -- Per-handle consecutive-check counter: after ~5 checks with no state - change — defined as the `(status, output_bytes_captured)` tuple - unchanged — append hint: "still running; call job__collect to block, or - do other work — a system notification will fire on completion" (H8's - semantic rate-limit, replacing tracker special-casing). - -### `job__collect { id, tail_lines?: number }` -- Mirrors `agent__collect`: poll loop with the same early-out that surfaces - `pending_escalations` instead of deadlocking (supervisor.rs:904-914 - pattern), then `Supervisor::take` + await. -- Result assembly per tool class (§5): argc/external → read `LLM_OUTPUT` - with the hardened missing/empty semantics, plus final `output_tail` and - exit status; MCP → the future's return value. Ownership split (gate-ruled): - the JOB TASK reads LLM_OUTPUT after `wait()` (protocol semantics §4) and - returns the RAW output in `JobResult`; the COLLECT HANDLER applies the - default tail cap / `tail_lines` and reports `result_truncated` in its own - response. -- `JoinError` (panic) maps to `status: "failed"` with the panic message — - never hangs — and still returns whatever ring-buffer content exists (H6). -- Output capping (RESOLVED §13.6): `max_tool_result_chars` CANNOT be the - backstop — its global default is null/no-cap (config mod.rs:266, - config.example.yaml:206) and `truncate_if_needed` keeps the HEAD - (mod.rs:404-413), the wrong end for build logs (failures land at the - tail). Job results get their own TAIL-biased cap: default keep the LAST - 50,000 chars with a `[truncated: kept last N of M chars]` header, plus - the optional `tail_lines` param for explicit control (`tail_lines` - applies to the RESULT text — LLM_OUTPUT contents / MCP return — never to - the ring buffer). Both the default cap and `tail_lines` cuts floor to a - char boundary (do not reintroduce the §9.1 UTF-8 bug on the tail side). - No LLM summarization in v1 (supervisor.rs:1401-1451 ships the entire - output to the summarizer as one message — a huge log would blow its - context window anyway). - -### `job__cancel { id }` -- Take handle → kill process group per §6 platform strategy (only if the - JobState pgid is still set) → abort JoinHandle → ≤5s join wait → - `{ status: "cancelled", id, output_tail }` (partial output included). - -### `job__list` -- All registered jobs for THIS agent's supervisor: id, tool, status (from - the JobState cell), elapsed, bytes captured. - -Cross-kind misuse errors teach: `agent__collect("job_x")` → "'job_x' is a -background job, not an agent — use job__collect"; `job__collect("agent_...")` -→ inverse. IDs are namespaced (`job_` prefix) to make this cheap. - -### Validation hardening (loophole audit, 2026-08-24) - -WHY this matters more than it looks: in the FOREGROUND, tool access is -enforced by request declaration — providers only let the model call tools -sent in the request payload, so `ToolCall::eval` merely checks POOL -membership (`extract_call_config_from_ctx` bails "Unexpected call" for -undeclared names, mod.rs:1814-1825; test mod.rs:2564) and then resolves -the name against bin dirs + PATH (`run_llm_function`). `job__start`'s -`tool` parameter is a FREE STRING — it bypasses the provider-level -constraint entirely, so the R11 declared-names check is THE enforcement -point (and is strictly tighter than foreground's pool check: enabled set, -not pool). Binding rules: - -1. **Exact-match only, validated name drives everything.** The R11 check - is exact string equality (post-trim) against the set of tool names - ACTUALLY DECLARED to the model for the CURRENT request. It must NOT - recompute `select_functions(&extract_role())` at eval time: - `extract_role` layers session → agent → role (rc.rs:996-1019), and - during a graph LLM-node run it returns the UN-narrowed - `agent.to_role()` while the request was built from the node's - swapped-in role (`ctx.role` — graph/llm.rs:174-188, 251-252 → - input.rs:433 `functions: ctx.select_functions(role)`), so recomputing - would validate against the FULL pool and reopen the exact bypass R11 - closes (Oracle B1). Mechanism (Oracle-ruled 2026-08-24): - `before_chat_completion` (rc.rs:896) — already called at ALL SEVEN - turn-loop sites (repl/mod.rs:1435/1444, main.rs:562/573/632, - acp/server.rs:204, supervisor.rs:456 child loop, graph/llm.rs:257) — - stashes the request's declared-function NAME SET (a cheap - `HashSet` from the Input's select_functions output, NOT a - Role clone) into ctx; `job__start` validates against the stash. - REFRESHED on every request — a stale stash from a prior turn must be - impossible. Desirable side effect: a mid-batch `skill__load` widening - the pool does NOT grant `job__start` access until the next request — - the check is literally "declared to the model this turn". - `mapping_tools` aliases are NOT expanded — concrete - declaration names only. The raw model string is NEVER passed to - `run_llm_function`/bin-dir/PATH resolution or to the agent-tool - mapping (mod.rs:1785-1812): class dispatch (external vs `mcp_invoke_*`) - and cmd resolution derive from the MATCHED declaration, not from the - model's string. (Otherwise `job__start {tool: "bash"}` or a - path-shaped name would execute an arbitrary PATH binary that was - never a declared tool.) -2. **Both gates always run**: §4 whitelist AND R11 context availability; - a name must pass both regardless of order (each has its own teaching - error). -3. **Disabled state never falls through.** VERIFIED eval order (Oracle - B2): `extract_call_config_from_ctx/from_agent` runs BEFORE the prefix - chain (mod.rs:1420-1428) and bails "Unexpected call" for any name not - in the declaration pool (mod.rs:1814-1825) — so with jobs disabled - (job__* never declared, R8) a hallucinated `job__start` dies at that - bail as a per-call soft-fail and can never reach the external-command - fallthrough. RULING (B2 option (b)): this existing "Unexpected call" - soft-fail IS the specified disabled-state behavior — exactly what - agent__* produces in non-agent contexts today. Option (a) - (pre-extract routing on the raw name to emit a nicer teaching error) - REJECTED: reordering eval semantics for an error-message nicety, zero - security gain; the mod.rs:2564 test must stay green as-is. The - `job__` prefix arm in the chain consequently only ever executes when - declarations exist (jobs enabled). -4. **MCP server derivation from the validated name.** `JobCtx`'s - source, the full `McpRuntime` map, contains every STARTED server - (src/config/tool_scope.rs:44-47), not just context-enabled ones. The - server is derived solely from the validated `mcp_invoke_` - declaration name (the invoke_mcp_tool pattern, mod.rs:1614-1636, - prefix-strip at :1619) — and to - make the discipline STRUCTURAL (Oracle N3): `JobCtx` is constructed - at `job__start` AFTER validation, so it snapshots a SINGLE-ENTRY - `McpRuntime` holding only the validated server's - `Arc` — the job task cannot reach any other server - even by bug. The - inner `tool` argument stays foreground-parity (opaque server-side - name — no new gate, none exists today). -5. **Handle isolation across contexts.** All five `job__*` handlers - operate EXCLUSIVELY on `ctx.supervisor` — never `parent_supervisor` - (which exists in child contexts for escalations) — so a subagent - cannot check/collect/cancel its parent's jobs or vice versa. Forged - or stale ids hit the cross-kind/unknown-id teaching errors. -6. **Residual, accepted + documented**: job OUTPUT is untrusted text — - the same prompt-injection surface as foreground tool output. It cannot - mint declarations, grant tools, or forge handles; worst case is - persuading the model to call tools it already has. No v1 mitigation - beyond the existing one (output is data, delivered inside a - ToolResult). - -## 4. Backgroundable-tool whitelist - -Forced by the architecture: `ToolCall::eval` takes `&mut RequestContext`, -which cannot move into a detached task (H2). Backgroundable = tools whose -execution can run from an owned snapshot: - -| Class | Backgroundable | Execution path in job task | -|---|---|---| -| `execute_command` + external command tools — bash (argc), JavaScript/TypeScript, Python (user-ruled 2026-08-21; no per-tool opt-in flag) — EXCEPT the fast grep-class carve-out row below | YES (the whole point) | Extract the spawn logic out of the eval path into a free function taking `(Arc, JobEnvSnapshot, args)` — do NOT route through `ToolCall::eval`. `JobEnvSnapshot` = the audited field set below. | -| `mcp_invoke_*` | YES | Already `&ctx` (`eval_mcp`); job task owns `JobCtx { mcp_runtime: McpRuntime, current_depth: usize }` — the runtime is a SINGLE-ENTRY snapshot holding only the validated server's `Arc` (§3 hardening rule 4, Oracle N3). Job MCP path = `invoke` → `render_tool_result` (mod.rs:1635 foreground parity; FREE function, see §13.1). AUDITED (§13.1): that is eval_mcp's COMPLETE transitive ctx surface. McpRuntime is `#[derive(Clone)]`, shallow Arc map (src/config/tool_scope.rs:44-47); OAuth refresh is transport-embedded (auth_client.rs) and needs no AppState. NOTE: MCP jobs have NO timeout (`COYOTE_TOOL_TIMEOUT` is process-path only); a hung MCP job is recoverable via `job__cancel` (abort drops the future) — accepted v1, stated in the tool description. | -| `mcp_search_/mcp_describe_/mcp_read_/mcp_prompt_` | NO (pointless — fast) | Teaching error: "sub-second call; invoke directly." (All four non-invoke meta-families; meta-declarations are capability-gated per server — `gated_meta_function_prefixes`, function/mod.rs:416-426: invoke⇔tools, read⇔resources, prompt⇔prompts — which composes with the declared-names stash automatically: a capability the server never advertised is never declared, so `job__start` rejects it with the standard not-declared error.) | -| `agent__*`, `job__*` | NO | "Already asynchronous — use them directly." | -| `todo__*`, `memory__*`, `skill__*`, `user__*` and other internal `&mut ctx` tools | NO | "Mutates agent/session state; must run in-turn." | -| fs_* / ast_grep / grep-class builtins | NO in v1 | Fast; not worth the snapshot surface. Revisit only with evidence. | - -Snapshot semantics (document in tool description): config/model/env changes -made after `job__start` do not affect a running job. The job must not mutate -shared session state — it only produces output. - -Directionality ruling (user, 2026-08-24): the async systems compose ONE WAY -ONLY — agents (and graph LLM nodes) may start jobs, but a job can NEVER -start an agent, another job, or invoke any built-in — anything else would -duplicate/defeat the agents' own parallelization system. This is enforced -twice: (1) the whitelist rows above (policy, with teaching errors); (2) -architecturally — every built-in handler requires `&mut RequestContext`, -which cannot move into a detached task (H2); a job task owns only -`JobEnvSnapshot`/`JobCtx`, so there is no ctx, supervisor, or eval loop -inside a job for a built-in to run against, even if the whitelist check -were bypassed. - -### JobEnvSnapshot & runner mechanics (§13.2 audit, 2026-08-21) - -| Field | Source (foreground receipt) | -|---|---| -| `cmd_name`, `cmd_args` | tool name / agent-tool mapping + JSON args pushed as last arg (mod.rs:1785-1812, 1269) | -| `envs` | `agent.variable_envs()` — `LLM_AGENT_VAR_*` with vault-secret interpolation (agent.rs:469-485); resolved INTO the snapshot at job__start (same plaintext-in-memory exposure as foreground) | -| `agent_name` | drives bin dir + AUTO_CONFIRM-for-graph (mod.rs:2136-2149) | -| `PATH` | functions/agent bin dirs prepended (mod.rs:2136-2149); env-derived dirs (paths.rs:285-287, 327-329) FROZEN at job__start per snapshot semantics | -| `LLM_OUTPUT` | fresh temp path, file not pre-created (mod.rs:2158-2159, utils/mod.rs:313-320) | -| `CLICOLOR_FORCE`/`FORCE_COLOR` | hardcoded =1 (mod.rs:2178-2179) | -| `COYOTE_TOOL_TIMEOUT` | default 1800s, 0 = unlimited (mod.rs:2247-2251); resolved at job__start; jobs honor it (expiry → kill process GROUP + `failed` status) | -| cwd | inherited (foreground sets none) | -| Windows | `LLM_TOOL_DATA_FILE` arg spill + PATHEXT polyfill (mod.rs:2161-2176) | - -Runner-mechanics rulings from the audit: -- The external branch of `ToolCall::eval` never actually uses `&mut` — - `run_llm_function(cmd_name, cmd_args, envs, agent_name)` takes zero ctx - (mod.rs:1246-1252) — so the extraction is clean, not a refactor risk. -- The foreground runner is SYNC `std::process` polled at 100ms from async - without spawn_blocking (mod.rs:2181-2316). v1 leaves the foreground path - UNTOUCHED; the background runner uses `tokio::process` + - `process_group(0)`. (Foreground sync-in-async is a pre-existing latent - issue — note as follow-up, out of scope.) -- Foreground TEES stdout/stderr live to the terminal via threads - (mod.rs:2193-2245); background jobs are CAPTURE-ONLY into the ring - buffer — no live tee (it would interleave with the foreground UI). -- `stdin = Stdio::null()` is already foreground behavior (mod.rs:2184) — no - delta for jobs. -- LLM_OUTPUT protocol semantics to replicate when the job task reads the - file after `wait()` (mod.rs:2283-2316): - nonzero exit → `tool_call_error` + stderr/stdout + partial output; zero - exit + missing/empty file → null → "DONE"; unreadable existing file → - hard error; timeout → kill + `tool_call_error`. - -## 5. Output channels: ring buffer vs result (H1 — do not conflate) - -For process-backed jobs there are TWO distinct channels: - -1. **Live telemetry**: stdout+stderr streamed into a bounded ring buffer on - the `JobHandle` (`Arc>`, default 64 KiB). Carries a - monotonic total-bytes counter and a truncation marker so the model knows - the tail is clipped. This is what `job__check` returns and what makes - "did it produce any output yet?" answerable mid-run. -2. **The tool result**: for argc/external tools this is the `LLM_OUTPUT` - file, read ONLY once — by the job task after `wait()` (§3 ownership - split; the model sees it at collect) — with the same missing/empty-file - error semantics as the foreground path. The ring buffer is never substituted - for the result. For MCP jobs the result is simply the future's return - value; the ring buffer is unused/empty. - -Background processes get `stdin = /dev/null` — a script that prompts must -fail fast, not hang the job forever (H3 corollary; already true in the -foreground runner — `Stdio::null()` at mod.rs:2184 — so no behavior delta). - -## 6. JobHandle, JobState, process lifecycle, and cleanup (H4) - -```rust -// src/supervisor/mod.rs (alongside AgentHandle) -pub struct JobHandle { - pub id: String, // "job_" - pub tool: String, - pub started_at: Instant, - pub join_handle: JoinHandle>, - pub abort_signal: AbortSignal, - pub state: Arc>, // shared with the job task - pub output_buf: Arc>, // telemetry channel (§5) - pub no_change_checks: u32, // §3 job__check counter -} - -pub struct JobState { - pub status: JobStatus, // Running | Completed | Failed - pub pgid: Option, // Unix pgid; None on Windows & MCP jobs; - // CLEARED by the job task right after wait() - // reaps the child (pid-reuse guard, see below) -} - -pub enum JobStatus { Running, Completed, Failed } -// job__check/job__list read this cell. Cancelled is unrepresentable here — -// cancel removes the handle from the registry (§3). The job task writes the -// final status (and clears pgid) BEFORE it returns. - -pub struct JobResult { - pub output: Value, // RAW tool result (job task reads LLM_OUTPUT after wait(); MCP return) - pub exit_code: Option, // process jobs only - pub output_bytes_captured: u64, -} -// The COLLECT HANDLER applies the tail cap / tail_lines (§3) and reports -// `result_truncated` in its own response — deliberately NOT a JobResult -// field (capping is a presentation concern owned by collect). - -pub enum TaskHandle { Agent(AgentHandle), Job(JobHandle) } // R1 -``` - -- `Supervisor.handles` becomes `HashMap`; `register()` - branches on variant for capacity; existing agent paths pattern-match. -- Supervisor creation per R9: `can_spawn_agents || jobs_enabled` (R8), - agent capacity 0 in jobs-only contexts — EAGER in `use_agent`, LAZY via - `job__start` - get-or-init in plain sessions (see R9 for the full site ruling incl. the - exit_agent rc.rs:4192 rule). Never-jobbing plain sessions keep - `supervisor: None`. -- **Kill discipline (pid-reuse guard — Oracle finding, MANDATORY)**: every - killer (`impl Drop for JobHandle`, `job__cancel`, `cancel_recursive`, - timeout expiry) kills the process group ONLY IF `state.pgid` is still - `Some`, and the job task sets `pgid = None` immediately after `wait()` - returns (the child is reaped; with `process_group(0)`, pgid == child pid, - so a stale killpg after reap can SIGTERM an innocent recycled pid). - Without this guard, the normal collect path (take → await → handle drop) - fires a killpg against a reaped group on EVERY successful job. -- Spawn processes with `process_group(0)`; cancellation = `killpg(SIGTERM, - grace 5s, SIGKILL)` then `JoinHandle::abort()` — `abort()` alone does NOT - kill OS processes, and `kill_on_drop` misses grandchildren. -- Context-reset sites (corrected — there is NO full-context `/clear`): - `use_agent` (rc.rs:4155) currently REPLACES `self.supervisor` without - cleanup — T1 adds `cancel_recursive()` on the old supervisor before the - replacement (Arc-drop alone is unreliable: child ctxs hold - `parent_supervisor` clones, rc.rs:494, deferring the Drop while the job - runs on with no transcript). `exit_agent` already calls it (rc.rs:4190). - A job surviving a context switch has no transcript to report into. -- **Shutdown mechanism**: explicit kill-all (cancel_recursive over the root - supervisor) on the REPL quit path, where destructors run today. Hard-exit - paths that bypass destructors (`process::exit` at main.rs:672/257, - logs.rs:18, config/mod.rs:783) and panics MAY orphan a process group — - documented, accepted v1 (§9.4.11 checks the normal path). Note: - `process_group(0)` detaches jobs from coyote's group, so the shell will - not reap them either. -- `cancel_recursive`/`cancel_all` must handle the Job variant (kill group - per the discipline above, not just abort signal). -- R5: nothing survives process exit; `job__start`'s response and the system - prompt say so. -- **Lock discipline** (copy the existing pattern, supervisor.rs:894-942): - NEVER hold the parking_lot supervisor lock across an await point in any - job handler; take what you need under a scoped read/write and drop the - guard before awaiting. The ring-buffer mutex is locked only for the - memcpy (never across awaits in the stdout pump task). - -Module placement: `TaskHandle`/`JobHandle`/`JobState`/`JobStatus`/`JobResult` -live in src/supervisor/mod.rs alongside AgentHandle; the five job__* -handlers, `RingBuf`, `JobEnvSnapshot`, `JobCtx`, and the extracted runner -live in src/function/jobs.rs (new file, mirroring -src/function/supervisor.rs); `NotificationQueue` lives in -src/supervisor/notifications.rs (new file, sibling and structural template: -src/supervisor/escalation.rs). - -### Platform strategy (gate finding — killpg is Unix-only) - -- Unix: set the group with std's `CommandExt::process_group(0)` - (std::os::unix::process — no crate needed); kill with - `libc::killpg(pgid, SIGTERM)` → 5s grace → SIGKILL. NEW SANCTIONED - DEPENDENCY: `[target.'cfg(unix)'.dependencies] libc = "0.2"` — Cargo.toml - has no nix/libc direct dep today; this is the approved addition. -- Windows: jobs remain ENABLED; cancellation/timeout uses tokio's - `Child::start_kill()` + `kill_on_drop(true)` — single-PID, grandchildren - may leak, which is exactly the foreground timeout path's behavior today - (mod.rs:2258-2266): parity, not regression. All group-kill code is - `#[cfg(unix)]`-gated. Win32 Job objects are out of scope v1. -- MCP jobs have no OS process: cancellation = abort signal + - `JoinHandle::abort()` on both platforms. - -## 7. Push notifications: per-context `NotificationQueue` - -### Ownership model (critical — opposite of escalations) -`NotificationQueue` follows the `supervisor` pattern in `new_for_child` -(fresh per child, rc.rs:493), NOT the `escalation_queue` pattern -(clone-inherited, rc.rs:497). Each agent gets notifications for ITS OWN -spawned jobs/agents. A shared inherited queue = first-drainer-wins race -delivering the root's notifications into a child's transcript. -Consequently: escalations stay `depth == 0` on the shared root queue; -notifications drain the ctx's OWN queue at ANY depth. - -### Event shape (terse — one line per event) -```json -{ "event": "job_completed" | "job_failed" | "agent_completed" | "agent_failed", - "id": "job_a1b2", "tool_or_agent": "execute_command", "status": "success", - "next_action": "job__collect --id job_a1b2 for output" } -``` - -### Producers -- Job task: pushes on completion/failure before returning. -- Agent task: the `tokio::spawn` wrapper in `handle_spawn` - (supervisor.rs:778-795) pushes into the SPAWNING ctx's queue before - returning. Always-on (R4). -- NO event on explicit `job__cancel`/`agent__cancel` (the cancel's own - ToolResult confirms it). YES on failure/panic — with one caveat: a PANIC - in the job task skips the push (the producer never runs). This is - INTENTIONAL and covered: the turn-end guardrail's - finished-but-uncollected predicate still surfaces the handle, and - collect's JoinError→failed mapping (H6) reports the panic. Do NOT "fix" - this with a Drop-guard push — it reopens double-delivery. - -### Delivery point 1 — mid-turn key-merge (the "push") -Generalize `inject_escalation_notification` (mod.rs:366-381) into ONE -`merge_system_channel(last: &mut ToolResult, escalations, notifications)` -applied at the end of `eval_tool_calls`, keeping the existing -AFTER-truncation ordering. Single pass is MANDATORY: two independent mergers -each applying the non-object wrap double-nest the output -(`{"output": {"output": ...}}`) (H9). Key order: `pending_escalations` first -(children are BLOCKED; completions are not urgent), then -`system_notifications` + a one-line instruction. - -Drain-time stale suppression: filter events against current supervisor -registration — if the handle was already `take()`n (model collected before -the drain), DROP the event. Otherwise the model chases a dead id into an -error loop. - -### Delivery point 2 — turn-end guardrail (H7 predicate fix) -`pending_agent_ids` → `pending_task_ids` returning `(id, kind, finished)`, -INCLUDING finished-but-uncollected handles (today's `Some(false)` filter -excludes them — supervisor.rs:48). Guardrail prompt renders two sections: -- still running: existing reclaim language, kind-specific commands; -- completed — collect NOW: exact `job__collect`/`agent__collect` commands - (instant on a finished handle, so this is cheap for the model). -On the `ForceTerminate` strike (3 reminders), discard finished results with -a logged warning — no infinite loop. NOTE: this predicate change also fixes -the latent agent bug where finished-but-uncollected output is silently -abandoned at turn end. All 5 guardrail call sites get this for free. -Deliberate delta, see §9.1. - -### Delivery point 3 — none -No other injection point exists or is needed (R3). If the model has nothing -else to do, blocking `job__collect`/`agent__collect` remains the correct -primitive; notifications improve the working-meanwhile case only. - -### Race sweep (Oracle-reviewed 2026-08-21 — all benign, no further holes) -- Job completes between drain and merge → event delivers on the next batch - drain or the turn-end guardrail catches the finished handle. Covered. -- Model collects before drain → stale suppression drops the event. Covered. -- Guardrail-prompted collect, then next batch drains the old event → stale - suppression. Covered. -- Duplicate mention (event + guardrail, same handle) → benign redundancy. -- check-then-collect on the JobState cell → no consuming race under R6; - collect's take-under-write-lock is atomic. - -## 8. Loop-detection & polling ergonomics (H8) - -- Exempt `job__check`, `job__list`, `agent__check`, `agent__list_running` - from `tool_tracker.check_loop` AND from `record_call` — recording them - would let `[check, X, check, X]` mask a real X-loop; they must be invisible - to the tracker. (Root tracker `new(2,3)` trips on just 2 identical calls.) -- Unbounded-polling backstop is the per-handle no-change counter hint (§3), - a semantic limit where it belongs — not a tracker special case. - -## 9. Regression parity: guarantee when jobs are disabled (user requirement) - -Hard requirement (user, 2026-08-21): with effective `max_concurrent_jobs: 0`, -ALL existing function-calling and agent__* behavior works IDENTICALLY to -today — as if the feature does not exist. - -### 9.1 The ONLY intentional behavior deltas (jobs-independent) - -Four approved changes apply even when jobs are disabled. Everything else is -bit-identical. Each ships as its own commit with its own dedicated tests so -it can be reviewed/reverted in isolation: - -| Delta | Ruling | What changes | -|---|---|---| -| Agent completion push notifications | R4 | `system_notifications` key can appear on the last ToolResult of a batch after an agent finishes; guardrail prompt gains a "completed — collect now" section. Own commit within T4. | -| `agent__check` never consumes | R6 | Finished-agent check returns status + the exact collect command (no preview — agents have no ring buffer) instead of delegating to collect (supervisor.rs:844). T2 commit (b). | -| Guardrail counts finished-but-uncollected | H7 | Turn-end with an uncollected finished agent now Injects instead of silently dropping the result (fixes latent output-abandonment bug). T2 commit (a). | -| `truncate_if_needed` UTF-8 boundary fix | §13.6 audit | Edge-case BUG today: a cap landing mid-UTF-8-char makes `s.get(..max_chars)` return None → falls back to the FULL untruncated string while still prepending the truncation marker (mod.rs:404-413). Fix: floor the cut to a char boundary. Foreground-visible only in the broken edge case. T2 commit (c). | - -If any of these must ALSO be gated off, say so before task -materialization — they are separable. - -### 9.2 Zero-diff invariants when jobs are OFF (encode as tests) - -1. Tool list byte-identical: no `job__*` declarations in agent or plain - sessions (`select_enabled_functions` output compared with effective - `max_concurrent_jobs` 0 vs >0, AND with `function_calling_support: - false` at any `max_concurrent_jobs` value — both legs of `jobs_enabled` - independently force the OFF state). -2. Prompt byte-identical: no job instructions in - `interpolated_instructions()` (plain sessions never get job prompt text - at any setting — §3); also byte-identical with - `function_calling_support: false` regardless of `max_concurrent_jobs`. -3. Supervisor creation condition unchanged for agent-only contexts: - `can_spawn_agents: false` + jobs 0 → `supervisor: None`, exactly today. -4. `eval_tool_calls` behavior on any batch without job__ calls: identical - partition/order/soft-fail/truncation/injection (pinned by T0 tests). -5. `Supervisor` agent paths (register/capacity/depth/take/cancel_recursive) - behave identically with the `TaskHandle::Agent` variant — the enum - refactor is mechanical; T0 tests written BEFORE T1 must pass unmodified - after it (except type-name churn). -6. `merge_system_channel` with zero notifications + pending escalations - produces byte-identical output to today's - `inject_escalation_notification` (both object and non-object wrap cases). -7. Loop-tracker behavior unchanged for all non-exempt tools; exemption list - is exactly {job__check, job__list, agent__check, agent__list_running} - (agent-check exemption is part of delta R6's ergonomics; verify it - cannot mask real loops via the interleave test). -8. No `NotificationQueue` allocation side effects in sessions that never - spawn/background anything (drain of an empty queue = no-op, no key - added). - -### 9.3 T0 characterization tests (write BEFORE T1; the refactor safety net) - -Pin current behavior for every §2 coverage gap (ranked by risk): -1. `handle_collect`: finished-agent take+await happy path; escalation - early-out returns `status: "pending"` without consuming; summarization - under/over threshold. -2. Guardrail: `ForceTerminate` after 3 strikes + `cancel_recursive` called; - counter reset on collect/cancel/no-pending; Inject prompt content (both - with and without escalations — exists, keep). -3. `eval_tool_calls` multi-call: MCP/sequential partition, result re-sort by - original index, per-call soft-fail isolation (one failing call doesn't - poison the batch), loop-alert result shape inside a batch, bail on - empty-after-dedup. -4. `handle_spawn`: capacity rejection, depth rejection, spawnable_agents - allow-list rejection — through the HANDLER, not just Supervisor units. -5. `max_tool_result_chars` truncation through eval_tool_calls (agent - override + global fallback + 0-disables). -6. `merge_tool_results` message-shape test. -7. `handle_check`: pin CURRENT finished→collect delegation, marked as - deliberately rewritten by T2 commit (b) so the T2 diff is explicit. -8. `handle_cancel` of a RUNNING (not pre-finished) agent: abort + wait path; - direct `cancel_recursive` unit test. -9. `ToolCall::eval` prefix-routing table test (each prefix → expected - handler family, unknown → catalog-hint error). -10. `run_child_agent`: BEST-EFFORT — needs a mock client; if infeasible - without large scaffolding, document as manual case (§9.4) instead of - faking it. - -T0 merges before any refactor commit; T1+ must keep T0 green (allowing only -mechanical type renames), except tests explicitly marked for T2's deliberate -deltas. - -### 9.4 Manual verification checklist (user-runnable, post-implementation) - -Run once with `max_concurrent_jobs: 0` (global), once unset (default 5), -and once with `function_calling_support: false` (any `max_concurrent_jobs`) -— the last leg must show NO `job__*` declarations and NO job instructions -anywhere in the assembled prompt (R8): -1. Plain REPL: chat + `execute_command` + an fs_* call + an MCP call — works - as today; `job__*` absent from the tool list (0-case) / present - (default case). -2. Agent session (e.g. a spawning-capable agent): fan out 2 explores → - check → collect both; verify outputs and summarization. -3. Escalation round-trip: child asks a user__* question → parent sees - `pending_escalations` on last tool result → `agent__reply_escalation` - unblocks child. -4. Guardrail: force a turn-end with a running agent → `[SYSTEM GUARDRAIL]` - message; let it strike 3 times → force-cancel. -5. Task queue: `task_create` with deps + auto-dispatch agent on - `task_complete`. -6. Mailbox: `send_message` → child `check_inbox`. -7. Ctrl-c mid-stream (partial text kept, turn ends), then `.recover`. -8. Auto-continue with todo list pending. -9. Load a pre-existing saved session (incl. one containing the old phantom - `__escalation_notification` if available) — replays benignly. -10. `.agent` enter/exit — tool list correct on both sides of `exit_agent`; - with a job running, entering/exiting an agent KILLS the job (§6 - reset-site rule). -11. (jobs enabled) `job__start` a 30s `execute_command` → `job__check` - twice (no loop alert; tail visible) → do other tool work → observe - `system_notifications` on completion → `job__collect` → result correct; - then a cancel case; then quit coyote (normal REPL quit) mid-job → - verify no orphan process (`pgrep`). -12. REPL surfaces (R10) + filtered contexts (R11), jobs enabled: - `.info tools` lists the five `job__*` entries; `.list tools` omits - them; `.tool enable`/`.tool disable` tab completion never offers them; - `.tool enable job__start` → "Unknown tool". An agent/role with an - `enabled_tools` filter still sees `job__*`; `job__start` of a - filtered-out (but whitelisted) tool → context-availability error; - `job__start` of an in-filter tool works. - -## 10. Prompt & docs updates - -- Roles/system prompts (src/config/prompts.rs:118-119,180 region + agent - definitions in assets/): document the `job__*` family alongside `agent__*`; - update the wait-protocol guidance: "for long-running commands, `job__start` - and keep working; completion arrives as a `system_notifications` entry on - your next tool result; collect blocks only when you have nothing else to - do." Note jobs die with the process (R5) and snapshot semantics (§4). Job - instructions injected only when enabled (R8). Graph-node line (§12 - iteration-burn hazard): "in graph LLM nodes, collect or cancel your jobs - before ending your final node turn — an uncollected job at node turn-end - burns node iterations via the guardrail and can fail the node." -- Repo assets are canonical. Sync mechanism: T6 produces the list of every - modified file under assets/; after merge, each is copied (plain `cp`) to - its mirrored path under ~/.config/coyote/ (e.g. assets/agents// - index.yaml → ~/.config/coyote/agents//index.yaml). That list is - recorded as a follow-up item in the PR body. Built-in prompt text in - src/config/prompts.rs is compiled into the binary and needs no sync. -- config.example.yaml: `max_concurrent_jobs` next to - compression_threshold/max_tool_result_chars (~lines 200-206) with the - `0 = disabled` semantics documented. -- CHANGELOG + README/docs section for the new tools. -- **Wiki (user-required 2026-08-24)** — users must be able to discover and - understand the subsystem. Target: the GitHub wiki - (github.com/Dark-Alex-17/coyote/wiki — a SEPARATE git repo, - `coyote.wiki`, OUTSIDE this run's write boundary). Required content, new - page `Background-Jobs.md`: - 1. What background jobs are (whitelisted tool calls running as detached - tasks) and when to use them vs. agents (§4 directionality ruling: - agents can start jobs, never the reverse — jobs run single tool - calls; agents think); - 2. The five `job__*` tools with a worked example (start a long build → - keep working → notification arrives → collect); - 3. The backgroundable whitelist + the teaching errors users will see; - 4. Push notifications: the `system_notifications` key and the turn-end - guardrail, in user-visible terms; - 5. Configuration: global `max_concurrent_jobs`, per-agent override, - `0` disables, and the function-calling requirement (R8) — - jobs simply don't appear otherwise; - 6. Behavioral fine print: snapshot semantics, jobs die with the coyote - process (R5, no persistence), output tail cap + `tail_lines`, - `job__*` visible in `.info tools` but deliberately not toggleable - via `.tool` (R10), graph LLM-node guidance (collect before the - final node turn). - Cross-links: Home.md feature blurb + README features list entry - pointing at the new page, and a "jobs vs. subagents" note on the - existing agents wiki page. Process (macros-run precedent, coyote.wiki - 198f82c): T6 DRAFTS the full page content and cross-link diffs as a - task artifact (task log.md); actual publication to coyote.wiki is - recorded in the PR body's Follow-up section and executed AFTER merge — - the wiki must only ever describe merged behavior, and the wiki repo is - outside the run branch's write target. - -## 11. Implementation sketch (task-shaped) - -0. **T0 — characterization tests (§9.3).** Pin current behavior for every - coverage gap. Merges FIRST; no production code changes. -1. **T1 — TaskHandle enum + Supervisor generalization.** `TaskHandle`, - `HashMap`, per-kind capacity (R7 resolution: - agent-override → global → 5; 0 disables), namespaced ids, JobState cell - + kill discipline (§6), `cancel_recursive`/`cancel_all` Job-variant - handling, Drop-kill (pgid-guarded), cross-kind teaching errors, - supervisor init condition (R9), `use_agent` gains `cancel_recursive()` - on the old supervisor before replacement (rc.rs:4155 — §6 reset-site - fix), explicit kill-all on the REPL quit path (§6 shutdown). Config - plumbing: global Config field + AgentConfig Option + all four AppConfig - touch points. T0 stays green. -2. **T2 — Deliberate deltas (THREE separate commits — §9.1 isolation).** - Commit (a): guardrail predicate fix — `pending_task_ids` incl. - finished-but-uncollected, kind-aware prompt, ForceTerminate - discard-with-warning (H7). Commit (b): align `agent__check` to - pure-probe semantics (R6, supervisor.rs:844) with tool-description/ - prompt updates. Commit (c): `truncate_if_needed` UTF-8-boundary fix. - Each commit rewrites its own T0-marked tests — explicitly, in that - commit. Independently shippable; fixes the agent output-abandonment - bug even without jobs. -3. **T3 — job runner + `job__*` handlers + conditional injection (R8/R9).** - Extracted process runner (`JobEnvSnapshot`, process_group(0) per §6 - platform strategy, stdin=null, ring buffer, LLM_OUTPUT read post-wait()), - `JobCtx` for MCP invokes, the five handlers (lock discipline per §6), - whitelist + teaching errors, sync pre-execution gates in `job__start`, - context-availability validation (R11) against the per-request - declared-names stash captured in `before_chat_completion` (rc.rs:896; - all seven turn-loop call sites — §3 hardening rule 1, Oracle B1), - single-entry JobCtx McpRuntime (§3 rule 4, Oracle N3), graph-LLM-node - lifecycle semantics (§12, Oracle B3), REPL-surface wiring (R10: - `job__` in `concrete_tool_names()` exclusion; `.info tools` verified - free), - declaration/prompt gating at all injection sites (agent init, - plain-session Functions::init sites, exit_agent rebuild, - select_enabled_functions carve-outs). Per the §13.2 audit: background - runner on `tokio::process` (foreground stays sync-std, untouched); - capture-only, no live tee; `COYOTE_TOOL_TIMEOUT` resolved at start and - enforced with a pgid-guarded group kill; env-derived bin dirs + - vault-interpolated agent envs frozen into the snapshot at start. - Includes the tail-biased result cap + `tail_lines` param (§3, char- - boundary floored). PLUS (Oracle finding 8): an explicit audit item — - enumerate every `ctx.supervisor`/`parent_supervisor` consumer - (guardrail, taskqueue/mailbox handlers, REPL displays, session - save/load) and verify each behaves correctly with an agent-capacity-0 - supervisor (the novel R9 default-on state in plain sessions). -4. **T4 — NotificationQueue + merge_system_channel.** Per-ctx queue, - producers (jobs, then the agent spawn wrapper), single-pass merger - refactor replacing `inject_escalation_notification` (byte-identical - output when notifications are empty — §9.2.6), stale suppression, - no-notify-on-cancel, panic-skip semantics per §7. Wire drain into - `eval_tool_calls` at any depth. The agent-producer (R4 — a §9.1 delta) - is its own commit within T4. -5. **T5 — loop-tracker exemptions + no-change check hint.** -6. **T6 — prompts/docs/CHANGELOG/config.example.yaml + config sync (§10), - including the README features-list entry and the DRAFT of the - `Background-Jobs.md` wiki page + cross-link diffs (§10 wiki bullet; - publication to the separate coyote.wiki repo is a post-merge follow-up - recorded in the PR body, never done on the run branch).** -7. **T7 — feature tests**: §9.2 invariants (0 vs >0 states), double-wrap - regression (escalation+notification same batch, non-object output), - stale-notification suppression, orphan process-group kill (incl. - grandchild, Unix), pgid-guard (no kill after normal collect), - LLM_OUTPUT-vs-ring-buffer split, JoinError→failed mapping, guardrail - enumeration of finished handles, tracker exemption masking test - (`[check, X, check, X]` still detects X), cross-kind id errors, - use_agent/exit_agent kills jobs, jobs-only supervisor (agent capacity 0) - rejects agent__spawn with today's capacity error, tail-cap char-boundary - tests. PLUS R10/R11 surface tests: `concrete_tool_names()` excludes - `job__`; extended infra-preservation test (job__* survive agent/role - `enabled_tools` filters, incl. the empty-list case); `job__start` - rejects a context-filtered tool and accepts an in-filter one; toggle - of `job__start` errors as unknown. PLUS §3 validation-hardening tests: - `job__start {tool: "bash"}` / a path-shaped name / a PATH-resolvable - binary that is not a declared tool → rejected with NO process spawned - (assert the runner is never invoked); jobs-disabled context + - hallucinated `job__start` → eval's existing "Unexpected call" - soft-fail (B2 option (b) ruling; mod.rs:2564 stays green); - `job__start {tool: "mcp_invoke_X"}` for - a started-but-not-context-enabled server X → R11 rejection; subagent - `job__check/collect/cancel` cannot reach a parent-supervisor job id - (isolation → unknown-id error); declared+enabled but NOT whitelisted - (e.g. `memory__write`, `fs_read`, and the directionality cases - `agent__spawn`/`user__select` — §4 ruling) → whitelist teaching error, - no spawn - (Oracle N1 — rule 2's owning test); `mapping_tools` alias name → - rejected (concrete names only, rule 1); stash freshness — stash - refreshed per request, mid-batch `skill__load` does not grant until - next request (B1); None→Some flip pinned both states + jobs-off guard - for the rc.rs:5493 None-test (N2); `.info tools` positive assertion - automated via the tools_info unit tests (rc.rs:6025-6079, N5); - graph-node job lifecycle: node-started job registers in the shared - ctx.supervisor, survives node completion, notification drains on a - later turn of the same ctx; guardrail iteration-burn characterized - (B3). - -Dependency order & worktree parallelization (Oracle-confirmed): -T0 → T1 (strictly sequential; T0 is the safety net) → **T2 ∥ T3** (disjoint -file sets EXCEPT function/mod.rs — T2 commit (c) touches :404-413 while T3 -touches the :1420 routing region: non-overlapping hunks, rebase-safe, not -conflict-free; otherwise T2 = function/supervisor.rs, T3 = function/jobs.rs -+ agent.rs/rc.rs gating + config) → T4 (after BOTH — touches -mod.rs's merger and supervisor.rs's spawn wrapper) → **T5 ∥ T6** → T7 last -(may overlap T6 only). - -## 12. Risks - -- **Prompt-budget creep**: `system_notifications` + guardrail enumeration + - `pending_escalations` can stack; keep entries one-line terse. -- **Snapshot drift**: JobEnvSnapshot must capture exactly what the foreground - runner reads, or background runs behave differently — enumerate the field - set during T3 with a test comparing foreground/background env of the same - tool. -- **AppConfig four-touch-point trap** (R7): struct, Default, From, - env override — missing one silently pins the default. -- **Sandbox mode**: RESOLVED (§13.5) — no risk. Sandboxed coyote runs INSIDE - the container (kit entrypoint IS the coyote binary; `sbx run` attach at - sandbox/mod.rs:1093-1100); tools are always local children of coyote; - src/function/ has ZERO sandbox awareness. killpg works identically in and - out of sbx. (Pre-existing caveat, unchanged by this design: the foreground - timeout path uses single-PID `child.kill()`, so grandchildren can survive - a foreground timeout — background jobs fix this for themselves via - process groups on Unix.) -- **MCP server lifecycle vs running jobs**: a job's `Arc` - keeps the old service instance alive across a registry restart/shutdown - mid-job — the job finishes against the OLD server. Acceptable v1; - document in the tool description. -- **Hard-exit orphans**: `process::exit`/panic paths skip destructors and - jobs are in their own process group — accepted v1 (§6 shutdown - mechanism); normal REPL quit kills all. -- **ACP/graph surfaces** — CORRECTED + RULED (Oracle B3, 2026-08-24): the - old "graph mode out of scope" claim conflated two node kinds. The - supervisor bypass (`run_agent_for_graph`, supervisor.rs:510-596) means - the jobs/notifications exclusion applies ONLY - to graph AGENT nodes (the `run_agent_for_graph` path). Graph **LLM - nodes** are IN SCOPE — consistent with R11's user-confirmed text: they - are one of the §2 turn-loop sites, run on the parent session's - `&mut RequestContext`, and already call the guardrail (graph/llm.rs:271). - Semantics (owned by T3, tested in T7): a node-started job registers in - the SHARED ctx.supervisor and OUTLIVES the node run — mechanically - coherent: notifications drain into later turns on the same ctx, and the - turn-end guardrail surfaces still-running handles. Iteration-burn - hazard: the guardrail Inject arm inside the node's run_chat_loop - consumes node `max_iterations` and bails the WHOLE node at the limit - (graph/llm.rs:281-291) — a node that backgrounds a long job with - nothing else to do converts success into an error. Mitigation is - prompt guidance (§10: collect/cancel before the final node turn), NOT - a mechanical carve-out in v1. - -## 13. Open questions / VERIFY — ALL RESOLVED - -1. **JobCtx vs full RequestContext clone** — RESOLVED 2026-08-21 (audit): - purpose-built `JobCtx { mcp_runtime: McpRuntime, current_depth: usize }` - WINS decisively. eval_mcp's complete transitive ctx surface is exactly - `ctx.tool_scope.mcp_runtime` (+ `current_depth` for print gating) — - the single partition + concurrent join in `eval_tool_calls` - (mod.rs:291-326) and the five `eval_mcp` arms (mod.rs:1367-1418); - `McpRuntime::invoke` is self-contained and returns the raw - `CallToolResult` (tool_scope.rs:289-304). Post-invoke, the foreground - routes that result through `render_tool_result` (mod.rs:1989, called at - :1635) — a FREE function (spill under `paths::cache_dir()`/mcp-resources, - `TEXT_MAX_BYTES_CLAMP` paging; zero ctx) — and the job path calls the - SAME function for parity, so the audit conclusion is UNCHANGED after the - 2026-08-25 MCP resources/prompts merge. RequestContext has NO Clone impl - at all; the nearest equivalent `fork_for_branch` (rc.rs:433-463) - deep-copies the ENTIRE session transcript (Vec ×2) + Functions - and would drag parking_lot supervisor locks into the detached task. - OAuth needs nothing from ctx: per-request bearer injection + 401 - force-refresh-retry-once live INSIDE the transport (`McpOAuthClient`, - auth_client.rs:42-99) with a process-global token store and per-server - single-flight locks — mid-job refresh works through JobCtx unchanged. -2. **JobEnvSnapshot field set** — RESOLVED 2026-08-21 (audit): exact field - table + runner-mechanics rulings in §4. Notable: NO pre-execution gates - exist (H3 is future-proofing); the external eval branch never uses - `&mut ctx` (clean extraction); foreground runner is sync-std polled from - async (background uses tokio::process; foreground untouched v1); - live-tee threads must not run for jobs; vault-interpolated agent envs - resolve into the snapshot at start (same exposure as foreground). -3. **`agent__check` consume-on-finished quirk** — RESOLVED 2026-08-21 - (user, R6): align — check never consumes, for agents AND jobs; - deliberate behavior change, tool descriptions/prompts updated in T2/T6. -4. **Defaults & knob placement** — RESOLVED 2026-08-21 (user, R7 amended): - `max_concurrent_jobs` = global Config field (default 5, `0 = disabled`) - + Option AgentConfig override, resolved agent-first - (max_tool_result_chars pattern); ring buffer 64 KiB, no-change hint - threshold 5, SIGTERM grace 5s stand. Role/session-runtime overrides - deferred for both concurrency knobs together. -5. **Sandbox (sbx) execute_command** — RESOLVED 2026-08-21 (audit): clean - answer — sandbox mode does NOT touch tool execution. Coyote itself runs - inside the container (kit entrypoint = coyote; the only `sbx exec` calls - are launch-time setup, sandbox/mod.rs:1051-1128); tools are plain local - `std::process` children everywhere; `grep sandbox src/function/` = zero - matches. killpg is fully viable in sbx; no whitelist carve-out needed. -6. **Huge job outputs at collect** — RESOLVED 2026-08-21 (audit): plain - `max_tool_result_chars` truncation is INADEQUATE (default null = no cap; - head-keeping = wrong end for logs; UTF-8 boundary bug — §9.1). Ruling: - tail-biased job-result cap (default keep last 50,000 chars, explicit - header, char-boundary floored) + optional `tail_lines` param on - `job__collect` (§3); no LLM summarization v1. -7. **Whitelist boundary for custom tools** — RESOLVED 2026-08-21 (user): - ALL external command tools are backgroundable — bash (argc), - JavaScript/TypeScript, and Python — no per-tool opt-in flag; they are - process-isolated by construction. -8. **Feature-off gating** — RESOLVED 2026-08-21 (user, R8/R9): effective - `max_concurrent_jobs == 0` → job__* declarations and prompt text not - injected (can_spawn_agents-style gating); jobs available in plain - sessions via the global value (R9). diff --git a/src/acp/server.rs b/src/acp/server.rs index 689a472..c8cdcec 100644 --- a/src/acp/server.rs +++ b/src/acp/server.rs @@ -1,7 +1,7 @@ use super::types::{METHOD_NOT_FOUND, PARSE_ERROR, Request, Response}; use crate::client::call_chat_completions_streaming; use crate::config::{Input, RenderMode, RequestContext}; -use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail}; +use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::utils; use crate::utils::AbortSignal; use anyhow::Result; @@ -211,7 +211,7 @@ async fn run_prompt_turn( input = input.merge_tool_results(output, tool_results); continue; } - match check_pending_agents_guardrail(ctx) { + match check_pending_tasks_guardrail(ctx) { GuardrailAction::Inject(prompt) => { input = Input::from_str(ctx, &prompt, None)?; } diff --git a/src/config/agent.rs b/src/config/agent.rs index 2be8b31..9fbf663 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -454,8 +454,6 @@ impl Agent { output.push_str(DEFAULT_SPAWN_INSTRUCTIONS); } - // Job declarations are appended at init iff jobs are enabled for this - // agent, so their presence doubles as the jobs_enabled predicate. if self .functions .declarations() diff --git a/src/config/app_config.rs b/src/config/app_config.rs index a050234..4f4e4b1 100644 --- a/src/config/app_config.rs +++ b/src/config/app_config.rs @@ -844,8 +844,8 @@ mod tests { unsafe { match prev { - Some(v) => std::env::set_var(&env_name, v), - None => std::env::remove_var(&env_name), + Some(v) => env::set_var(&env_name, v), + None => env::remove_var(&env_name), } } } @@ -868,35 +868,35 @@ mod tests { #[serial_test::serial] fn load_envs_overrides_max_concurrent_jobs() { let env_name = get_env_name("max_concurrent_jobs"); - let prev = std::env::var_os(&env_name); + let prev = env::var_os(&env_name); let mut app = AppConfig::default(); - unsafe { std::env::set_var(&env_name, "7") }; + unsafe { env::set_var(&env_name, "7") }; app.load_envs(); assert_eq!(app.max_concurrent_jobs, Some(7)); - unsafe { std::env::set_var(&env_name, "0") }; + unsafe { env::set_var(&env_name, "0") }; app.load_envs(); assert_eq!(app.max_concurrent_jobs, Some(0)); - unsafe { std::env::remove_var(&env_name) }; + unsafe { env::remove_var(&env_name) }; app.max_concurrent_jobs = Some(2); app.load_envs(); assert_eq!(app.max_concurrent_jobs, Some(2)); unsafe { match prev { - Some(v) => std::env::set_var(&env_name, v), - None => std::env::remove_var(&env_name), + Some(v) => env::set_var(&env_name, v), + None => env::remove_var(&env_name), } } } #[test] fn editor_returns_configured_value() { - let configured = cached_editor() - .unwrap_or_else(|| std::env::current_exe().unwrap().display().to_string()); + let configured = + cached_editor().unwrap_or_else(|| env::current_exe().unwrap().display().to_string()); let app = AppConfig { editor: Some(configured.clone()), ..AppConfig::default() @@ -913,9 +913,9 @@ mod tests { return; } - let expected = std::env::current_exe().unwrap().display().to_string(); + let expected = env::current_exe().unwrap().display().to_string(); unsafe { - std::env::set_var("VISUAL", &expected); + env::set_var("VISUAL", &expected); } let app = AppConfig::default(); @@ -983,7 +983,7 @@ mod tests { let app = AppConfig::from_config(cfg).unwrap(); let ua = app.user_agent.as_deref().unwrap(); - assert!(ua != "auto", "user_agent should have been resolved"); + assert_ne!(ua, "auto", "user_agent should have been resolved"); assert!(ua.contains('/'), "user_agent should be '/'"); } diff --git a/src/config/app_state.rs b/src/config/app_state.rs index d4c9079..5cbd452 100644 --- a/src/config/app_state.rs +++ b/src/config/app_state.rs @@ -73,6 +73,7 @@ impl AppState { if !mcp_registry.is_empty() && config.mcp_server_support { functions.append_mcp_meta_functions(mcp_registry.server_features()); } + if jobs_enabled(None, &config) { functions.append_job_functions(); } diff --git a/src/config/input.rs b/src/config/input.rs index 85a6a68..52b389c 100644 --- a/src/config/input.rs +++ b/src/config/input.rs @@ -163,7 +163,6 @@ impl Input { self.data_urls.clone() } - /// Names of the function declarations this request will send to the model. pub fn declared_function_names(&self) -> HashSet { self.functions .as_ref() diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 25f9204..138b4ac 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -334,7 +334,7 @@ pub struct RequestContext { pub notification_queue: Arc, pub current_depth: usize, pub auto_continue_count: usize, - pub pending_agents_guardrail_count: u32, + pub pending_tasks_guardrail_count: u32, pub todo_list: TodoList, pub skill_registry: SkillRegistry, pub last_continuation_response: Option, @@ -369,7 +369,7 @@ impl RequestContext { notification_queue: Arc::new(NotificationQueue::new()), current_depth: 0, auto_continue_count: 0, - pending_agents_guardrail_count: 0, + pending_tasks_guardrail_count: 0, todo_list: TodoList::default(), skill_registry: SkillRegistry::default(), last_continuation_response: None, @@ -430,7 +430,7 @@ impl RequestContext { notification_queue: Arc::new(NotificationQueue::new()), current_depth: 0, auto_continue_count: 0, - pending_agents_guardrail_count: 0, + pending_tasks_guardrail_count: 0, todo_list: TodoList::default(), skill_registry: SkillRegistry::default(), last_continuation_response: None, @@ -478,7 +478,7 @@ impl RequestContext { notification_queue: self.notification_queue.clone(), current_depth: self.current_depth, auto_continue_count: 0, - pending_agents_guardrail_count: 0, + pending_tasks_guardrail_count: 0, todo_list: self.todo_list.clone(), skill_registry: self.skill_registry.clone(), last_continuation_response: None, @@ -524,7 +524,7 @@ impl RequestContext { notification_queue: Arc::new(NotificationQueue::new()), current_depth, auto_continue_count: 0, - pending_agents_guardrail_count: 0, + pending_tasks_guardrail_count: 0, todo_list: TodoList::default(), skill_registry: SkillRegistry::default(), last_continuation_response: None, @@ -923,6 +923,12 @@ impl RequestContext { pub fn before_chat_completion(&mut self, input: &Input) -> Result<()> { // `job__start` validates against exactly what was declared to the // model for THIS request; refresh it every time. + // + // This is necessary to prevent the model from invoking functions it + // otherwise wouldn't have access to by going through the free `tool` + // argument of `job__start`. If a function is disabled, the model + // shouldn't be able to invoke it at all in any way. This prevents + // that backdoor. self.declared_function_names = input.declared_function_names(); self.last_message = Some(LastMessage::new(input.clone(), String::new())); Ok(()) @@ -4082,11 +4088,6 @@ impl RequestContext { Ok(()) } - #[allow(dead_code)] - pub fn jobs_enabled(&self) -> bool { - jobs_enabled(self.agent.as_ref(), &self.app.config) - } - pub async fn use_agent( &mut self, app: &AppConfig, @@ -4179,7 +4180,7 @@ impl RequestContext { let jobs_enabled = jobs_enabled(Some(&agent), app); let should_init_supervisor = agent.can_spawn_agents() || jobs_enabled; - let max_concurrent = if agent.can_spawn_agents() { + let max_concurrent_agents = if agent.can_spawn_agents() { agent.max_concurrent_agents() } else { 0 @@ -4188,7 +4189,8 @@ impl RequestContext { let max_jobs = effective_max_concurrent_jobs(Some(&agent), app); let supervisor = should_init_supervisor.then(|| { Arc::new(RwLock::new( - Supervisor::new(max_concurrent, max_depth).with_max_concurrent_jobs(max_jobs), + Supervisor::new(max_concurrent_agents, max_depth) + .with_max_concurrent_jobs(max_jobs), )) }); @@ -4254,7 +4256,7 @@ impl RequestContext { self.notification_queue = Arc::new(NotificationQueue::new()); self.current_depth = 0; self.auto_continue_count = 0; - self.pending_agents_guardrail_count = 0; + self.pending_tasks_guardrail_count = 0; self.todo_list = TodoList::default(); self.rag.take(); // Cleared alongside `rag` so the pair never disagrees: an agent RAG is @@ -4750,18 +4752,22 @@ mod tests { use super::*; use crate::config::AppState; use crate::config::agent::AgentConfig; + use crate::function::jobs::RingBuf; use crate::function::{ToolCall, skill}; use crate::mcp::{McpServer, McpServerFeatures, McpServersConfig, McpTransportType}; + use crate::supervisor::{ + AgentExitStatus, AgentHandle, AgentResult, JobHandle, JobResult, JobState, JobStatus, + }; use crate::utils; use crate::utils::get_env_name; use crate::vault::Vault; use rmcp::model::PromptArgument; use serde_json::json; use serial_test::serial; - use std::env; use std::fs::{create_dir_all, remove_dir_all, write}; use std::path::PathBuf; - use std::time::{SystemTime, UNIX_EPOCH}; + use std::time::{Instant, SystemTime, UNIX_EPOCH}; + use std::{env, mem}; struct TestConfigDirGuard { key: String, @@ -5339,14 +5345,14 @@ mod tests { .unwrap() .block_on(async { let join_handle = tokio::spawn(async { - Ok(crate::supervisor::AgentResult { + Ok(AgentResult { id: "a1".into(), agent_name: "explore".into(), output: String::new(), - exit_status: crate::supervisor::AgentExitStatus::Completed, + exit_status: AgentExitStatus::Completed, }) }); - let handle = crate::supervisor::AgentHandle { + let handle = AgentHandle { id: "a1".to_string(), agent_name: "explore".to_string(), depth: 1, @@ -7696,7 +7702,7 @@ mod tests { } } - fn make_running_job(abort_signal: utils::AbortSignal) -> crate::supervisor::JobHandle { + fn make_running_job(abort_signal: utils::AbortSignal) -> JobHandle { // Leak the runtime so the spawned task is never polled and the job // stays running for the duration of the test. let rt = tokio::runtime::Builder::new_current_thread() @@ -7704,26 +7710,24 @@ mod tests { .build() .unwrap(); let join_handle = rt.spawn(async { - Ok(crate::supervisor::JobResult { + Ok(JobResult { output: serde_json::Value::Null, exit_code: Some(0), output_bytes_captured: 0, }) }); - std::mem::forget(rt); - crate::supervisor::JobHandle { + mem::forget(rt); + JobHandle { id: "j1".to_string(), tool: "execute_command".to_string(), - started_at: std::time::Instant::now(), + started_at: Instant::now(), join_handle, abort_signal, - state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { - status: crate::supervisor::JobStatus::Running, + state: Arc::new(parking_lot::Mutex::new(JobState { + status: JobStatus::Running, pgid: None, })), - output_buf: Arc::new(parking_lot::Mutex::new( - crate::function::jobs::RingBuf::default(), - )), + output_buf: Arc::new(parking_lot::Mutex::new(RingBuf::default())), no_change_checks: 0, last_check_state: None, } diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 7d140a4..0f4fd7d 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -70,17 +70,20 @@ impl RingBuf { if self.capacity == 0 { return; } + let src = if bytes.len() > self.capacity { &bytes[bytes.len() - self.capacity..] } else { bytes }; + for &byte in src { if self.buf.len() < self.capacity { self.buf.push(byte); } else { self.buf[self.write_pos] = byte; } + self.write_pos = (self.write_pos + 1) % self.capacity; } } @@ -93,6 +96,7 @@ impl RingBuf { if self.buf.len() < self.capacity { return self.buf.clone(); } + let mut out = Vec::with_capacity(self.capacity); out.extend_from_slice(&self.buf[self.write_pos..]); out.extend_from_slice(&self.buf[..self.write_pos]); @@ -367,7 +371,8 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { return Ok(json!({ "status": "error", "message": format!( - "'{tool}' is not enabled in this context — job__start can only background tools declared to you in this request. Use the exact name of a tool from your current catalog." + "'{tool}' is not enabled in this context — job__start can only background tools declared to you in this \ + request. Use the exact name of a tool from your current catalog." ), })); } @@ -464,7 +469,9 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { } else { JobStatus::Failed }; + drop(job_state); + task_notifications.push(job_notification(¬ify_id, ¬ify_tool, success)); result }) @@ -495,7 +502,8 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { "status": "ok", "job_id": job_id, "tool": tool, - "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting.", + "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. \ + You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting.", })) } @@ -520,12 +528,14 @@ fn handle_check(ctx: &RequestContext, args: &Value) -> Result { (buf.tail(), buf.total_written()) }; let check_state = (status, total_written); + if job.last_check_state == Some(check_state) { job.no_change_checks += 1; } else { job.no_change_checks = 0; job.last_check_state = Some(check_state); } + let tail_truncated = (tail.len() as u64) < total_written; let mut result = json!({ "status": job_status_str(status), @@ -536,10 +546,12 @@ fn handle_check(ctx: &RequestContext, args: &Value) -> Result { "output_bytes_captured": total_written, "tail_truncated": tail_truncated, }); + if matches!(status, JobStatus::Running) { result["message"] = json!( "Job is still running. Call job__collect to block for the result, or do other work — you will be notified on completion." ); + if job.no_change_checks >= 3 { result["hint"] = json!( "No change across repeated checks — still running; call job__collect to block, or do other work — a system notification will fire on completion." @@ -606,11 +618,14 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { let sup = supervisor.read(); sup.job(id).is_none_or(|job| job.join_handle.is_finished()) }; + if is_finished { break; } + time::sleep(Duration::from_millis(50)).await; } + break; } @@ -642,6 +657,7 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { if let Some(pgid) = handle.state.lock().pgid { unsafe { libc::killpg(pgid, libc::SIGKILL) }; } + match time::timeout(JOB_KILL_GRACE, &mut handle.join_handle).await { Ok(joined) => joined, Err(_) => { @@ -694,6 +710,7 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { "output_tail": output_tail, "output_bytes_captured": job_result.output_bytes_captured, }); + if let Some(exit_code) = job_result.exit_code { response["exit_code"] = json!(exit_code); } @@ -968,6 +985,7 @@ async fn run_process_job( "Tool call '{}' timed out after {}s and was killed (set COYOTE_TOOL_TIMEOUT to adjust; 0 = unlimited)", snapshot.display_name, snapshot.timeout_secs ); + return Ok(JobResult { output: json!({"tool_call_error": message}), exit_code: None, @@ -1011,6 +1029,7 @@ async fn run_process_job( { error_json["output"] = json!(contents); } + return Ok(JobResult { output: error_json, exit_code, @@ -1075,6 +1094,7 @@ async fn run_mcp_job( } }; let output = render_tool_result(serde_json::to_value(raw)?, &server)?; + Ok(JobResult { output, exit_code: None, @@ -1105,6 +1125,7 @@ fn cap_result(output: Value, tail_lines: Option) -> (Value, bool) { text = capped; truncated = true; } + if truncated { (json!(text), true) } else { @@ -1117,11 +1138,13 @@ fn tail_chars(text: &str, max_chars: usize) -> Option { if total <= max_chars { return None; } + let cut = text .char_indices() .nth(total - max_chars) .map(|(i, _)| i) .unwrap_or(0); + Some(format!( "[truncated: kept last {max_chars} of {total} chars]\n{}", &text[cut..] @@ -1133,11 +1156,12 @@ mod tests { use super::*; use crate::config::{AppConfig, AppState, WorkingMode}; use crate::function::supervisor::{ - GuardrailAction, check_pending_agents_guardrail, handle_supervisor_tool, + GuardrailAction, check_pending_tasks_guardrail, handle_supervisor_tool, }; use crate::supervisor::mailbox::Inbox; use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult}; use std::future::Future; + use std::mem; fn default_app_state() -> Arc { Arc::new(AppState::test_default()) @@ -1175,7 +1199,7 @@ mod tests { output_bytes_captured: 0, }) }); - std::mem::forget(rt); + mem::forget(rt); JobHandle { id: id.to_string(), tool: "execute_command".to_string(), @@ -1219,8 +1243,10 @@ mod tests { #[test] fn ring_buf_returns_contents_below_capacity() { let mut buf = RingBuf::new(8); + buf.push(b"abc"); buf.push(b"de"); + assert_eq!(buf.tail(), b"abcde"); assert_eq!(buf.total_written(), 5); } @@ -1228,7 +1254,9 @@ mod tests { #[test] fn ring_buf_exact_fit_keeps_everything() { let mut buf = RingBuf::new(5); + buf.push(b"abcde"); + assert_eq!(buf.tail(), b"abcde"); assert_eq!(buf.total_written(), 5); } @@ -1236,8 +1264,10 @@ mod tests { #[test] fn ring_buf_wrap_around_keeps_newest_bytes() { let mut buf = RingBuf::new(5); + buf.push(b"abcde"); buf.push(b"fg"); + assert_eq!(buf.tail(), b"cdefg"); assert_eq!(buf.total_written(), 7); } @@ -1245,7 +1275,9 @@ mod tests { #[test] fn ring_buf_oversize_push_keeps_last_capacity_bytes() { let mut buf = RingBuf::new(4); + buf.push(b"abcdefghij"); + assert_eq!(buf.tail(), b"ghij"); assert_eq!(buf.total_written(), 10); } @@ -1254,7 +1286,9 @@ mod tests { fn ring_buf_default_capacity_is_64_kib() { let mut buf = RingBuf::default(); let payload = vec![b'x'; 64 * 1024 + 1]; + buf.push(&payload); + assert_eq!(buf.tail().len(), 64 * 1024); assert_eq!(buf.total_written(), 64 * 1024 + 1); } @@ -1280,7 +1314,7 @@ mod tests { exit_status: AgentExitStatus::Completed, }) }); - std::mem::forget(rt); + mem::forget(rt); let handle = AgentHandle { id: "a1".to_string(), agent_name: "explore".to_string(), @@ -1304,6 +1338,7 @@ mod tests { .into_iter() .map(|d| d.name) .collect(); + assert_eq!( names, vec![ @@ -1320,6 +1355,7 @@ mod tests { fn whitelist_rejects_state_mutating_tools() { for tool in ["memory__write", "todo__add", "skill__load", "rag__query"] { let rejection = whitelist_rejection(tool).unwrap(); + let message = rejection["message"].as_str().unwrap(); assert!( message.contains("mutates agent/session state"), @@ -1335,6 +1371,7 @@ mod tests { .as_str() .unwrap() .to_string(); + assert!( message.contains("already asynchronous"), "unexpected message for {tool}: {message}" @@ -1359,6 +1396,7 @@ mod tests { .as_str() .unwrap() .to_string(); + assert!( message.contains("sub-second"), "unexpected message for {tool}: {message}" @@ -1373,6 +1411,7 @@ mod tests { .as_str() .unwrap() .to_string(); + assert!( message.contains("is fast"), "unexpected message for {tool}: {message}" @@ -1550,7 +1589,9 @@ mod tests { #[test] fn handle_check_unknown_id_teaches_job_list() { let ctx = ctx_with_job_supervisor(4); + let result = handle_check(&ctx, &json!({"id": "job_x"})).unwrap(); + assert_eq!(result["status"], "error"); assert!( result["message"] @@ -1708,7 +1749,9 @@ mod tests { #[test] fn job_handlers_miss_without_supervisor() { let ctx = plain_ctx(); + let result = handle_check(&ctx, &json!({"id": "job_x"})).unwrap(); + assert_eq!(result["status"], "error"); assert!( result["message"] @@ -2049,7 +2092,9 @@ mod tests { #[test] fn handle_list_without_supervisor_reports_empty() { let ctx = plain_ctx(); + let result = handle_list(&ctx).unwrap(); + assert_eq!(result["active_jobs"], 0); assert_eq!(result["max_concurrent_jobs"], 5); assert_eq!(result["jobs"].as_array().unwrap().len(), 0); @@ -2058,6 +2103,7 @@ mod tests { #[test] fn cap_result_normalizes_null_to_done() { let (value, truncated) = cap_result(Value::Null, None); + assert_eq!(value, json!("DONE")); assert!(!truncated); } @@ -2065,6 +2111,7 @@ mod tests { #[test] fn cap_result_preserves_small_values() { let (value, truncated) = cap_result(json!({"a": 1}), None); + assert_eq!(value, json!({"a": 1})); assert!(!truncated); } @@ -2085,6 +2132,7 @@ mod tests { #[test] fn tail_chars_floors_to_char_boundary() { let capped = tail_chars("aébc", 2).unwrap(); + assert!(capped.ends_with("bc")); assert!(capped.starts_with("[truncated: kept last 2 of 4 chars]")); assert!(tail_chars("abc", 3).is_none()); @@ -2321,18 +2369,18 @@ mod tests { .register(make_running_job("j1")) .unwrap(); - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(prompt.contains("j1")); assert!(prompt.contains("job__collect")); } _ => panic!("expected Inject for a running job"), } - assert_eq!(ctx.pending_agents_guardrail_count, 1); + assert_eq!(ctx.pending_tasks_guardrail_count, 1); let empty_ctx = &mut ctx_with_job_supervisor(5); assert!(matches!( - check_pending_agents_guardrail(empty_ctx), + check_pending_tasks_guardrail(empty_ctx), GuardrailAction::NoAction )); } diff --git a/src/function/mod.rs b/src/function/mod.rs index f05e9fc..eef4694 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -378,6 +378,7 @@ fn drain_live_notifications(ctx: &RequestContext) -> Vec { if events.is_empty() { return vec![]; } + let Some(supervisor) = ctx.supervisor.as_ref() else { return vec![]; }; @@ -399,6 +400,7 @@ fn merge_system_channel(last: &mut ToolResult, escalations: Vec, notifica if escalations.is_empty() && notifications.is_empty() { return; } + let escalation_instruction = "Child agents are BLOCKED waiting for your reply. \ Call agent__reply_escalation for each pending escalation to unblock them."; let notification_instruction = @@ -416,6 +418,7 @@ fn merge_system_channel(last: &mut ToolResult, escalations: Vec, notifica } } }; + if !escalations.is_empty() { map.insert("pending_escalations".into(), json!(escalations)); map.insert( @@ -423,6 +426,7 @@ fn merge_system_channel(last: &mut ToolResult, escalations: Vec, notifica json!(escalation_instruction), ); } + if !notifications.is_empty() { map.insert("system_notifications".into(), json!(notifications)); map.insert( @@ -468,6 +472,7 @@ impl ToolResult { while !s.is_char_boundary(cut) { cut -= 1; } + let prefix = &s[..cut]; self.output = json!(format!( "[truncated: tool output exceeded {max_chars} chars]\n{prefix}" @@ -2446,6 +2451,7 @@ impl ToolCallTracker { if is_loop_tracker_exempt(&new_call.name) { return None; } + if self.last_calls.len() < self.max_repeats { return None; } @@ -2515,6 +2521,7 @@ impl ToolCallTracker { if is_loop_tracker_exempt(&call.name) { return; } + if self.last_calls.len() >= self.chain_len * self.max_repeats { self.last_calls.pop_front(); } @@ -2572,14 +2579,20 @@ mod tests { }; use crate::config::{Agent, AgentConfig, AppConfig, AppState, WorkingMode}; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; + use crate::supervisor::mailbox::Inbox; use crate::supervisor::notification::{agent_notification, job_notification}; + use crate::supervisor::{ + AgentExitStatus, AgentHandle, AgentResult, JobHandle, JobResult, JobState, JobStatus, + Supervisor, + }; use base64::Engine; use base64::engine::general_purpose::STANDARD; + use jobs::RingBuf; use rmcp::model::{CallToolResult, ContentBlock}; use serde_json::json; use serial_test::serial; - use std::process; use std::sync::Arc; + use std::{mem, process}; fn call(name: &str, id: Option<&str>) -> ToolCall { ToolCall::new(name.to_string(), json!({}), id.map(|s| s.to_string())) @@ -2663,28 +2676,28 @@ mod tests { .build() .unwrap(); let join_handle = rt.spawn(async { - Ok(crate::supervisor::JobResult { + Ok(JobResult { output: Value::Null, exit_code: Some(0), output_bytes_captured: 0, }) }); - std::mem::forget(rt); - let handle = crate::supervisor::JobHandle { + mem::forget(rt); + let handle = JobHandle { id: id.to_string(), tool: "execute_command".to_string(), - started_at: std::time::Instant::now(), + started_at: Instant::now(), join_handle, - abort_signal: crate::utils::create_abort_signal(), - state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { - status: crate::supervisor::JobStatus::Completed, + abort_signal: create_abort_signal(), + state: Arc::new(parking_lot::Mutex::new(JobState { + status: JobStatus::Completed, pgid: None, })), - output_buf: Arc::new(parking_lot::Mutex::new(jobs::RingBuf::default())), + output_buf: Arc::new(parking_lot::Mutex::new(RingBuf::default())), no_change_checks: 0, last_check_state: None, }; - let mut sup = crate::supervisor::Supervisor::new(0, 3).with_max_concurrent_jobs(4); + let mut sup = Supervisor::new(0, 3).with_max_concurrent_jobs(4); sup.register(handle).unwrap(); let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); @@ -2724,7 +2737,9 @@ mod tests { #[test] fn merge_system_channel_adds_notifications_without_escalation_keys() { let mut result = ToolResult::new(call("t", Some("id-1")), json!({"status": "ok"})); + merge_system_channel(&mut result, vec![], vec![json!({"id": "job_1"})]); + assert_eq!(result.output["status"], "ok"); assert_eq!( result.output["system_notifications"], @@ -2743,11 +2758,13 @@ mod tests { #[test] fn merge_system_channel_wraps_non_object_once_with_both_channels() { let mut result = ToolResult::new(call("t", Some("id-1")), json!("DONE")); + merge_system_channel( &mut result, vec![json!({"escalation_id": "esc_1"})], vec![json!({"id": "job_1"})], ); + assert_eq!(result.output["output"], json!("DONE")); assert_eq!( result.output["pending_escalations"][0]["escalation_id"], @@ -2806,24 +2823,24 @@ mod tests { .unwrap(); let agent_id = id.to_string(); let join_handle = rt.spawn(async move { - Ok(crate::supervisor::AgentResult { + Ok(AgentResult { id: agent_id, agent_name: "explore".into(), output: String::new(), - exit_status: crate::supervisor::AgentExitStatus::Completed, + exit_status: AgentExitStatus::Completed, }) }); - std::mem::forget(rt); - let handle = crate::supervisor::AgentHandle { + mem::forget(rt); + let handle = AgentHandle { id: id.to_string(), agent_name: "explore".to_string(), depth: 1, - inbox: Arc::new(crate::supervisor::mailbox::Inbox::new()), - abort_signal: crate::utils::create_abort_signal(), + inbox: Arc::new(Inbox::new()), + abort_signal: create_abort_signal(), join_handle, child_supervisor: None, }; - let mut sup = crate::supervisor::Supervisor::new(4, 3); + let mut sup = Supervisor::new(4, 3); sup.register(handle).unwrap(); let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); @@ -2958,7 +2975,7 @@ mod tests { let job_id = started["job_id"].as_str().unwrap().to_string(); let supervisor = ctx.supervisor.clone().unwrap(); - let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(5); + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); while tokio::time::Instant::now() < deadline { let finished = supervisor .read() @@ -2967,7 +2984,7 @@ mod tests { if finished { break; } - tokio::time::sleep(std::time::Duration::from_millis(10)).await; + tokio::time::sleep(Duration::from_millis(10)).await; } let calls = vec![call("unknown_tool", Some("id-2"))]; @@ -3172,9 +3189,8 @@ mod tests { #[test] fn loop_tracker_exempt_list_is_exactly_the_polling_tools() { - let actual: std::collections::HashSet<&str> = - LOOP_TRACKER_EXEMPT_TOOLS.iter().copied().collect(); - let expected: std::collections::HashSet<&str> = [ + let actual: HashSet<&str> = LOOP_TRACKER_EXEMPT_TOOLS.iter().copied().collect(); + let expected: HashSet<&str> = [ "job__check", "job__list", "agent__check", @@ -3210,10 +3226,12 @@ mod tests { fn tracker_exempt_interleave_does_not_mask_real_loop() { let mut tracker = ToolCallTracker::default(); let x = call_with_args("execute_command", json!({"command": "ls"})); + tracker.record_call(call_with_args("job__check", json!({"id": "j1"}))); tracker.record_call(x.clone()); tracker.record_call(call_with_args("job__check", json!({"id": "j1"}))); tracker.record_call(x.clone()); + assert!(tracker.check_loop(&x).is_some()); } @@ -3221,8 +3239,10 @@ mod tests { fn tracker_non_exempt_behavior_unchanged() { let mut tracker = ToolCallTracker::default(); let c = call_with_args("fs_cat", json!({"path": "a.txt"})); + tracker.record_call(c.clone()); tracker.record_call(c.clone()); + assert!(tracker.check_loop(&c).is_some()); } @@ -3300,7 +3320,9 @@ mod tests { #[test] fn functions_append_job_adds_declarations() { let mut f = Functions::default(); + f.append_job_functions(); + assert!(f.contains("job__start")); assert!(f.contains("job__check")); assert!(f.contains("job__collect")); @@ -3313,7 +3335,9 @@ mod tests { let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); ctx.tool_scope.functions.append_job_functions(); let calls = vec![call("job__list", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + assert_eq!(results.len(), 1); assert_eq!(results[0].output["active_jobs"], 0); assert_eq!(results[0].output["jobs"], json!([])); @@ -3323,7 +3347,9 @@ mod tests { fn eval_soft_fails_job_calls_when_jobs_not_declared() { let mut ctx = RequestContext::new(Arc::new(AppState::test_default()), WorkingMode::Cmd); let calls = vec![call("job__start", Some("id-1"))]; + let results = run_async(eval_tool_calls(&mut ctx, calls)).unwrap(); + let err = results[0].output["tool_call_error"].as_str().unwrap(); assert!(err.contains("Unexpected call")); } diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index e276c47..e24a3b9 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -25,7 +25,7 @@ use uuid::Uuid; pub const SUPERVISOR_FUNCTION_PREFIX: &str = "agent__"; -pub const PENDING_AGENTS_GUARDRAIL_MAX: u32 = 3; +pub const PENDING_TASKS_GUARDRAIL_MAX: u32 = 3; fn agent_permitted(whitelist: Option<&[String]>, target: &str) -> bool { match whitelist { @@ -73,11 +73,12 @@ pub fn pending_tasks(ctx: &RequestContext) -> Vec { finished, }) .collect(); + tasks.sort_by(|a, b| a.id.cmp(&b.id)); tasks } -pub fn build_pending_agents_guardrail_prompt(tasks: &[PendingTask]) -> String { +pub fn build_pending_tasks_guardrail_prompt(tasks: &[PendingTask]) -> String { let running: Vec<&PendingTask> = tasks.iter().filter(|t| !t.finished).collect(); let finished: Vec<&PendingTask> = tasks.iter().filter(|t| t.finished).collect(); @@ -105,6 +106,7 @@ pub fn build_pending_agents_guardrail_prompt(tasks: &[PendingTask]) -> String { count = running.len() )); } + if !finished.is_empty() { let cmd_list = finished .iter() @@ -124,6 +126,7 @@ pub fn build_pending_agents_guardrail_prompt(tasks: &[PendingTask]) -> String { count = finished.len() )); } + format!( "[SYSTEM GUARDRAIL] You attempted to end your turn with {count} unreclaimed background \ task(s).\n\n{body}", @@ -132,14 +135,14 @@ pub fn build_pending_agents_guardrail_prompt(tasks: &[PendingTask]) -> String { ) } -pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailAction { +pub fn check_pending_tasks_guardrail(ctx: &mut RequestContext) -> GuardrailAction { let pending = pending_tasks(ctx); if pending.is_empty() { - ctx.pending_agents_guardrail_count = 0; + ctx.pending_tasks_guardrail_count = 0; return GuardrailAction::NoAction; } - if ctx.pending_agents_guardrail_count >= PENDING_AGENTS_GUARDRAIL_MAX { + if ctx.pending_tasks_guardrail_count >= PENDING_TASKS_GUARDRAIL_MAX { if let Some(sup) = ctx.supervisor.as_ref().cloned() { sup.read().cancel_recursive(); let finished: Vec<&PendingTask> = pending.iter().filter(|t| t.finished).collect(); @@ -162,13 +165,13 @@ pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailActi } } } - ctx.pending_agents_guardrail_count = 0; + ctx.pending_tasks_guardrail_count = 0; return GuardrailAction::ForceTerminate(pending.into_iter().map(|t| t.id).collect()); } - ctx.pending_agents_guardrail_count += 1; - let mut prompt = build_pending_agents_guardrail_prompt(&pending); + ctx.pending_tasks_guardrail_count += 1; + let mut prompt = build_pending_tasks_guardrail_prompt(&pending); if let Some(queue) = ctx.root_escalation_queue() && queue.has_pending() { @@ -184,7 +187,8 @@ pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailActi pub fn escalation_function_declarations() -> Vec { vec![FunctionDeclaration { name: format!("{SUPERVISOR_FUNCTION_PREFIX}reply_escalation"), - description: "Reply to a pending escalation from a child agent. The child is blocked waiting for this reply. Use this after seeing pending_escalations notifications.".to_string(), + description: "Reply to a pending escalation from a child agent. The child is blocked waiting for this reply. \ + Use this after seeing pending_escalations notifications.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([ @@ -200,7 +204,8 @@ pub fn escalation_function_declarations() -> Vec { "reply".to_string(), JsonSchema { type_value: Some("string".to_string()), - description: Some("Your answer to the child agent's question. For ask/confirm questions, use the exact option text. For input questions, provide the text response.".into()), + description: Some("Your answer to the child agent's question. For ask/confirm questions, use \ + the exact option text. For input questions, provide the text response.".into()), ..Default::default() }, ), @@ -256,7 +261,8 @@ pub fn supervisor_function_declarations() -> Vec { }, FunctionDeclaration { name: format!("{SUPERVISOR_FUNCTION_PREFIX}check"), - description: "Non-blocking status probe: reports whether a spawned agent is still running or finished. NEVER returns or consumes the result — when finished, call agent__collect to retrieve it.".to_string(), + description: "Non-blocking status probe: reports whether a spawned agent is still running or finished. \ + NEVER returns or consumes the result — when finished, call agent__collect to retrieve it.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([( @@ -558,10 +564,10 @@ pub fn run_child_agent( } if tool_results.is_empty() { - match check_pending_agents_guardrail(&mut child_ctx) { + match check_pending_tasks_guardrail(&mut child_ctx) { GuardrailAction::NoAction => break, GuardrailAction::ForceTerminate(ids) => { - log::warn!( + warn!( "Pending-agent guardrail force-cancelled {} agent(s) after max reminders: {:?}", ids.len(), ids @@ -642,7 +648,7 @@ pub async fn run_agent_for_graph( let session = agent.agent_session().map(|v| v.to_string()); let child_jobs_enabled = jobs_enabled(Some(&agent), app_config.as_ref()); let should_init_supervisor = agent.can_spawn_agents() || child_jobs_enabled; - let agent_max_concurrent = if agent.can_spawn_agents() { + let agent_max_concurrent_subagents = if agent.can_spawn_agents() { agent.max_concurrent_agents() } else { 0 @@ -661,7 +667,7 @@ pub async fn run_agent_for_graph( child_ctx.agent = Some(agent); if should_init_supervisor { child_ctx.supervisor = Some(Arc::new(RwLock::new( - Supervisor::new(agent_max_concurrent, agent_max_depth) + Supervisor::new(agent_max_concurrent_subagents, agent_max_depth) .with_max_concurrent_jobs(agent_max_jobs), ))); } @@ -827,7 +833,7 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { let session = agent.agent_session().map(|v| v.to_string()); let child_jobs_enabled = jobs_enabled(Some(&agent), app_config.as_ref()); let should_init_supervisor = agent.can_spawn_agents() || child_jobs_enabled; - let max_concurrent = if agent.can_spawn_agents() { + let max_concurrent_agents = if agent.can_spawn_agents() { agent.max_concurrent_agents() } else { 0 @@ -845,7 +851,7 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { child_ctx.agent = Some(agent); if should_init_supervisor { child_ctx.supervisor = Some(Arc::new(RwLock::new( - Supervisor::new(max_concurrent, max_depth).with_max_concurrent_jobs(max_jobs), + Supervisor::new(max_concurrent_agents, max_depth).with_max_concurrent_jobs(max_jobs), ))); } @@ -893,6 +899,7 @@ async fn handle_spawn(ctx: &mut RequestContext, args: &Value) -> Result { &agent_result.agent_name, success, )); + Ok(agent_result) }); @@ -975,6 +982,7 @@ async fn handle_check(ctx: &mut RequestContext, args: &Value) -> Result { if is_job_task(ctx.supervisor.as_ref(), id) { return Ok(job_id_teaching_error(id)); } + Ok(json!({ "status": "error", "message": format!("No agent found with id '{id}'") @@ -1001,6 +1009,7 @@ async fn handle_collect(ctx: &mut RequestContext, args: &Value) -> Result if id.starts_with("job_") || sup.has_job(id) { return Ok(job_id_teaching_error(id)); } + return Ok(json!({ "status": "error", "message": format!("Agent '{id}' not found. Use agent__check to verify it exists and is finished.") @@ -1068,7 +1077,7 @@ async fn handle_collect(ctx: &mut RequestContext, args: &Value) -> Result .map_err(|e| anyhow!("Agent failed: {e}"))?; let output = summarize_output(ctx, &result.agent_name, &result.output).await?; - ctx.pending_agents_guardrail_count = 0; + ctx.pending_tasks_guardrail_count = 0; Ok(json!({ "status": "completed", @@ -1169,7 +1178,7 @@ async fn handle_cancel(ctx: &mut RequestContext, args: &Value) -> Result let cleanup = tokio::time::timeout(Duration::from_secs(5), handle.join_handle).await; - ctx.pending_agents_guardrail_count = 0; + ctx.pending_tasks_guardrail_count = 0; let message = match cleanup { Ok(_) => format!("Cancelled agent '{agent_name}' and waited for cleanup."), @@ -1187,6 +1196,7 @@ async fn handle_cancel(ctx: &mut RequestContext, args: &Value) -> Result if is_job_task(ctx.supervisor.as_ref(), id) { return Ok(job_id_teaching_error(id)); } + Ok(json!({ "status": "error", "message": format!("No agent found with id '{id}'"), @@ -1244,6 +1254,7 @@ fn handle_send_message(ctx: &mut RequestContext, args: &Value) -> Result { return Ok(job_id_teaching_error(id)); } + Ok(json!({ "status": "error", "message": format!("No agent found with id '{id}'. Agent may not exist or may have already completed."), @@ -1591,6 +1602,7 @@ mod tests { use parking_lot::Mutex; use serde_json::json; use serial_test::serial; + use std::mem; fn default_app_state() -> Arc { Arc::new(AppState::test_default()) @@ -1622,7 +1634,7 @@ mod tests { output_bytes_captured: 0, }) }); - std::mem::forget(rt); + mem::forget(rt); JobHandle { id: id.to_string(), tool: "execute_command".to_string(), @@ -1681,7 +1693,7 @@ mod tests { exit_status: AgentExitStatus::Completed, }) }); - std::mem::forget(rt); + mem::forget(rt); let handle = AgentHandle { id: id.to_string(), @@ -2314,7 +2326,7 @@ mod tests { reply_tx: tx, }); - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(prompt.contains("agent__reply_escalation")); assert!(prompt.contains("esc_9")); @@ -2358,7 +2370,7 @@ mod tests { .register(handle) .unwrap(); - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(!prompt.contains("agent__reply_escalation")); } @@ -2371,7 +2383,7 @@ mod tests { fn handle_collect_finished_agent_returns_output_and_consumes_handle() { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); - ctx.pending_agents_guardrail_count = 2; + ctx.pending_tasks_guardrail_count = 2; let result = run_async(handle_collect(&mut ctx, &json!({"id": "a1"}))).unwrap(); @@ -2380,7 +2392,7 @@ mod tests { assert_eq!(result["agent"], "explore"); assert_eq!(result["exit_status"], "Completed"); assert_eq!(result["output"], "fake output"); - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); assert_eq!( ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), None @@ -2424,7 +2436,9 @@ mod tests { #[test] fn handle_collect_unknown_agent_errors() { let mut ctx = ctx_with_supervisor(4, 3); + let result = run_async(handle_collect(&mut ctx, &json!({"id": "missing"}))).unwrap(); + assert_eq!(result["status"], "error"); assert!(result["message"].as_str().unwrap().contains("not found")); } @@ -2473,13 +2487,13 @@ mod tests { #[test] fn guardrail_no_supervisor_is_no_action_and_resets_counter() { let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd); - ctx.pending_agents_guardrail_count = 2; + ctx.pending_tasks_guardrail_count = 2; assert!(matches!( - check_pending_agents_guardrail(&mut ctx), + check_pending_tasks_guardrail(&mut ctx), GuardrailAction::NoAction )); - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); } /// A finished-but-uncollected agent counts as pending: the turn-end @@ -2490,9 +2504,9 @@ mod tests { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); wait_until_finished(&ctx, "a1"); - ctx.pending_agents_guardrail_count = 2; + ctx.pending_tasks_guardrail_count = 2; - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(prompt.contains("a1")); assert!(prompt.contains("agent__collect --id a1")); @@ -2500,7 +2514,7 @@ mod tests { } _ => panic!("expected Inject action"), } - assert_eq!(ctx.pending_agents_guardrail_count, 3); + assert_eq!(ctx.pending_tasks_guardrail_count, 3); assert_eq!( ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), Some(true) @@ -2512,15 +2526,15 @@ mod tests { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); wait_until_finished(&ctx, "a1"); - ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + ctx.pending_tasks_guardrail_count = PENDING_TASKS_GUARDRAIL_MAX; - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::ForceTerminate(ids) => { assert_eq!(ids, vec!["a1".to_string()]); } _ => panic!("expected ForceTerminate action"), } - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); assert_eq!( ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), None @@ -2540,7 +2554,7 @@ mod tests { register_fake_agent(&mut ctx, "a1", "explore"); wait_until_finished(&ctx, "a1"); - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(prompt.contains("Still running")); assert!(prompt.contains("slow (agent)")); @@ -2567,7 +2581,7 @@ mod tests { }, ]; - let prompt = build_pending_agents_guardrail_prompt(&tasks); + let prompt = build_pending_tasks_guardrail_prompt(&tasks); assert!(prompt.contains("job_1 (job)")); assert!(prompt.contains("job__cancel")); @@ -2596,15 +2610,15 @@ mod tests { rt.block_on(async { let mut ctx = ctx_with_supervisor(4, 3); let abort = register_running_agent(&mut ctx, "slow", "test"); - ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + ctx.pending_tasks_guardrail_count = PENDING_TASKS_GUARDRAIL_MAX; - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::ForceTerminate(ids) => { assert_eq!(ids, vec!["slow".to_string()]); } _ => panic!("expected ForceTerminate action"), } - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); assert!(abort.aborted()); }); } @@ -2619,16 +2633,16 @@ mod tests { rt.block_on(async { let mut ctx = ctx_with_supervisor(4, 3); let _abort = register_running_agent(&mut ctx, "slow", "test"); - ctx.pending_agents_guardrail_count = 1; + ctx.pending_tasks_guardrail_count = 1; - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => { assert!(prompt.contains("slow")); assert!(prompt.contains("agent__collect")); } _ => panic!("expected Inject action"), } - assert_eq!(ctx.pending_agents_guardrail_count, 2); + assert_eq!(ctx.pending_tasks_guardrail_count, 2); }); } @@ -2636,12 +2650,12 @@ mod tests { fn handle_cancel_resets_guardrail_counter() { let mut ctx = ctx_with_supervisor(4, 3); register_fake_agent(&mut ctx, "a1", "explore"); - ctx.pending_agents_guardrail_count = 2; + ctx.pending_tasks_guardrail_count = 2; let result = run_async(handle_cancel(&mut ctx, &json!({"id": "a1"}))).unwrap(); assert_eq!(result["status"], "ok"); - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); } #[test] @@ -2798,7 +2812,7 @@ mod tests { .write() .register(handle) .unwrap(); - ctx.pending_agents_guardrail_count = 2; + ctx.pending_tasks_guardrail_count = 2; let result = handle_cancel(&mut ctx, &json!({"id": "a1"})).await.unwrap(); @@ -2811,7 +2825,7 @@ mod tests { ctx.supervisor.as_ref().unwrap().read().is_finished("a1"), None ); - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); }); } @@ -2951,21 +2965,21 @@ mod tests { .register(handle) .unwrap(); - for expected_count in 1..=PENDING_AGENTS_GUARDRAIL_MAX { - match check_pending_agents_guardrail(&mut ctx) { + for expected_count in 1..=PENDING_TASKS_GUARDRAIL_MAX { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::Inject(prompt) => assert!(prompt.contains("job_1")), _ => panic!("expected Inject below max"), } - assert_eq!(ctx.pending_agents_guardrail_count, expected_count); + assert_eq!(ctx.pending_tasks_guardrail_count, expected_count); } - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::ForceTerminate(ids) => { assert_eq!(ids, vec!["job_1".to_string()]); } _ => panic!("expected ForceTerminate at max"), } - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); assert!(abort.aborted()); }); } @@ -2982,15 +2996,15 @@ mod tests { ); std::thread::sleep(Duration::from_millis(10)); } - ctx.pending_agents_guardrail_count = PENDING_AGENTS_GUARDRAIL_MAX; + ctx.pending_tasks_guardrail_count = PENDING_TASKS_GUARDRAIL_MAX; - match check_pending_agents_guardrail(&mut ctx) { + match check_pending_tasks_guardrail(&mut ctx) { GuardrailAction::ForceTerminate(ids) => { assert_eq!(ids, vec!["job_1".to_string()]); } _ => panic!("expected ForceTerminate action"), } - assert_eq!(ctx.pending_agents_guardrail_count, 0); + assert_eq!(ctx.pending_tasks_guardrail_count, 0); assert!(!ctx.supervisor.as_ref().unwrap().read().has_job("job_1")); } } diff --git a/src/graph/executor.rs b/src/graph/executor.rs index 70ec480..d7fe458 100644 --- a/src/graph/executor.rs +++ b/src/graph/executor.rs @@ -563,8 +563,10 @@ mod tests { mod integration_tests { use super::*; use crate::config::{AppState, WorkingMode}; + use crate::function::jobs::RingBuf; + use crate::supervisor::{JobHandle, JobResult, JobState, JobStatus, Supervisor, notification}; use crate::utils::{create_abort_signal, temp_file}; - use std::fs; + use std::{fs, mem}; fn cmd_available(name: &str) -> bool { which::which(name).is_ok() @@ -887,40 +889,37 @@ nodes: .build() .unwrap(); let join_handle = rt.spawn(async { - Ok(crate::supervisor::JobResult { + Ok(JobResult { output: Value::Null, exit_code: Some(0), output_bytes_captured: 0, }) }); - std::mem::forget(rt); - let handle = crate::supervisor::JobHandle { + mem::forget(rt); + let handle = JobHandle { id: "job_bg".to_string(), tool: "execute_command".to_string(), started_at: Instant::now(), join_handle, abort_signal: create_abort_signal(), - state: Arc::new(parking_lot::Mutex::new(crate::supervisor::JobState { - status: crate::supervisor::JobStatus::Completed, + state: Arc::new(parking_lot::Mutex::new(JobState { + status: JobStatus::Completed, pgid: None, })), - output_buf: Arc::new(parking_lot::Mutex::new( - crate::function::jobs::RingBuf::default(), - )), + output_buf: Arc::new(parking_lot::Mutex::new(RingBuf::default())), no_change_checks: 0, last_check_state: None, }; - let mut sup = crate::supervisor::Supervisor::new(0, 3).with_max_concurrent_jobs(4); + let mut sup = Supervisor::new(0, 3).with_max_concurrent_jobs(4); sup.register(handle).unwrap(); let mut ctx = make_ctx(); ctx.supervisor = Some(Arc::new(parking_lot::RwLock::new(sup))); - ctx.notification_queue - .push(crate::supervisor::notification::job_notification( - "job_bg", - "execute_command", - true, - )); + ctx.notification_queue.push(notification::job_notification( + "job_bg", + "execute_command", + true, + )); let abort = create_abort_signal(); let result = GraphExecutor::new(graph, &ws.dir) diff --git a/src/graph/llm.rs b/src/graph/llm.rs index b62f4a2..7e19a19 100644 --- a/src/graph/llm.rs +++ b/src/graph/llm.rs @@ -7,7 +7,7 @@ use crate::config::{ Input, RequestContext, Role, RoleLike, SkillPolicy, should_inject_skill_instructions, }; use crate::function::skill::skill_function_declarations; -use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail}; +use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::utils::create_abort_signal; use anyhow::{Context, Error, Result, anyhow, bail}; use log::warn; @@ -268,7 +268,7 @@ async fn run_chat_loop(node: &LlmNode, prompt: &str, ctx: &mut RequestContext) - } if tool_results.is_empty() { - match check_pending_agents_guardrail(ctx) { + match check_pending_tasks_guardrail(ctx) { GuardrailAction::NoAction => return Ok(accumulated), GuardrailAction::ForceTerminate(ids) => { warn!( diff --git a/src/main.rs b/src/main.rs index 3e5d9b0..0dcb9ae 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,7 +29,7 @@ use crate::config::{ install_builtins, list_agents, load_env_file, macro_execute, sync_models, }; use crate::config::{memory, paths}; -use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail}; +use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::mcp::McpServersConfig; use crate::render::{prompt_theme, render_error}; use crate::repl::Repl; @@ -595,7 +595,7 @@ async fn start_directive( ) .await?; } else { - match check_pending_agents_guardrail(ctx) { + match check_pending_tasks_guardrail(ctx) { GuardrailAction::Inject(prompt) => { let guardrail_input = Input::from_str(ctx, &prompt, None)?; return start_directive(ctx, guardrail_input, code_mode, abort_signal).await; diff --git a/src/repl/mod.rs b/src/repl/mod.rs index f3b0d0f..be3b742 100644 --- a/src/repl/mod.rs +++ b/src/repl/mod.rs @@ -16,7 +16,7 @@ use crate::config::{ StateFlags, flatten_prompt_messages, macro_execute, resolve_prompt_args, sanitize_display_text, }; use crate::config::{AssetCategory, paths}; -use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail}; +use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::render::render_error; use crate::utils::{ AbortSignal, SHELL, abortable_run_with_spinner, create_abort_signal, dimmed_text, @@ -602,7 +602,7 @@ pub async fn run_repl_command( abort_signal: AbortSignal, mut line: &str, ) -> Result { - ctx.pending_agents_guardrail_count = 0; + ctx.pending_tasks_guardrail_count = 0; if let Ok(Some(captures)) = MULTILINE_RE.captures(line) && let Some(text_match) = captures.get(1) { @@ -1475,7 +1475,7 @@ async fn ask( ) .await } else { - match check_pending_agents_guardrail(ctx) { + match check_pending_tasks_guardrail(ctx) { GuardrailAction::Inject(prompt) => { let guardrail_input = Input::from_str(ctx, &prompt, None)?; return ask(ctx, abort_signal, guardrail_input, false).await; diff --git a/src/supervisor/mod.rs b/src/supervisor/mod.rs index ef8e2fc..d3e0487 100644 --- a/src/supervisor/mod.rs +++ b/src/supervisor/mod.rs @@ -352,6 +352,7 @@ mod tests { use super::*; use crate::utils::create_abort_signal; use anyhow::Error; + use std::mem; use tokio::runtime::Builder; fn make_handle(id: &str, agent_name: &str, depth: usize) -> AgentHandle { @@ -386,7 +387,7 @@ mod tests { output_bytes_captured: 0, }) }); - std::mem::forget(rt); + mem::forget(rt); JobHandle { id: id.to_string(), tool: "execute_command".to_string(), @@ -545,7 +546,9 @@ mod tests { #[test] fn job_registration_rejects_when_job_capacity_zero() { let mut sup = Supervisor::new(4, 3); + let result = sup.register(make_job("j1", create_abort_signal())); + assert!(result.is_err()); assert!(result.unwrap_err().to_string().contains("at capacity")); } @@ -554,7 +557,9 @@ mod tests { fn job_registration_rejects_at_job_capacity() { let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(1); sup.register(make_job("j1", create_abort_signal())).unwrap(); + let result = sup.register(make_job("j2", create_abort_signal())); + assert!(result.is_err()); assert!(result.unwrap_err().to_string().contains("at capacity")); } @@ -562,8 +567,10 @@ mod tests { #[test] fn job_capacity_is_independent_of_agent_capacity() { let mut sup = Supervisor::new(1, 3).with_max_concurrent_jobs(1); + sup.register(make_job("j1", create_abort_signal())).unwrap(); sup.register(make_handle("a1", "explore", 1)).unwrap(); + assert_eq!(sup.active_job_count(), 1); assert_eq!(sup.active_count(), 1); assert_eq!(sup.max_concurrent_jobs(), 1); @@ -572,6 +579,7 @@ mod tests { #[test] fn agent_accessors_ignore_jobs() { let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(2); + sup.register(make_job("j1", create_abort_signal())).unwrap(); assert_eq!(sup.active_count(), 0); @@ -589,6 +597,7 @@ mod tests { #[test] fn take_job_removes_job_but_not_agents() { let mut sup = Supervisor::new(4, 3).with_max_concurrent_jobs(2); + sup.register(make_job("j1", create_abort_signal())).unwrap(); sup.register(make_handle("a1", "explore", 1)).unwrap(); From c376737bbdde3eeb98735f8ab5bd9ca2780b1c33 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 12:53:17 -0600 Subject: [PATCH 20/28] refactor(function): rename agent-tool symbols out of supervisor vocabulary Since the supervisor registry became kind-generic (agents AND jobs), 'supervisor' naming on the agent__* tool plumbing was misleading: job__* handlers operate on the same supervisor. Rename SUPERVISOR_FUNCTION_PREFIX -> AGENT_FUNCTION_PREFIX, supervisor_function_declarations -> agent_function_declarations, handle_supervisor_tool -> handle_agent_tool. No behavior change. --- src/config/request_context.rs | 4 +-- src/function/jobs.rs | 14 +++++----- src/function/mod.rs | 10 ++++---- src/function/supervisor.rs | 48 ++++++++++++++++------------------- 4 files changed, 36 insertions(+), 40 deletions(-) diff --git a/src/config/request_context.rs b/src/config/request_context.rs index 138b4ac..fd00b11 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -22,7 +22,7 @@ use crate::function::{ memory::MEMORY_FUNCTION_PREFIX, rag_query::RAG_FUNCTION_PREFIX, skill::SKILL_FUNCTION_PREFIX, - supervisor::SUPERVISOR_FUNCTION_PREFIX, + supervisor::AGENT_FUNCTION_PREFIX, todo::TODO_FUNCTION_PREFIX, user_interaction::USER_FUNCTION_PREFIX, }; @@ -2181,7 +2181,7 @@ impl RequestContext { && v.name.starts_with(SKILL_FUNCTION_PREFIX)) || v.name.starts_with(USER_FUNCTION_PREFIX) || v.name.starts_with(TODO_FUNCTION_PREFIX) - || v.name.starts_with(SUPERVISOR_FUNCTION_PREFIX) + || v.name.starts_with(AGENT_FUNCTION_PREFIX) || v.name.starts_with(MEMORY_FUNCTION_PREFIX) || v.name.starts_with(RAG_FUNCTION_PREFIX) || v.name.starts_with(JOB_FUNCTION_PREFIX) diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 0f4fd7d..59b52ca 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -1,7 +1,7 @@ use super::memory::MEMORY_FUNCTION_PREFIX; use super::rag_query::RAG_FUNCTION_PREFIX; use super::skill::SKILL_FUNCTION_PREFIX; -use super::supervisor::SUPERVISOR_FUNCTION_PREFIX; +use super::supervisor::AGENT_FUNCTION_PREFIX; use super::todo::TODO_FUNCTION_PREFIX; use super::user_interaction::USER_FUNCTION_PREFIX; use super::{FunctionDeclaration, JsonSchema, PATH_SEP, mcp_error_display, render_tool_result}; @@ -303,7 +303,7 @@ fn whitelist_rejection(tool: &str) -> Option { MCP_READ_META_FUNCTION_NAME_PREFIX, MCP_PROMPT_META_FUNCTION_NAME_PREFIX, ]; - let reason = if tool.starts_with(SUPERVISOR_FUNCTION_PREFIX) + let reason = if tool.starts_with(AGENT_FUNCTION_PREFIX) || tool.starts_with(JOB_FUNCTION_PREFIX) { Some(format!( @@ -1156,7 +1156,7 @@ mod tests { use super::*; use crate::config::{AppConfig, AppState, WorkingMode}; use crate::function::supervisor::{ - GuardrailAction, check_pending_tasks_guardrail, handle_supervisor_tool, + GuardrailAction, check_pending_tasks_guardrail, handle_agent_tool, }; use crate::supervisor::mailbox::Inbox; use crate::supervisor::{AgentExitStatus, AgentHandle, AgentResult}; @@ -2343,7 +2343,7 @@ mod tests { fn jobs_only_supervisor_rejects_agent_spawn_at_capacity_zero() { let mut ctx = ctx_with_job_supervisor(5); - let result = run_async(handle_supervisor_tool( + let result = run_async(handle_agent_tool( &mut ctx, "agent__spawn", &json!({"agent": "explore", "prompt": "x"}), @@ -2389,7 +2389,7 @@ mod tests { fn jobs_only_supervisor_agent_surfaces_stay_functional() { let mut ctx = ctx_with_job_supervisor(5); - let listed = run_async(handle_supervisor_tool( + let listed = run_async(handle_agent_tool( &mut ctx, "agent__list_running", &json!({}), @@ -2398,7 +2398,7 @@ mod tests { assert_eq!(listed["active_count"], 0); assert_eq!(listed["max_concurrent"], 0); - let created = run_async(handle_supervisor_tool( + let created = run_async(handle_agent_tool( &mut ctx, "agent__task_create", &json!({"subject": "research"}), @@ -2406,7 +2406,7 @@ mod tests { .unwrap(); assert_eq!(created["status"], "ok"); - let tasks = run_async(handle_supervisor_tool( + let tasks = run_async(handle_agent_tool( &mut ctx, "agent__task_list", &json!({}), diff --git a/src/function/mod.rs b/src/function/mod.rs index eef4694..3969aaf 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -48,7 +48,7 @@ use std::{ time::{Duration, Instant}, }; use strum_macros::AsRefStr; -use supervisor::SUPERVISOR_FUNCTION_PREFIX; +use supervisor::AGENT_FUNCTION_PREFIX; use todo::TODO_FUNCTION_PREFIX; use user_interaction::USER_FUNCTION_PREFIX; @@ -684,7 +684,7 @@ impl Functions { pub fn append_supervisor_functions(&mut self) { self.declarations - .extend(supervisor::supervisor_function_declarations()); + .extend(supervisor::agent_function_declarations()); self.declarations .extend(supervisor::escalation_function_declarations()); } @@ -1589,8 +1589,8 @@ impl ToolCall { json!({"tool_call_error": error_msg}) }) } - _ if cmd_name.starts_with(SUPERVISOR_FUNCTION_PREFIX) => { - supervisor::handle_supervisor_tool(ctx, &cmd_name, &json_data) + _ if cmd_name.starts_with(AGENT_FUNCTION_PREFIX) => { + supervisor::handle_agent_tool(ctx, &cmd_name, &json_data) .await .unwrap_or_else(|e| { let error_msg = format!("Supervisor tool failed: {e}"); @@ -3249,7 +3249,7 @@ mod tests { #[test] fn prefix_constants_are_correct() { assert_eq!(TODO_FUNCTION_PREFIX, "todo__"); - assert_eq!(SUPERVISOR_FUNCTION_PREFIX, "agent__"); + assert_eq!(AGENT_FUNCTION_PREFIX, "agent__"); assert_eq!(USER_FUNCTION_PREFIX, "user__"); assert_eq!(MCP_INVOKE_META_FUNCTION_NAME_PREFIX, "mcp_invoke"); assert_eq!(MCP_SEARCH_META_FUNCTION_NAME_PREFIX, "mcp_search"); diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index e24a3b9..5e05d2c 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -23,7 +23,7 @@ use tokio::time; use tokio::time::Instant; use uuid::Uuid; -pub const SUPERVISOR_FUNCTION_PREFIX: &str = "agent__"; +pub const AGENT_FUNCTION_PREFIX: &str = "agent__"; pub const PENDING_TASKS_GUARDRAIL_MAX: u32 = 3; @@ -186,7 +186,7 @@ pub fn check_pending_tasks_guardrail(ctx: &mut RequestContext) -> GuardrailActio pub fn escalation_function_declarations() -> Vec { vec![FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}reply_escalation"), + name: format!("{AGENT_FUNCTION_PREFIX}reply_escalation"), description: "Reply to a pending escalation from a child agent. The child is blocked waiting for this reply. \ Use this after seeing pending_escalations notifications.".to_string(), parameters: JsonSchema { @@ -217,10 +217,10 @@ pub fn escalation_function_declarations() -> Vec { }] } -pub fn supervisor_function_declarations() -> Vec { +pub fn agent_function_declarations() -> Vec { vec![ FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}spawn"), + name: format!("{AGENT_FUNCTION_PREFIX}spawn"), description: "Spawn a subagent to run in the background. Returns an `id` immediately so you can continue \ working in parallel. CRITICAL: every spawned agent MUST be reclaimed before you end your \ turn — call `agent__collect` to retrieve its output, or `agent__cancel` if you no longer \ @@ -260,7 +260,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}check"), + name: format!("{AGENT_FUNCTION_PREFIX}check"), description: "Non-blocking status probe: reports whether a spawned agent is still running or finished. \ NEVER returns or consumes the result — when finished, call agent__collect to retrieve it.".to_string(), parameters: JsonSchema { @@ -279,7 +279,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}collect"), + name: format!("{AGENT_FUNCTION_PREFIX}collect"), description: "Block until the named spawned agent finishes and return its result. This is your primary \ wait primitive — it pauses your execution until the agent completes (or you are interrupted). \ Call this for every agent you spawned before ending your turn. Do NOT end your turn assuming \ @@ -301,7 +301,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}list_running"), + name: format!("{AGENT_FUNCTION_PREFIX}list_running"), description: "List all subagents YOU have spawned that are still tracked by the supervisor, with their \ status. Use this to see which of your background agents are still active. To discover which \ agent types you can spawn in the first place, use `agent__list_available` instead.".to_string(), @@ -313,7 +313,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}list_available"), + name: format!("{AGENT_FUNCTION_PREFIX}list_available"), description: "List all agent types installed and available to spawn (name + description). Use this to \ discover what specialists exist before calling `agent__spawn` — especially when you're unsure \ which agent to delegate to. This is the discovery counterpart to `agent__list_running` \ @@ -326,7 +326,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}cancel"), + name: format!("{AGENT_FUNCTION_PREFIX}cancel"), description: "Cancel a running subagent by its ID. Use this when an agent's output is no longer needed \ (e.g. you changed direction, or you're about to end your turn and don't want to wait). \ Cancellation cascades: all of the cancelled agent's own descendants are also cancelled. This \ @@ -347,7 +347,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}task_create"), + name: format!("{AGENT_FUNCTION_PREFIX}task_create"), description: "Create a task in the task queue. Returns the task ID.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -403,7 +403,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}task_list"), + name: format!("{AGENT_FUNCTION_PREFIX}task_list"), description: "List all tasks in the task queue with their status and dependencies.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -413,7 +413,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}task_complete"), + name: format!("{AGENT_FUNCTION_PREFIX}task_complete"), description: "Mark a task as completed. Returns any newly unblocked task IDs.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -431,7 +431,7 @@ pub fn supervisor_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}task_fail"), + name: format!("{AGENT_FUNCTION_PREFIX}task_fail"), description: "Mark a task as failed. Dependents will remain blocked.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -454,7 +454,7 @@ pub fn supervisor_function_declarations() -> Vec { pub fn teammate_function_declarations() -> Vec { vec![ FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}send_message"), + name: format!("{AGENT_FUNCTION_PREFIX}send_message"), description: "Send a text message to a sibling or child agent's inbox. Use to share cross-cutting findings or coordinate with teammates.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -482,7 +482,7 @@ pub fn teammate_function_declarations() -> Vec { agent: false, }, FunctionDeclaration { - name: format!("{SUPERVISOR_FUNCTION_PREFIX}check_inbox"), + name: format!("{AGENT_FUNCTION_PREFIX}check_inbox"), description: "Check for and drain all pending messages in your inbox from sibling agents or your parent.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), @@ -494,13 +494,13 @@ pub fn teammate_function_declarations() -> Vec { ] } -pub async fn handle_supervisor_tool( +pub async fn handle_agent_tool( ctx: &mut RequestContext, cmd_name: &str, args: &Value, ) -> Result { let action = cmd_name - .strip_prefix(SUPERVISOR_FUNCTION_PREFIX) + .strip_prefix(AGENT_FUNCTION_PREFIX) .unwrap_or(cmd_name); match action { @@ -2137,7 +2137,7 @@ mod tests { #[test] fn dispatch_unknown_action_errors() { let mut ctx = ctx_with_supervisor(4, 3); - let result = run_async(handle_supervisor_tool(&mut ctx, "agent__bogus", &json!({}))); + let result = run_async(handle_agent_tool(&mut ctx, "agent__bogus", &json!({}))); assert!(result.is_err()); assert!( result @@ -2150,7 +2150,7 @@ mod tests { #[test] fn dispatch_routes_list_running() { let mut ctx = ctx_with_supervisor(4, 3); - let result = run_async(handle_supervisor_tool( + let result = run_async(handle_agent_tool( &mut ctx, "agent__list_running", &json!({}), @@ -2162,7 +2162,7 @@ mod tests { #[test] fn dispatch_routes_list_available() { let mut ctx = ctx_with_supervisor(4, 3); - let result = run_async(handle_supervisor_tool( + let result = run_async(handle_agent_tool( &mut ctx, "agent__list_available", &json!({}), @@ -2175,12 +2175,8 @@ mod tests { #[test] fn dispatch_routes_task_list() { let mut ctx = ctx_with_supervisor(4, 3); - let result = run_async(handle_supervisor_tool( - &mut ctx, - "agent__task_list", - &json!({}), - )) - .unwrap(); + let result = + run_async(handle_agent_tool(&mut ctx, "agent__task_list", &json!({}))).unwrap(); assert!(result["tasks"].is_array()); } From 074083af31b973de9732b9cfb840c35f557bf1c5 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 12:53:17 -0600 Subject: [PATCH 21/28] feat(jobs): allow uncapped collect via full_result MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit job__collect's 50k-char tail cap is a safety default, but collect is consume-once and the cap was mandatory — a model that genuinely needed the complete output had no recourse. Add a full_result boolean that skips the cap (tail_lines still honored; the session-wide max_tool_result_chars limit still applies downstream), teach the truncation banner to name the recourse, and point job__check's output_bytes_captured at the collect decision. --- src/config/prompts.rs | 4 +- src/function/jobs.rs | 117 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 100 insertions(+), 21 deletions(-) diff --git a/src/config/prompts.rs b/src/config/prompts.rs index 58ac50e..beb4485 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -198,7 +198,9 @@ pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" working instead of blocking — completion arrives as a `system_notifications` entry on your next tool result. Check progress with `job__check` (sparingly), block on the result with `job__collect` (only when you have nothing else to do), cancel with `job__cancel`, and list - jobs with `job__list`. Collect or cancel every job you started before ending your turn. In + jobs with `job__list`. Collected results over 50,000 chars are tail-capped; collecting is + consume-once, so when you need the complete output pass `full_result: true` (or have the + command write to a file). Collect or cancel every job you started before ending your turn. In graph LLM nodes, collect or cancel your jobs before ending your final node turn — an uncollected job at node turn-end burns node iterations via the guardrail and can fail the node. Jobs run against a snapshot of the current config/environment and do not survive diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 59b52ca..0b5fba4 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -171,7 +171,9 @@ pub fn job_function_declarations() -> Vec { description: "Non-blocking status probe for a background job. Returns status, elapsed time, and a tail \ of the output captured so far; it NEVER consumes the result — use `job__collect` for \ that. Call sparingly: if repeated checks show no change, do other work instead — you \ - will be notified when the job completes.".to_string(), + will be notified when the job completes. `output_bytes_captured` reports the total \ + output size so far — use it to decide how to collect (`tail_lines`, `full_result`, or \ + having the command write to a file).".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([( @@ -191,7 +193,11 @@ pub fn job_function_declarations() -> Vec { name: format!("{JOB_FUNCTION_PREFIX}collect"), description: "Block until the named background job finishes, then return its result and remove the \ job. The result keeps the LAST 50,000 chars by default (failures land at the tail of \ - build logs); pass `tail_lines` to keep only the last N lines instead.".to_string(), + build logs); pass `tail_lines` to keep only the last N lines instead, or \ + `full_result: true` to skip the cap entirely (the session-wide tool-output limit still \ + applies). Collecting is consume-once — decide first via `job__check`'s \ + `output_bytes_captured`. For very large outputs, prefer having the command write to a \ + file and paging it with `fs_read`.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([ @@ -211,6 +217,14 @@ pub fn job_function_declarations() -> Vec { ..Default::default() }, ), + ( + "full_result".to_string(), + JsonSchema { + type_value: Some("boolean".to_string()), + description: Some("Return the complete result, skipping the default 50,000-char tail cap (default: false)".into()), + ..Default::default() + }, + ), ])), required: Some(vec!["id".to_string()]), ..Default::default() @@ -303,8 +317,7 @@ fn whitelist_rejection(tool: &str) -> Option { MCP_READ_META_FUNCTION_NAME_PREFIX, MCP_PROMPT_META_FUNCTION_NAME_PREFIX, ]; - let reason = if tool.starts_with(AGENT_FUNCTION_PREFIX) - || tool.starts_with(JOB_FUNCTION_PREFIX) + let reason = if tool.starts_with(AGENT_FUNCTION_PREFIX) || tool.starts_with(JOB_FUNCTION_PREFIX) { Some(format!( "'{tool}' is already asynchronous — call it directly. Agents may start jobs, but jobs never start agents or other jobs." @@ -575,6 +588,10 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { .get("tail_lines") .and_then(Value::as_u64) .map(|n| n as usize); + let full_result = args + .get("full_result") + .and_then(Value::as_bool) + .unwrap_or(false); let Some(supervisor) = ctx.supervisor.as_ref().cloned() else { return Ok(job_miss_error(None, id)); @@ -700,7 +717,7 @@ async fn handle_collect(ctx: &RequestContext, args: &Value) -> Result { Ok(Ok(job_result)) => job_result, }; - let (result_value, result_truncated) = cap_result(job_result.output, tail_lines); + let (result_value, result_truncated) = cap_result(job_result.output, tail_lines, full_result); let mut response = json!({ "status": job_status_str(status), "id": id, @@ -1104,8 +1121,10 @@ async fn run_mcp_job( /// Tail-biased result capping owned by the collect handler: keeps the LAST /// `tail_lines`/50,000 chars (build failures land at the tail), always cutting -/// on a char boundary. -fn cap_result(output: Value, tail_lines: Option) -> (Value, bool) { +/// on a char boundary. `full_result` lifts the 50,000-char ceiling (the +/// session-wide tool-output limit still applies downstream); `tail_lines` is +/// honored either way. +fn cap_result(output: Value, tail_lines: Option, full_result: bool) -> (Value, bool) { if output.is_null() { return (json!("DONE"), false); } @@ -1121,7 +1140,7 @@ fn cap_result(output: Value, tail_lines: Option) -> (Value, bool) { truncated = true; } } - if let Some(capped) = tail_chars(&text, JOB_RESULT_TAIL_CAP_CHARS) { + if !full_result && let Some(capped) = tail_chars(&text, JOB_RESULT_TAIL_CAP_CHARS) { text = capped; truncated = true; } @@ -1146,7 +1165,8 @@ fn tail_chars(text: &str, max_chars: usize) -> Option { .unwrap_or(0); Some(format!( - "[truncated: kept last {max_chars} of {total} chars]\n{}", + "[truncated: kept last {max_chars} of {total} chars — the rest was not retained; next time \ + collect with full_result: true, or have the command write its output to a file]\n{}", &text[cut..] )) } @@ -1934,6 +1954,50 @@ mod tests { }); } + #[test] + fn handle_collect_full_result_returns_uncapped() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + let big = "x".repeat(JOB_RESULT_TAIL_CAP_CHARS + 10); + let payload = big.clone(); + let join_handle = tokio::spawn(async move { + Ok(JobResult { + output: json!(payload), + exit_code: Some(0), + output_bytes_captured: 0, + }) + }); + let handle = JobHandle { + id: "j1".to_string(), + tool: "execute_command".to_string(), + started_at: Instant::now(), + join_handle, + abort_signal: create_abort_signal(), + state: Arc::new(Mutex::new(JobState { + status: JobStatus::Completed, + pgid: None, + })), + output_buf: Arc::new(Mutex::new(RingBuf::default())), + no_change_checks: 0, + last_check_state: None, + }; + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(handle) + .unwrap(); + + let result = handle_collect(&ctx, &json!({"id": "j1", "full_result": true})) + .await + .unwrap(); + + assert_eq!(result["status"], "completed"); + assert_eq!(result["result"], json!(big)); + assert!(result.get("result_truncated").is_none()); + }); + } + #[test] fn handle_collect_maps_panic_to_failed_with_ring_content() { run_async(async { @@ -2102,7 +2166,7 @@ mod tests { #[test] fn cap_result_normalizes_null_to_done() { - let (value, truncated) = cap_result(Value::Null, None); + let (value, truncated) = cap_result(Value::Null, None, false); assert_eq!(value, json!("DONE")); assert!(!truncated); @@ -2110,7 +2174,7 @@ mod tests { #[test] fn cap_result_preserves_small_values() { - let (value, truncated) = cap_result(json!({"a": 1}), None); + let (value, truncated) = cap_result(json!({"a": 1}), None, false); assert_eq!(value, json!({"a": 1})); assert!(!truncated); @@ -2119,22 +2183,40 @@ mod tests { #[test] fn cap_result_keeps_last_chars_on_char_boundary() { let text = "é".repeat(JOB_RESULT_TAIL_CAP_CHARS + 10); - let (value, truncated) = cap_result(json!(text), None); + let (value, truncated) = cap_result(json!(text), None, false); assert!(truncated); let capped = value.as_str().unwrap(); assert!(capped.starts_with(&format!( - "[truncated: kept last {} of {} chars]", + "[truncated: kept last {} of {} chars — ", JOB_RESULT_TAIL_CAP_CHARS, JOB_RESULT_TAIL_CAP_CHARS + 10 ))); } + #[test] + fn cap_result_full_result_skips_char_cap() { + let text = "a".repeat(JOB_RESULT_TAIL_CAP_CHARS + 10); + let (value, truncated) = cap_result(json!(text), None, true); + + assert!(!truncated); + assert_eq!(value, json!(text)); + } + + #[test] + fn cap_result_full_result_still_applies_tail_lines() { + let (value, truncated) = cap_result(json!("l1\nl2\nl3"), Some(2), true); + + assert!(truncated); + assert_eq!(value, json!("l2\nl3")); + } + #[test] fn tail_chars_floors_to_char_boundary() { let capped = tail_chars("aébc", 2).unwrap(); assert!(capped.ends_with("bc")); - assert!(capped.starts_with("[truncated: kept last 2 of 4 chars]")); + assert!(capped.starts_with("[truncated: kept last 2 of 4 chars — ")); + assert!(capped.contains("full_result: true")); assert!(tail_chars("abc", 3).is_none()); } @@ -2406,12 +2488,7 @@ mod tests { .unwrap(); assert_eq!(created["status"], "ok"); - let tasks = run_async(handle_agent_tool( - &mut ctx, - "agent__task_list", - &json!({}), - )) - .unwrap(); + let tasks = run_async(handle_agent_tool(&mut ctx, "agent__task_list", &json!({}))).unwrap(); assert_eq!(tasks["tasks"].as_array().unwrap().len(), 1); } From fa04e0937305a800a6f15fc2ebd5d6b367c02fb1 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 12:53:17 -0600 Subject: [PATCH 22/28] feat(graph): support max_concurrent_jobs at the graph level Graph agents could only inherit the app-wide job budget; the agent-level header in graph.yaml now accepts max_concurrent_jobs alongside model/temperature, flowing through AgentConfig::from_graph into the run-wide supervisor. Deliberately graph-wide, not per-node: jobs outlive the node that started them. --- graph.example.yaml | 6 ++++++ src/config/agent.rs | 3 +++ src/graph/types.rs | 4 ++++ src/graph/validator.rs | 1 + 4 files changed, 14 insertions(+) diff --git a/graph.example.yaml b/graph.example.yaml index f1ea9ae..f027538 100644 --- a/graph.example.yaml +++ b/graph.example.yaml @@ -36,6 +36,12 @@ top_p: null # Default sampling top-p for `llm` nodes reasoning_effort: null # Default reasoning effort for `llm` nodes that don't override it. # Only valid when the model declares reasoning_levels. +max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once across the + # whole graph run. Jobs live in the run-wide supervisor and outlive + # the `llm` node that started them, so the budget is graph-wide — + # there is no per-node override. Overrides the global setting; + # 0 disables background jobs for this graph agent. + global_tools: # Tool universe an `llm` node's `tools:` whitelist draws from - web_search_coyote.sh - fetch_url_via_curl.sh diff --git a/src/config/agent.rs b/src/config/agent.rs index 9fbf663..15638ea 100644 --- a/src/config/agent.rs +++ b/src/config/agent.rs @@ -854,6 +854,7 @@ impl AgentConfig { variables: graph.variables.clone(), can_spawn_agents: graph.has_agent_node(), max_concurrent_agents: default_max_concurrent_agents(), + max_concurrent_jobs: graph.max_concurrent_jobs, max_agent_depth: default_max_agent_depth(), escalation_timeout: default_escalation_timeout(), ..AgentConfig::default() @@ -1328,6 +1329,7 @@ variables: model: claude:claude-sonnet-4-6 temperature: 0.3 top_p: 0.8 + max_concurrent_jobs: 2 global_tools: - fetch_pdf.sh mcp_servers: @@ -1350,6 +1352,7 @@ variables: assert_eq!(config.model_id.as_deref(), Some("claude:claude-sonnet-4-6")); assert_eq!(config.temperature, Some(0.3)); assert_eq!(config.top_p, Some(0.8)); + assert_eq!(config.max_concurrent_jobs, Some(2)); assert_eq!(config.global_tools, vec!["fetch_pdf.sh"]); assert_eq!(config.mcp_servers, vec!["pubmed-search"]); assert_eq!(config.conversation_starters, vec!["Start here"]); diff --git a/src/graph/types.rs b/src/graph/types.rs index 18bdf5d..e45322b 100644 --- a/src/graph/types.rs +++ b/src/graph/types.rs @@ -28,6 +28,9 @@ pub struct Graph { #[serde(default, skip_serializing_if = "Option::is_none")] pub reasoning_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub max_concurrent_jobs: Option, + #[serde(default)] pub global_tools: Vec, @@ -895,6 +898,7 @@ nodes: assert!(graph.model.is_none()); assert!(graph.temperature.is_none()); assert!(graph.top_p.is_none()); + assert!(graph.max_concurrent_jobs.is_none()); assert!(graph.global_tools.is_empty()); assert!(graph.mcp_servers.is_empty()); assert!(graph.conversation_starters.is_empty()); diff --git a/src/graph/validator.rs b/src/graph/validator.rs index fe438f7..4b7cc64 100644 --- a/src/graph/validator.rs +++ b/src/graph/validator.rs @@ -998,6 +998,7 @@ mod tests { temperature: None, top_p: None, reasoning_effort: None, + max_concurrent_jobs: None, global_tools: Vec::new(), mcp_servers: Vec::new(), skills_enabled: None, From 1650196cae9b002917d195a4080df9437942c47a Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 12:53:17 -0600 Subject: [PATCH 23/28] docs(config): document max_concurrent_jobs in agent example config --- config.agent.example.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/config.agent.example.yaml b/config.agent.example.yaml index c4562df..18c731c 100644 --- a/config.agent.example.yaml +++ b/config.agent.example.yaml @@ -41,6 +41,8 @@ can_spawn_agents: false # Enable the agent to spawn child agents # Graph agents (graph.yaml) ignore this; they declare spawn targets in agent nodes. max_concurrent_agents: 4 # Maximum number of agents that can run simultaneously max_agent_depth: 3 # Maximum nesting depth for sub-agents (prevents runaway spawning) +max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once for this agent + # (overrides the global setting; 0 disables background jobs for this agent) inject_spawn_instructions: true # Inject the default agent spawning instructions into the agent's system prompt summarization_model: null # Model to use for summarizing sub-agent output (e.g. 'openai:gpt-4o-mini'); defaults to current model summarization_threshold: 4000 # Character threshold above which sub-agent output is summarized before returning to parent From bfb8105682210ab393eb16d6d41bd55482a585d3 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 13:00:16 -0600 Subject: [PATCH 24/28] fix(graph): gate unix-only test imports behind cfg(unix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The executor integration-test module hoisted job-test paths into module-level imports, but their only consumer is a #[cfg(unix)] test — on Windows the imports went unused and failed -D warnings. --- src/graph/executor.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/graph/executor.rs b/src/graph/executor.rs index d7fe458..4c2366b 100644 --- a/src/graph/executor.rs +++ b/src/graph/executor.rs @@ -563,10 +563,14 @@ mod tests { mod integration_tests { use super::*; use crate::config::{AppState, WorkingMode}; + #[cfg(unix)] use crate::function::jobs::RingBuf; + #[cfg(unix)] use crate::supervisor::{JobHandle, JobResult, JobState, JobStatus, Supervisor, notification}; use crate::utils::{create_abort_signal, temp_file}; - use std::{fs, mem}; + use std::fs; + #[cfg(unix)] + use std::mem; fn cmd_available(name: &str) -> bool { which::which(name).is_ok() From 404a45a31141390e73d71b16e587735b5c15476d Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 13:43:05 -0600 Subject: [PATCH 25/28] feat(jobs): node-local job ownership and capability-gated job__* visibility MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Graph LLM nodes now own the jobs they start, on every exit path. A new node_job_scope on RequestContext records job ids started while a node runs: the turn-end guardrail nags only about the node's own jobs (parallel branches no longer see each other's), and the node executor reaps — cancels and deregisters — anything left registered when the node exits, including error, timeout, and retry-exhaustion paths. Cross-node job handoff is no longer possible; a crashed node takes its in-flight jobs with it. With inheritance gone, job__* declarations are gated on capability: the family is only declared when at least one declared tool would pass job__start's whitelist (shared predicate: is_backgroundable_tool). One carve-out — while a context still owns registered jobs (job started, tool disabled mid-session), the lifecycle verbs stay declared so a running job can never become unreachable; job__start alone disappears. A graph node with tools: [] now sees no job__* tools at all. Prompt instructions, tool declarations, and graph.example.yaml updated to the node-local semantics; +7 tests, 8 visibility pins rewritten. --- graph.example.yaml | 10 +- src/config/prompts.rs | 8 +- src/config/request_context.rs | 204 ++++++++++++++++++++++++++++------ src/function/jobs.rs | 100 ++++++++++++++++- src/function/mod.rs | 4 + src/function/supervisor.rs | 23 ++++ src/graph/llm.rs | 7 ++ 7 files changed, 311 insertions(+), 45 deletions(-) diff --git a/graph.example.yaml b/graph.example.yaml index f027538..19a5d0b 100644 --- a/graph.example.yaml +++ b/graph.example.yaml @@ -37,10 +37,12 @@ reasoning_effort: null # Default reasoning effort for `llm` nodes th # Only valid when the model declares reasoning_levels. max_concurrent_jobs: 5 # Max background jobs (`job__*` tools) running at once across the - # whole graph run. Jobs live in the run-wide supervisor and outlive - # the `llm` node that started them, so the budget is graph-wide — - # there is no per-node override. Overrides the global setting; - # 0 disables background jobs for this graph agent. + # whole graph run: every `llm` node (including parallel branches) + # draws from this one pool, so the budget is graph-wide — there is + # no per-node override. Jobs themselves are node-local: the node + # that starts a job must collect or cancel it before it ends, and + # anything left running at node exit is cancelled. Overrides the + # global setting; 0 disables background jobs for this graph agent. global_tools: # Tool universe an `llm` node's `tools:` whitelist draws from - web_search_coyote.sh diff --git a/src/config/prompts.rs b/src/config/prompts.rs index beb4485..db1e3ce 100644 --- a/src/config/prompts.rs +++ b/src/config/prompts.rs @@ -201,10 +201,10 @@ pub(in crate::config) const DEFAULT_JOB_INSTRUCTIONS: &str = indoc! {" jobs with `job__list`. Collected results over 50,000 chars are tail-capped; collecting is consume-once, so when you need the complete output pass `full_result: true` (or have the command write to a file). Collect or cancel every job you started before ending your turn. In - graph LLM nodes, collect or cancel your jobs before ending your final node turn — an - uncollected job at node turn-end burns node iterations via the guardrail and can fail the - node. Jobs run against a snapshot of the current config/environment and do not survive - coyote exiting. + graph LLM nodes, jobs are node-local: collect or cancel every job you start before the node + ends — an uncollected job burns node iterations via the guardrail, and anything still + running when the node exits is cancelled with its result discarded. Jobs run against a + snapshot of the current config/environment and do not survive coyote exiting. " }; diff --git a/src/config/request_context.rs b/src/config/request_context.rs index fd00b11..fd28018 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -18,7 +18,7 @@ use super::{MessageContentToolCalls, prompts}; use crate::client::{Model, ModelType, list_models}; use crate::function::{ FunctionDeclaration, Functions, ToolCallTracker, ToolResult, - jobs::{DEFAULT_MAX_CONCURRENT_JOBS, JOB_FUNCTION_PREFIX}, + jobs::{DEFAULT_MAX_CONCURRENT_JOBS, JOB_FUNCTION_PREFIX, is_backgroundable_tool}, memory::MEMORY_FUNCTION_PREFIX, rag_query::RAG_FUNCTION_PREFIX, skill::SKILL_FUNCTION_PREFIX, @@ -326,6 +326,13 @@ pub struct RequestContext { pub declared_function_names: HashSet, + /// Ids of jobs started by the currently executing graph LLM node. + /// `Some` only while a node runs: `job__start` records into it, the + /// turn-end guardrail scopes its nag to it, and the node executor reaps + /// whatever is left in it on exit. `None` outside graph nodes — there the + /// context owns every job in its supervisor. + pub node_job_scope: Option>, + pub supervisor: Option>>, pub parent_supervisor: Option>>, pub self_agent_id: Option, @@ -361,6 +368,7 @@ impl RequestContext { last_message: None, tool_scope: ToolScope::default(), declared_function_names: Default::default(), + node_job_scope: None, supervisor: None, parent_supervisor: None, self_agent_id: None, @@ -422,6 +430,7 @@ impl RequestContext { tool_tracker: ToolCallTracker::default(), }, declared_function_names: Default::default(), + node_job_scope: None, supervisor: None, parent_supervisor: None, self_agent_id: None, @@ -470,6 +479,7 @@ impl RequestContext { last_message: self.last_message.clone(), tool_scope: self.tool_scope.clone(), declared_function_names: self.declared_function_names.clone(), + node_job_scope: None, supervisor: self.supervisor.clone(), parent_supervisor: self.parent_supervisor.clone(), self_agent_id: self.self_agent_id.clone(), @@ -516,6 +526,7 @@ impl RequestContext { tool_tracker: tool_call_tracker, }, declared_function_names: Default::default(), + node_job_scope: None, supervisor: None, parent_supervisor: parent.supervisor.clone(), self_agent_id: Some(self_agent_id), @@ -2317,6 +2328,7 @@ impl RequestContext { let mut functions = vec![]; functions.extend(self.select_enabled_functions(role)); functions.extend(self.select_enabled_mcp_servers(role)); + self.apply_job_tool_visibility(&mut functions); if functions.is_empty() { None @@ -2325,6 +2337,38 @@ impl RequestContext { } } + /// Node-local job-ownership visibility rule: the `job__*` family is only + /// declared where it can do something. `job__start` requires at least one + /// backgroundable tool among this request's declarations; the lifecycle + /// verbs (`check`/`collect`/`cancel`/`list`) additionally survive while + /// the context still owns registered jobs, so a job started before a + /// filter change stays reachable. + fn apply_job_tool_visibility(&self, functions: &mut Vec) { + let has_backgroundable = functions.iter().any(|f| is_backgroundable_tool(&f.name)); + if has_backgroundable { + return; + } + let owns_jobs = self.owns_active_jobs(); + let start_name = format!("{JOB_FUNCTION_PREFIX}start"); + functions.retain(|f| { + !f.name.starts_with(JOB_FUNCTION_PREFIX) || (owns_jobs && f.name != start_name) + }); + } + + /// Whether this context has registered jobs it is responsible for: + /// inside a graph LLM node, only the jobs that node started; everywhere + /// else, any job in the context's supervisor. + pub fn owns_active_jobs(&self) -> bool { + let Some(supervisor) = self.supervisor.as_ref() else { + return false; + }; + let sup = supervisor.read(); + match self.node_job_scope.as_ref() { + Some(ids) => ids.iter().any(|id| sup.job(id).is_some()), + None => sup.jobs().next().is_some(), + } + } + pub fn retrieve_role(&self, app: &AppConfig, name: &str) -> Result { let names = paths::list_roles(false); let mut role = if names.contains(&name.to_string()) { @@ -4819,6 +4863,15 @@ mod tests { RequestContext::new(default_app_state(), WorkingMode::Cmd) } + fn test_decl(name: &str) -> FunctionDeclaration { + FunctionDeclaration { + name: name.to_string(), + description: String::new(), + parameters: Default::default(), + agent: false, + } + } + fn tools_only_features(name: &str) -> McpServerFeatures { McpServerFeatures { name: name.to_string(), @@ -5776,37 +5829,49 @@ mod tests { } #[test] - fn select_functions_returns_job_functions_even_with_no_enabled_tools() { + fn select_functions_hides_job_functions_without_backgroundable_tools() { let mut ctx = create_test_ctx(); ctx.tool_scope.functions.append_job_functions(); - let fns = ctx.select_functions(&Role::default()).unwrap(); - let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); - - assert_eq!( - names, - vec![ - "job__start", - "job__check", - "job__collect", - "job__cancel", - "job__list" - ] + assert!( + ctx.select_functions(&Role::default()).is_none(), + "job__ tools must not be declared when nothing backgroundable is declared" ); } #[test] - fn select_functions_preserves_job_tools_under_role_filter() { + fn select_functions_keeps_job_tools_when_filter_includes_backgroundable_tool() { let mut ctx = create_test_ctx(); ctx.tool_scope.functions.append_job_functions(); + ctx.tool_scope + .functions + .append_declaration(test_decl("my_build_tool")); let mut role = Role::new("r", "p"); - role.set_enabled_tools(Some(vec!["foo".to_string()])); + role.set_enabled_tools(Some(vec!["my_build_tool".to_string()])); let fns = ctx.select_functions(&role).unwrap(); assert!( fns.iter().any(|f| f.name == "job__start"), - "job__ tools must survive a role tool filter" + "job__ tools must survive a role tool filter that declares a backgroundable tool" + ); + } + + #[test] + fn select_functions_hides_job_tools_when_filter_has_only_non_backgroundable_tools() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + ctx.tool_scope + .functions + .append_declaration(test_decl("fs_cat")); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["fs_cat".to_string()])); + + let fns = ctx.select_functions(&role).unwrap(); + assert!( + !fns.iter().any(|f| f.name.starts_with("job__")), + "job__ tools must be hidden when no declared tool is backgroundable" ); } @@ -5822,14 +5887,22 @@ mod tests { fn before_chat_completion_refreshes_declared_function_names() { let mut ctx = create_test_ctx(); ctx.tool_scope.functions.append_job_functions(); - let input = Input::from_str(&ctx, "hello", None).unwrap(); + ctx.tool_scope + .functions + .append_declaration(test_decl("echo")); + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["echo".to_string()])); + let input = Input::from_str(&ctx, "hello", Some(role)).unwrap(); ctx.before_chat_completion(&input).unwrap(); - assert_eq!(ctx.declared_function_names.len(), 5); + assert_eq!(ctx.declared_function_names.len(), 6); assert!(ctx.declared_function_names.contains("job__start")); + assert!(ctx.declared_function_names.contains("echo")); ctx.tool_scope = ToolScope::default(); - let input = Input::from_str(&ctx, "hello again", None).unwrap(); + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["echo".to_string()])); + let input = Input::from_str(&ctx, "hello again", Some(role)).unwrap(); ctx.before_chat_completion(&input).unwrap(); assert!( @@ -6084,6 +6157,9 @@ mod tests { let abort = utils::create_abort_signal(); run_async(ctx.use_agent(&app, &agent_name, None, abort)).unwrap(); + ctx.tool_scope + .functions + .append_declaration(test_decl("foo")); let mut role = Role::new("r", "p"); role.set_enabled_tools(Some(vec!["foo".to_string()])); @@ -6092,7 +6168,7 @@ mod tests { let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); assert!( names.contains(&"job__start"), - "job__ tools must survive an agent tool filter, got: {names:?}" + "job__ tools must survive an agent tool filter that declares a backgroundable tool, got: {names:?}" ); assert!(names.contains(&"job__collect")); } @@ -7563,31 +7639,72 @@ mod tests { } #[test] - fn select_functions_preserves_job_tools_under_empty_role_filter() { + fn select_functions_hides_job_tools_under_empty_role_filter() { let mut ctx = create_test_ctx(); ctx.tool_scope.functions.append_job_functions(); let mut role = Role::new("r", "p"); role.set_enabled_tools(Some(vec![])); + assert!( + ctx.select_functions(&role).is_none(), + "an empty tool filter declares nothing backgroundable, so job__ tools must be hidden" + ); + } + + #[test] + fn select_functions_keeps_lifecycle_job_tools_when_context_owns_jobs() { + let mut ctx = create_test_ctx(); + ctx.tool_scope.functions.append_job_functions(); + let sup = Arc::new(RwLock::new( + Supervisor::new(4, 3).with_max_concurrent_jobs(1), + )); + sup.write() + .register(make_running_job(utils::create_abort_signal())) + .unwrap(); + ctx.supervisor = Some(sup); + + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec![])); + let fns = ctx.select_functions(&role).unwrap(); let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); assert_eq!( names, - vec![ - "job__start", - "job__check", - "job__collect", - "job__cancel", - "job__list" - ], - "job__ tools must survive an empty role tool filter" + vec!["job__check", "job__collect", "job__cancel", "job__list"], + "lifecycle verbs must stay reachable while the context owns a job; job__start must not" ); } + #[test] + fn owns_active_jobs_respects_node_scope() { + let mut ctx = create_test_ctx(); + let sup = Arc::new(RwLock::new( + Supervisor::new(4, 3).with_max_concurrent_jobs(1), + )); + sup.write() + .register(make_running_job(utils::create_abort_signal())) + .unwrap(); + ctx.supervisor = Some(sup); + + assert!( + ctx.owns_active_jobs(), + "outside a node, the context owns every registry job" + ); + + ctx.node_job_scope = Some(vec![]); + assert!( + !ctx.owns_active_jobs(), + "a node owns only jobs it started, not other registry entries" + ); + + ctx.node_job_scope = Some(vec!["j1".to_string()]); + assert!(ctx.owns_active_jobs()); + } + #[test] #[serial] - fn select_functions_preserves_job_tools_under_empty_agent_filter() { + fn select_functions_hides_job_tools_under_empty_agent_filter() { let _guard = TestConfigDirGuard::new(); let mut ctx = create_test_ctx(); let app = ctx.app.config.clone(); @@ -7612,13 +7729,16 @@ mod tests { let mut role = Role::new("r", "p"); role.set_enabled_tools(Some(vec![])); - let fns = ctx.select_functions(&role).unwrap(); - let names: Vec<&str> = fns.iter().map(|f| f.name.as_str()).collect(); + let names: Vec = ctx + .select_functions(&role) + .unwrap_or_default() + .iter() + .map(|f| f.name.clone()) + .collect(); assert!( - names.contains(&"job__start"), - "job__ tools must survive an empty agent tool filter, got: {names:?}" + !names.iter().any(|n| n.starts_with("job__")), + "job__ tools must be hidden under an empty agent filter, got: {names:?}" ); - assert!(names.contains(&"job__collect")); } #[test] @@ -7638,6 +7758,9 @@ mod tests { ..(*app).clone() }; run_async(ctx.rebuild_tool_scope(&jobs_off, None, abort.clone())).unwrap(); + ctx.tool_scope + .functions + .append_declaration(test_decl("echo")); let without_jobs = serde_json::to_string(&ctx.select_functions(&role)).unwrap(); assert!( !without_jobs.contains("job__"), @@ -7645,6 +7768,9 @@ mod tests { ); run_async(ctx.rebuild_tool_scope(&app, None, abort)).unwrap(); + ctx.tool_scope + .functions + .append_declaration(test_decl("echo")); let with_jobs = ctx.select_functions(&role).unwrap(); assert!(with_jobs.iter().any(|f| f.name.starts_with("job__"))); let stripped: Vec = with_jobs @@ -7685,6 +7811,12 @@ mod tests { fn tools_info_lists_job_tools_when_enabled() { let mut ctx = create_test_ctx(); ctx.tool_scope.functions.append_job_functions(); + ctx.tool_scope + .functions + .append_declaration(test_decl("echo")); + let mut role = Role::new("r", "p"); + role.set_enabled_tools(Some(vec!["echo".to_string()])); + ctx.role = Some(role); let info = ctx.tools_info().unwrap(); diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 0b5fba4..135a97e 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -140,7 +140,8 @@ pub fn job_function_declarations() -> Vec { tools cannot be backgrounded. The job runs against a snapshot of the current config and \ environment; later changes do not affect it. Process jobs honor COYOTE_TOOL_TIMEOUT; MCP \ jobs have NO timeout — cancel a hung one with `job__cancel`. Jobs do not survive coyote \ - exiting.".to_string(), + exiting. In graph LLM nodes, jobs are node-local: collect or cancel every job you start \ + before the node ends — leftovers are cancelled at node exit.".to_string(), parameters: JsonSchema { type_value: Some("object".to_string()), properties: Some(IndexMap::from([ @@ -359,6 +360,13 @@ fn whitelist_rejection(tool: &str) -> Option { }) } +/// Whether a declared tool could be run as a background job. This is the +/// declare-side twin of `whitelist_rejection`: a tool is backgroundable +/// exactly when `job__start` would not reject it by name. +pub fn is_backgroundable_tool(tool: &str) -> bool { + whitelist_rejection(tool).is_none() +} + async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { if !jobs_enabled(ctx.agent.as_ref(), &ctx.app.config) { return Ok(json!({ @@ -511,6 +519,10 @@ async fn handle_start(ctx: &mut RequestContext, args: &Value) -> Result { })); } + if let Some(scope) = ctx.node_job_scope.as_mut() { + scope.push(job_id.clone()); + } + Ok(json!({ "status": "ok", "job_id": job_id, @@ -803,6 +815,36 @@ async fn kill_job_with_grace(handle: &mut JobHandle) { let _ = time::timeout(JOB_KILL_GRACE, &mut handle.join_handle).await; } +/// Cancel and deregister the named jobs, if still registered. Used by graph +/// LLM nodes to enforce node-local job ownership: any job the node started +/// but did not collect or cancel by the time it exits is killed here, on +/// every exit path. Returns the ids actually reaped. +pub async fn reap_jobs( + supervisor: Option<&Arc>>, + ids: &[String], +) -> Vec { + let Some(supervisor) = supervisor else { + return Vec::new(); + }; + let mut reaped = Vec::new(); + for id in ids { + let handle = { + let mut sup = supervisor.write(); + sup.take_job(id) + }; + if let Some(mut handle) = handle { + handle.abort_signal.set_ctrlc(); + kill_job_with_grace(&mut handle).await; + warn!( + "Reaped background job '{id}' ({}): left unreclaimed at graph node exit", + handle.tool + ); + reaped.push(id.clone()); + } + } + reaped +} + fn handle_list(ctx: &RequestContext) -> Result { let Some(supervisor) = ctx.supervisor.as_ref() else { return Ok(json!({ @@ -1552,6 +1594,62 @@ mod tests { ); } + #[test] + fn is_backgroundable_tool_matches_start_whitelist() { + assert!(is_backgroundable_tool("execute_command")); + assert!(is_backgroundable_tool("my_custom_tool.sh")); + assert!(is_backgroundable_tool("mcp_invoke_github")); + assert!(!is_backgroundable_tool("job__start")); + assert!(!is_backgroundable_tool("agent__spawn")); + assert!(!is_backgroundable_tool("user__confirm")); + assert!(!is_backgroundable_tool("todo__add")); + assert!(!is_backgroundable_tool("fs_read")); + assert!(!is_backgroundable_tool("ast_grep")); + assert!(!is_backgroundable_tool("mcp_search_github")); + } + + #[cfg(unix)] + #[test] + fn handle_start_records_job_in_node_scope() { + run_async(async { + let mut ctx = plain_ctx(); + ctx.node_job_scope = Some(Vec::new()); + ctx.declared_function_names.insert("echo".into()); + + let started = handle_start(&mut ctx, &json!({"tool": "echo", "arguments": {}})) + .await + .unwrap(); + + let job_id = started["job_id"].as_str().unwrap().to_string(); + assert_eq!(ctx.node_job_scope.clone().unwrap(), vec![job_id.clone()]); + + let collected = handle_collect(&ctx, &json!({"id": job_id})).await.unwrap(); + assert_eq!(collected["status"], "completed"); + }); + } + + #[test] + fn reap_jobs_kills_registered_jobs_and_reports_ids() { + run_async(async { + let ctx = ctx_with_job_supervisor(4); + ctx.supervisor + .as_ref() + .unwrap() + .write() + .register(make_running_job("j1")) + .unwrap(); + + let reaped = reap_jobs( + ctx.supervisor.as_ref(), + &["j1".to_string(), "missing".to_string()], + ) + .await; + + assert_eq!(reaped, vec!["j1".to_string()]); + assert!(!ctx.supervisor.as_ref().unwrap().read().has_job("j1")); + }); + } + #[test] fn handle_start_rejects_unconnected_mcp_server() { let mut ctx = plain_ctx(); diff --git a/src/function/mod.rs b/src/function/mod.rs index 3969aaf..87d6aad 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -693,6 +693,10 @@ impl Functions { self.declarations.extend(jobs::job_function_declarations()); } + pub fn append_declaration(&mut self, declaration: FunctionDeclaration) { + self.declarations.push(declaration); + } + pub fn append_teammate_functions(&mut self) { self.declarations .extend(supervisor::teammate_function_declarations()); diff --git a/src/function/supervisor.rs b/src/function/supervisor.rs index 5e05d2c..52a4081 100644 --- a/src/function/supervisor.rs +++ b/src/function/supervisor.rs @@ -74,6 +74,13 @@ pub fn pending_tasks(ctx: &RequestContext) -> Vec { }) .collect(); + // Inside a graph LLM node, jobs are node-owned: the guardrail must only + // nag about jobs this node started. Jobs belonging to a parallel branch + // live in the same shared registry but are that branch's to reclaim. + if let Some(scope) = ctx.node_job_scope.as_ref() { + tasks.retain(|t| t.kind != TaskKind::Job || scope.contains(&t.id)); + } + tasks.sort_by(|a, b| a.id.cmp(&b.id)); tasks } @@ -2596,6 +2603,22 @@ mod tests { assert_eq!(tasks[0].kind, TaskKind::Job); } + #[test] + fn pending_tasks_scopes_jobs_to_node_scope() { + let mut ctx = ctx_with_job_capable_supervisor(); + register_fake_job(&mut ctx, "job_mine"); + register_fake_job(&mut ctx, "job_other"); + + ctx.node_job_scope = Some(vec!["job_mine".to_string()]); + let tasks = pending_tasks(&ctx); + assert_eq!(tasks.len(), 1); + assert_eq!(tasks[0].id, "job_mine"); + assert_eq!(tasks[0].kind, TaskKind::Job); + + ctx.node_job_scope = None; + assert_eq!(pending_tasks(&ctx).len(), 2); + } + #[test] fn guardrail_force_terminates_at_max_and_cancels_agents() { let rt = tokio::runtime::Builder::new_current_thread() diff --git a/src/graph/llm.rs b/src/graph/llm.rs index 7e19a19..36a17dd 100644 --- a/src/graph/llm.rs +++ b/src/graph/llm.rs @@ -6,6 +6,7 @@ use crate::config::prompts::DEFAULT_SKILL_INSTRUCTIONS; use crate::config::{ Input, RequestContext, Role, RoleLike, SkillPolicy, should_inject_skill_instructions, }; +use crate::function::jobs::reap_jobs; use crate::function::skill::skill_function_declarations; use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::utils::create_abort_signal; @@ -173,6 +174,9 @@ async fn run( let saved_role = parent_ctx.role.clone(); parent_ctx.role = Some(composed_role); + // Jobs are node-local: everything job__start registers while this node + // runs is recorded here and reaped on every exit path below. + let saved_job_scope = parent_ctx.node_job_scope.replace(Vec::new()); let result = match node.timeout { Some(secs) => match timeout( Duration::from_secs(secs), @@ -186,6 +190,9 @@ async fn run( None => run_with_retries(node, &prompt, parent_ctx).await, }; parent_ctx.role = saved_role; + let node_jobs = + std::mem::replace(&mut parent_ctx.node_job_scope, saved_job_scope).unwrap_or_default(); + reap_jobs(parent_ctx.supervisor.as_ref(), &node_jobs).await; restore_agent_skill_policy(parent_ctx, saved_agent_skill_state); result } From 198c9f42df32499a13cfcda06c758163319de20e Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 13:50:44 -0600 Subject: [PATCH 26/28] fix(function): gate test-only declaration appender behind cfg(test) append_declaration is exercised only by unit tests; in the plain bin target it tripped dead_code under CI's RUSTFLAGS --deny warnings. --- src/function/mod.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/src/function/mod.rs b/src/function/mod.rs index 87d6aad..acb5fcd 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -693,6 +693,7 @@ impl Functions { self.declarations.extend(jobs::job_function_declarations()); } + #[cfg(test)] pub fn append_declaration(&mut self, declaration: FunctionDeclaration) { self.declarations.push(declaration); } From 304b8f635f65b57576bfa97e34ee0fadd9749eb9 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 14:15:34 -0600 Subject: [PATCH 27/28] fix(tools): interactive-shell semantics and stderr capture in execute_command Two long-standing agent-facing defects: 1. bash -e aborted the model's script at the first intermediate non-zero status (grep with no matches exits 1, inspecting a failing test run, a probing subshell), so trailing guards like '; exit 0' never executed and output was partially or entirely lost. Dropped -e: the last statement now decides the exit code, matching the interactive-shell semantics models expect. pipefail is kept so a failing pipeline stage still surfaces in the exit code. 2. Only stdout was redirected into $LLM_OUTPUT, and the harness returns just $LLM_OUTPUT on success, so commands whose useful output goes to stderr (git push, cargo progress, curl -v) returned empty on success. Added 2>&1. --- assets/functions/tools/execute_command.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/assets/functions/tools/execute_command.sh b/assets/functions/tools/execute_command.sh index 4105a85..c360aef 100755 --- a/assets/functions/tools/execute_command.sh +++ b/assets/functions/tools/execute_command.sh @@ -20,5 +20,12 @@ main() { trap "rm -f '$script'" EXIT # shellcheck disable=SC2154 printf '%s\n' "$argc_command" > "$script" - bash -e -o pipefail "$script" >> "$LLM_OUTPUT" + # No -e: the command gets standard interactive-shell semantics — the last + # statement decides the exit code, so trailing guards like `; exit 0` work + # and an intermediate non-zero status (grep with no matches, a failing + # test run being inspected) cannot abort the script mid-way. pipefail is + # kept so a failing pipeline stage still surfaces in the exit code. 2>&1: + # the harness only returns $LLM_OUTPUT on success, so without it stderr + # (git push, cargo progress, curl -v) vanishes from successful calls. + bash -o pipefail "$script" >> "$LLM_OUTPUT" 2>&1 } From 429ae3cc8e4e847235532a5fde2199a203423d04 Mon Sep 17 00:00:00 2001 From: Alex Clarke Date: Wed, 26 Aug 2026 14:57:36 -0600 Subject: [PATCH 28/28] refactor(function): finish supervisor-to-agent vocabulary migration The supervisor registry went kind-generic (TaskHandle::Agent | Job) earlier in this branch, but the module holding the agent__* handlers and two model-facing error strings still carried the old name: - src/function/supervisor.rs -> src/function/agents.rs (it contains only agent__* tool handlers, pairing with function/jobs.rs; the kind-generic src/supervisor/ registry keeps its name) - 'Supervisor tool failed' -> 'Agent tool failed' - 'Unknown supervisor action' -> 'Unknown agent action' --- src/acp/server.rs | 2 +- src/config/request_context.rs | 2 +- src/function/{supervisor.rs => agents.rs} | 4 ++-- src/function/jobs.rs | 4 ++-- src/function/mod.rs | 16 ++++++++-------- src/graph/agent.rs | 2 +- src/graph/llm.rs | 2 +- src/main.rs | 2 +- src/repl/mod.rs | 2 +- 9 files changed, 18 insertions(+), 18 deletions(-) rename src/function/{supervisor.rs => agents.rs} (99%) diff --git a/src/acp/server.rs b/src/acp/server.rs index c8cdcec..506c15e 100644 --- a/src/acp/server.rs +++ b/src/acp/server.rs @@ -1,7 +1,7 @@ use super::types::{METHOD_NOT_FOUND, PARSE_ERROR, Request, Response}; use crate::client::call_chat_completions_streaming; use crate::config::{Input, RenderMode, RequestContext}; -use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; +use crate::function::agents::{GuardrailAction, check_pending_tasks_guardrail}; use crate::utils; use crate::utils::AbortSignal; use anyhow::Result; diff --git a/src/config/request_context.rs b/src/config/request_context.rs index fd28018..8fd9ab4 100644 --- a/src/config/request_context.rs +++ b/src/config/request_context.rs @@ -18,11 +18,11 @@ use super::{MessageContentToolCalls, prompts}; use crate::client::{Model, ModelType, list_models}; use crate::function::{ FunctionDeclaration, Functions, ToolCallTracker, ToolResult, + agents::AGENT_FUNCTION_PREFIX, jobs::{DEFAULT_MAX_CONCURRENT_JOBS, JOB_FUNCTION_PREFIX, is_backgroundable_tool}, memory::MEMORY_FUNCTION_PREFIX, rag_query::RAG_FUNCTION_PREFIX, skill::SKILL_FUNCTION_PREFIX, - supervisor::AGENT_FUNCTION_PREFIX, todo::TODO_FUNCTION_PREFIX, user_interaction::USER_FUNCTION_PREFIX, }; diff --git a/src/function/supervisor.rs b/src/function/agents.rs similarity index 99% rename from src/function/supervisor.rs rename to src/function/agents.rs index 52a4081..e7803ba 100644 --- a/src/function/supervisor.rs +++ b/src/function/agents.rs @@ -524,7 +524,7 @@ pub async fn handle_agent_tool( "task_complete" => handle_task_complete(ctx, args).await, "task_fail" => handle_task_fail(ctx, args), "reply_escalation" => handle_reply_escalation(ctx, args), - _ => bail!("Unknown supervisor action: {action}"), + _ => bail!("Unknown agent action: {action}"), } } @@ -2150,7 +2150,7 @@ mod tests { result .unwrap_err() .to_string() - .contains("Unknown supervisor action") + .contains("Unknown agent action") ); } diff --git a/src/function/jobs.rs b/src/function/jobs.rs index 135a97e..262b3ef 100644 --- a/src/function/jobs.rs +++ b/src/function/jobs.rs @@ -1,7 +1,7 @@ +use super::agents::AGENT_FUNCTION_PREFIX; use super::memory::MEMORY_FUNCTION_PREFIX; use super::rag_query::RAG_FUNCTION_PREFIX; use super::skill::SKILL_FUNCTION_PREFIX; -use super::supervisor::AGENT_FUNCTION_PREFIX; use super::todo::TODO_FUNCTION_PREFIX; use super::user_interaction::USER_FUNCTION_PREFIX; use super::{FunctionDeclaration, JsonSchema, PATH_SEP, mcp_error_display, render_tool_result}; @@ -1217,7 +1217,7 @@ fn tail_chars(text: &str, max_chars: usize) -> Option { mod tests { use super::*; use crate::config::{AppConfig, AppState, WorkingMode}; - use crate::function::supervisor::{ + use crate::function::agents::{ GuardrailAction, check_pending_tasks_guardrail, handle_agent_tool, }; use crate::supervisor::mailbox::Inbox; diff --git a/src/function/mod.rs b/src/function/mod.rs index acb5fcd..38e72a9 100644 --- a/src/function/mod.rs +++ b/src/function/mod.rs @@ -1,8 +1,8 @@ +pub(crate) mod agents; pub(crate) mod jobs; pub(crate) mod memory; pub(crate) mod rag_query; pub(crate) mod skill; -pub(crate) mod supervisor; pub(crate) mod todo; pub(crate) mod user_interaction; @@ -24,6 +24,7 @@ use crate::mcp::{ McpServersConfig, is_mcp_meta_function, render, }; use crate::parsers::{bash, python, typescript}; +use agents::AGENT_FUNCTION_PREFIX; use anyhow::{Context, Result, anyhow, bail}; use futures_util::future; use indexmap::IndexMap; @@ -48,7 +49,6 @@ use std::{ time::{Duration, Instant}, }; use strum_macros::AsRefStr; -use supervisor::AGENT_FUNCTION_PREFIX; use todo::TODO_FUNCTION_PREFIX; use user_interaction::USER_FUNCTION_PREFIX; @@ -684,9 +684,9 @@ impl Functions { pub fn append_supervisor_functions(&mut self) { self.declarations - .extend(supervisor::agent_function_declarations()); + .extend(agents::agent_function_declarations()); self.declarations - .extend(supervisor::escalation_function_declarations()); + .extend(agents::escalation_function_declarations()); } pub fn append_job_functions(&mut self) { @@ -700,7 +700,7 @@ impl Functions { pub fn append_teammate_functions(&mut self) { self.declarations - .extend(supervisor::teammate_function_declarations()); + .extend(agents::teammate_function_declarations()); } pub fn append_user_interaction_functions(&mut self) { @@ -1595,10 +1595,10 @@ impl ToolCall { }) } _ if cmd_name.starts_with(AGENT_FUNCTION_PREFIX) => { - supervisor::handle_agent_tool(ctx, &cmd_name, &json_data) + agents::handle_agent_tool(ctx, &cmd_name, &json_data) .await .unwrap_or_else(|e| { - let error_msg = format!("Supervisor tool failed: {e}"); + let error_msg = format!("Agent tool failed: {e}"); eprintln!("{}", muted_warning_text(&format!("⚠️ {error_msg} ⚠️"))); json!({"tool_call_error": error_msg}) }) @@ -4743,7 +4743,7 @@ mod tests { run_async(call_with_args("agent__check", json!({"id": "x"})).eval(&mut ctx)).unwrap(); let err = out["tool_call_error"].as_str().unwrap(); - assert!(err.starts_with("Supervisor tool failed"), "{err}"); + assert!(err.starts_with("Agent tool failed"), "{err}"); assert!(err.contains("No supervisor active"), "{err}"); } diff --git a/src/graph/agent.rs b/src/graph/agent.rs index de95cfd..9bde501 100644 --- a/src/graph/agent.rs +++ b/src/graph/agent.rs @@ -2,7 +2,7 @@ use super::state::StateManager; use super::structured; use super::types::AgentNode; use crate::config::RequestContext; -use crate::function::supervisor::run_agent_for_graph; +use crate::function::agents::run_agent_for_graph; use anyhow::{Context, Result}; use serde_json::Value; use std::time::Duration; diff --git a/src/graph/llm.rs b/src/graph/llm.rs index 36a17dd..623d110 100644 --- a/src/graph/llm.rs +++ b/src/graph/llm.rs @@ -6,9 +6,9 @@ use crate::config::prompts::DEFAULT_SKILL_INSTRUCTIONS; use crate::config::{ Input, RequestContext, Role, RoleLike, SkillPolicy, should_inject_skill_instructions, }; +use crate::function::agents::{GuardrailAction, check_pending_tasks_guardrail}; use crate::function::jobs::reap_jobs; use crate::function::skill::skill_function_declarations; -use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; use crate::utils::create_abort_signal; use anyhow::{Context, Error, Result, anyhow, bail}; use log::warn; diff --git a/src/main.rs b/src/main.rs index 0dcb9ae..9309149 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,7 +29,7 @@ use crate::config::{ install_builtins, list_agents, load_env_file, macro_execute, sync_models, }; use crate::config::{memory, paths}; -use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; +use crate::function::agents::{GuardrailAction, check_pending_tasks_guardrail}; use crate::mcp::McpServersConfig; use crate::render::{prompt_theme, render_error}; use crate::repl::Repl; diff --git a/src/repl/mod.rs b/src/repl/mod.rs index be3b742..c53202c 100644 --- a/src/repl/mod.rs +++ b/src/repl/mod.rs @@ -16,7 +16,7 @@ use crate::config::{ StateFlags, flatten_prompt_messages, macro_execute, resolve_prompt_args, sanitize_display_text, }; use crate::config::{AssetCategory, paths}; -use crate::function::supervisor::{GuardrailAction, check_pending_tasks_guardrail}; +use crate::function::agents::{GuardrailAction, check_pending_tasks_guardrail}; use crate::render::render_error; use crate::utils::{ AbortSignal, SHELL, abortable_run_with_spinner, create_abort_signal, dimmed_text,