diff --git a/Agents.md b/Agents.md index 70567ac..c117a78 100644 --- a/Agents.md +++ b/Agents.md @@ -541,6 +541,12 @@ For a working example of an orchestrator agent that uses sub-agent spawning, see [sisyphus](https://github.com/Dark-Alex-17/coyote/blob/main/assets/agents/sisyphus) agent. For an example of the teammate messaging pattern with parallel sub-agents, see the [code-reviewer](https://github.com/Dark-Alex-17/coyote/blob/main/assets/agents/code-reviewer) agent. +> **Jobs vs. sub-agents:** sub-agents are for work that needs *reasoning*: they run a full LLM loop. To background a +> single long-running **tool call** (a build, a test suite, a slow MCP invocation), use the +> [Background Jobs](Background-Jobs) system (`job__*` tools) instead. The two compose one way only: agents may start +> jobs, but a job can never start an agent or another job. Completion notifications +> (`system_notifications` entries on the next tool result) cover both systems. + ## Spawning Configuration | Setting | Type | Default | Description | diff --git a/Background-Jobs.md b/Background-Jobs.md new file mode 100644 index 0000000..b115ecb --- /dev/null +++ b/Background-Jobs.md @@ -0,0 +1,216 @@ +# Background Jobs + +Coyote can run long tool calls (e.g. builds, test suites, slow shell commands, slow MCP calls, etc.) as **background jobs**. +A background job is a single whitelisted tool call running as a detached task: the model starts it with `job__start`, +keeps working while it runs, gets a push notification when it finishes, and retrieves the result with `job__collect`. + +Background jobs are available everywhere function calling is: plain REPL sessions, roles, agents, sub-agents, and +graph LLM nodes. + +## Jobs vs. Sub-Agents + +Jobs and [sub-agents](Agents#7-sub-agent-spawning-system) are complementary async systems: + +| | Background job | Sub-agent | +|---|---|---| +| What runs | ONE tool call (a process or an MCP invocation) | A full agent with its own LLM loop | +| Thinks? | No. It only produces output | Yes. Plans, calls tools, can spawn its own agents/jobs | +| Use for | "This single command takes minutes" | "This sub-task needs reasoning" | +| Budget | `max_concurrent_jobs` (default 5) | `max_concurrent_agents` (default 4) | + +The two systems compose **one way only**: agents (and graph LLM nodes) may start jobs, but a job can never start an +agent, another job, or invoke any built-in tool. A job task owns only a frozen snapshot of what it needs to run its +one tool call. There is no agent context inside a job for anything else to run against. + +## The Five Tools + +| Tool | Purpose | +|------|---------| +| `job__start` | Run a whitelisted tool call in the background. Returns a `job_` id immediately. | +| `job__check` | Non-blocking status probe: status, elapsed time, and a tail of output captured so far. Never consumes the result. | +| `job__collect` | Block until the job finishes, then return its result and remove the job. The single retrieval verb. Results over 50k chars are tail-capped by default; `full_result: true` skips the cap, `tail_lines: N` keeps only the last N lines. | +| `job__cancel` | Kill the job's process group (SIGTERM, then SIGKILL after a 5s grace) and discard the handle. Returns partial output. | +| `job__list` | List registered jobs with status, elapsed time, and bytes of output captured. | + +`job__start` takes `{ tool, arguments }`, where `arguments` is the same object the tool takes when called directly. +It responds immediately: + +```json +{ + "status": "ok", + "job_id": "job_a1b2c3d4", + "tool": "execute_command", + "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting." +} +``` + +### Worked example: a long build + +1. **Start:** The model backgrounds the slow command and keeps working: + + ``` + job__start --tool execute_command --arguments {"command": "cargo build --release"} + → { "status": "ok", "job_id": "job_a1b2c3d4", ... } + ``` + +2. **Keep working:** The model edits files, runs other tools, or answers questions. An occasional + `job__check --id job_a1b2c3d4` shows live progress (`output_tail` is the last chunk of stdout+stderr captured + so far), without consuming anything. + +3. **Notification arrives:** When the build finishes, the model's *next tool result* carries a + `system_notifications` entry (see [Push Notifications](#push-notifications) below) naming the exact collect + command. + +4. **Collect:** `job__collect --id job_a1b2c3d4` returns the result (status, `result`, `exit_code`, elapsed time, + final output tail) and removes the job. Collecting a still-running job simply blocks until it finishes. This is + useful when the model has nothing else to do. + +## What Can Be Backgrounded + +Backgroundable tools: + +* **External command tools:** `execute_command` and any custom Bash/Python/JavaScript tool + (see [Function Calling](Tools)), including agent-specific tools. +* **`mcp_invoke_*`:** MCP server tool invocations (see [MCP Servers](MCP-Servers)). + +Everything else is rejected with a teaching error explaining what to do instead: + +| Attempted tool | Error | +|---|---| +| `agent__*`, `job__*` | `'' is already asynchronous — call it directly. Agents may start jobs, but jobs never start agents or other jobs.` | +| `user__*` | `'' is interactive and must run in-turn — a background job cannot touch the terminal. Call it directly.` | +| `todo__*`, `memory__*`, `skill__*`, `rag__*` | `'' mutates agent/session state and must run in-turn. Call it directly.` | +| `fs_*`, `ast_grep` | `'' is fast — invoke it directly instead of backgrounding it.` | +| `mcp_search_*`, `mcp_describe_*`, `mcp_read_*`, `mcp_prompt_*` | `'' is a sub-second call; invoke it directly.` | + +Each rejection ends with: `Backgroundable tools: external command tools (e.g. execute_command) and mcp_invoke_* calls.` + +Two more gates always apply: + +* **Context availability:** `job__start` can only background tools that were actually declared to the model in the + current request. A tool filtered out by a role/session/agent/graph-node `enabled_tools` list is rejected with + `'' is not enabled in this context — job__start can only background tools declared to you in this request. + Use the exact name of a tool from your current catalog.` Backgrounding is never a way around tool filters. +* **Capacity:** At the concurrency limit, `job__start` rejects with + `At capacity: N/M jobs running. Collect or cancel one first.` + +And visibility follows capability: the `job__*` tools are only **declared** where they can do something. A context +whose declared tools include nothing backgroundable (e.g. a graph `llm` node with `tools: []`) sees no `job__*` +declarations at all. One carve-out: while a context still owns registered jobs (say a job was started and the tool +it used was then disabled mid-session), the lifecycle verbs (`job__check`/`collect`/`cancel`/`list`) stay declared +until the registry drains, only `job__start` disappears. A running job can never become unreachable. + +Other errors you may see: `No job '' is registered — it may have already been collected or cancelled. job__list +shows active jobs.` (unknown/consumed id), and cross-kind teaching errors — `'' is a spawned agent, not a +background job — use agent__check / agent__collect / agent__cancel` (and the inverse from the `agent__*` tools). + +## Push Notifications + +When a job (or a spawned agent; completion notifications cover both) finishes, coyote merges a +`system_notifications` entry onto the **last tool result of the model's next tool batch**: + +```json +{ + "output": "...the tool's own result...", + "system_notifications": [ + { + "event": "job_completed", + "id": "job_a1b2c3d4", + "tool_or_agent": "execute_command", + "status": "success", + "next_action": "job__collect --id job_a1b2c3d4 for output" + } + ], + "notification_instruction": "Background tasks have finished; collect each result with its next_action command." +} +``` + +`event` is one of `job_completed`, `job_failed`, `agent_completed`, `agent_failed`. Notifications for jobs that were +already collected or cancelled are dropped. The model is never pointed at a dead id. + +### The turn-end guardrail + +If the model tries to end its turn with running or finished-but-uncollected background tasks, coyote injects a +system reminder instead of ending the turn: + +``` +[SYSTEM GUARDRAIL] You attempted to end your turn with 2 unreclaimed background task(s). + +Still running (1): +- job_a1b2c3d4 (job): call `job__collect` (blocks until done, returns output) or `job__cancel` (discards) +... +Completed but UNCOLLECTED — collect NOW (1): +- `job__collect --id job_ffee0011` +``` + +After 3 reminders without action, coyote cancels the remaining tasks and discards any uncollected results, then +lets the turn end. In practice the model collects on the first reminder; the guardrail exists so results are never +silently abandoned. + +## Configuration + +```yaml +# config.yaml (global) +max_concurrent_jobs: 5 # default: 5; 0 disables background jobs entirely +``` + +```yaml +# agents//config.yaml (per-agent override) +max_concurrent_jobs: 2 # this agent gets its own budget; omit to inherit the global value +``` + +```yaml +# agents//graph.yaml (graph-agent override, agent-level — next to model/temperature) +max_concurrent_jobs: 2 # budget for the whole graph run; omit to inherit the global value +``` + +* Resolution is agent-override → global → default `5`, exactly like `max_tool_result_chars`. +* Also settable via the `COYOTE_MAX_CONCURRENT_JOBS` environment variable + (see [Environment Variables](Environment-Variables)). +* **`0` disables the feature** for that context: the `job__*` tools and their prompt instructions are simply never + offered to the model. It's as if the feature doesn't exist. +* **Function calling is required.** With a model or configuration that doesn't support function calling, jobs are + off for the same reason: no tool declarations ever reach the model. +* Every job budget is per-context: a sub-agent's jobs register with the sub-agent, not its parent, and each agent + resolves its own `max_concurrent_jobs`. Cancelling an agent also cancels its jobs. +* There is deliberately **no per-node budget** in graph agents: every `llm` node (including parallel branches in + the same super-step) draws from the graph run's one shared pool, so the budget belongs to the run as a whole. + (Job *ownership* is still node-local; see the fine print below.) + +## Behavioral Fine Print + +* **Snapshot semantics.** A job runs against a snapshot of the config, environment, and `PATH` taken at + `job__start`. Changes made afterwards don't affect a running job. A job only produces output; it never mutates + session state. +* **No persistence.** Jobs die with the coyote process. Quitting the REPL kills every job's process group; there is + no reattach-after-restart. Switching agents (`.agent`) also cancels running jobs. Background work belongs to the + context that started it. +* **Two output channels.** While running, stdout+stderr stream into a bounded ring buffer (last 64 KiB). That's the + `output_tail` that `job__check` shows, with `output_bytes_captured` counting everything ever written and + `tail_truncated` flagging a clipped tail. The *result* returned by `job__collect` is separate (the tool's actual + output), capped tail-first at the last 50,000 characters by default — build failures land at the tail, so that's + the end that's kept. The truncation header says exactly what was kept and what to do next time. Collecting is + consume-once, so decide **before** collecting (`job__check`'s `output_bytes_captured` shows the size): + pass `tail_lines: N` to keep only the last N lines, or `full_result: true` to skip the cap and return everything + (the session-wide `max_tool_result_chars` limit still applies). For very large outputs, prefer having the command + write to a file and paging it with `fs_read`. +* **Timeouts.** Process jobs honor `COYOTE_TOOL_TIMEOUT` (default 1800s, `0` = unlimited), resolved at + `job__start`; on expiry the process group is killed and collect reports the timeout as a tool error. MCP jobs have + no timeout; cancel a hung one with `job__cancel`. +* **Polling ergonomics.** `job__check` (and `job__list`) never trip coyote's tool-call loop detector, so a model can + legitimately poll. As a nudge against busy-waiting, once several consecutive `job__check` calls return an unchanged + status and output, the result gains a `hint` telling the model to stop polling and rely on the completion + notification instead; the counter resets the moment anything changes. +* **REPL surfaces.** When enabled, the `job__*` tools appear in `.info tools`, but they're built-in infrastructure. + `.list tools` and `.tool enable`/`.tool disable` deliberately exclude them, so they can't be individually toggled + (`.tool enable job__start` errors with "Unknown tool"). Availability is governed solely by `max_concurrent_jobs` + and function-calling support. +* **Graph LLM nodes.** Jobs are **node-local**: the [graph](Graph-Agents) `llm` node that starts a job must collect + or cancel it before the node ends. The turn-end guardrail enforces this on clean paths (an uncollected job burns + node iterations and can fail the node at its iteration limit), and anything still registered when the node exits, + on any path, including errors and timeouts, is cancelled with its result discarded. There is no handing a job + to a downstream node. Parallel branches only ever see (and are nagged about) their own jobs. A node whose + `tools:` whitelist declares nothing backgroundable sees no `job__*` tools at all. +* **Escalations still surface.** If a `job__collect` would block while child agents have pending escalations, it + returns early with the escalation summary instead of deadlocking, and tells the model to reply first and collect + again. diff --git a/Graph-Agents.md b/Graph-Agents.md index bd5ebe2..4f9d41c 100644 --- a/Graph-Agents.md +++ b/Graph-Agents.md @@ -65,6 +65,8 @@ model: anthropic:claude-sonnet-4-6 # default model for llm nodes temperature: 0.0 # default sampling temperature top_p: null # default sampling top-p reasoning_effort: null # default reasoning effort for llm nodes (model-dependent; e.g. low, medium, high) +max_concurrent_jobs: null # optional; graph-wide [background job](Background-Jobs) budget shared by all llm nodes + # (jobs themselves are node-local); overrides the global setting, 0 disables global_tools: # global tools available to nodes - web_search_coyote.sh mcp_servers: # MCP servers available to nodes diff --git a/Home.md b/Home.md index 838be3b..0df4704 100644 --- a/Home.md +++ b/Home.md @@ -45,6 +45,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g * [Skills](Skills): Modular knowledge or capability packs the LLM can load and unload mid-conversation. Multiple skills compose; instructions stack, tools and MCPs union. * [Agents](Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools. * [Graph Agents](Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns). +* [Background Jobs](Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working; completion arrives as a push notification. * [Todo System](TODO-System): Built-in task tracking for improved agent reliability with smaller models. * [Environment Variables](Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables. * [Client Configurations](Clients): Configuration instructions for various LLM providers. diff --git a/_Sidebar.md b/_Sidebar.md index 1577de2..ed5c5fd 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -53,6 +53,11 @@ - [State & Templates](Graph-Agents#state-and-template-syntax) - [Structured Output](Graph-Agents#structured-output-output_schema) - [Limitations](Graph-Agents#limitations--gotchas) +- [Background Jobs](Background-Jobs) + - [Jobs vs. Sub-Agents](Background-Jobs#jobs-vs-sub-agents) + - [The Five Tools](Background-Jobs#the-five-tools) + - [Push Notifications](Background-Jobs#push-notifications) + - [Configuration](Background-Jobs#configuration) ## Knowledge & Automation - [Memory](Memory)