docs: Created documentation on the new background job subsystem

Agents.md

@@ -541,6 +541,12 @@ For a working example of an orchestrator agent that uses sub-agent spawning, see
 [sisyphus](https://github.com/Dark-Alex-17/coyote/blob/main/assets/agents/sisyphus) agent. For an example of the teammate messaging pattern with parallel sub-agents, 
 see the [code-reviewer](https://github.com/Dark-Alex-17/coyote/blob/main/assets/agents/code-reviewer) agent.
 
+> **Jobs vs. sub-agents:** sub-agents are for work that needs *reasoning*: they run a full LLM loop. To background a
+> single long-running **tool call** (a build, a test suite, a slow MCP invocation), use the
+> [Background Jobs](Background-Jobs) system (`job__*` tools) instead. The two compose one way only: agents may start
+> jobs, but a job can never start an agent or another job. Completion notifications
+> (`system_notifications` entries on the next tool result) cover both systems.
+
 ## Spawning Configuration
 
 | Setting                     | Type              | Default       | Description                                                                    |

Background-Jobs.md

@@ -0,0 +1,216 @@
+# Background Jobs
+
+Coyote can run long tool calls (e.g. builds, test suites, slow shell commands, slow MCP calls, etc.) as **background jobs**.
+A background job is a single whitelisted tool call running as a detached task: the model starts it with `job__start`,
+keeps working while it runs, gets a push notification when it finishes, and retrieves the result with `job__collect`.
+
+Background jobs are available everywhere function calling is: plain REPL sessions, roles, agents, sub-agents, and
+graph LLM nodes.
+
+## Jobs vs. Sub-Agents
+
+Jobs and [sub-agents](Agents#7-sub-agent-spawning-system) are complementary async systems:
+
+| | Background job | Sub-agent |
+|---|---|---|
+| What runs | ONE tool call (a process or an MCP invocation) | A full agent with its own LLM loop |
+| Thinks? | No. It only produces output | Yes. Plans, calls tools, can spawn its own agents/jobs |
+| Use for | "This single command takes minutes" | "This sub-task needs reasoning" |
+| Budget | `max_concurrent_jobs` (default 5) | `max_concurrent_agents` (default 4) |
+
+The two systems compose **one way only**: agents (and graph LLM nodes) may start jobs, but a job can never start an
+agent, another job, or invoke any built-in tool. A job task owns only a frozen snapshot of what it needs to run its
+one tool call. There is no agent context inside a job for anything else to run against.
+
+## The Five Tools
+
+| Tool | Purpose |
+|------|---------|
+| `job__start` | Run a whitelisted tool call in the background. Returns a `job_<hex>` id immediately. |
+| `job__check` | Non-blocking status probe: status, elapsed time, and a tail of output captured so far. Never consumes the result. |
+| `job__collect` | Block until the job finishes, then return its result and remove the job. The single retrieval verb. Results over 50k chars are tail-capped by default; `full_result: true` skips the cap, `tail_lines: N` keeps only the last N lines. |
+| `job__cancel` | Kill the job's process group (SIGTERM, then SIGKILL after a 5s grace) and discard the handle. Returns partial output. |
+| `job__list` | List registered jobs with status, elapsed time, and bytes of output captured. |
+
+`job__start` takes `{ tool, arguments }`, where `arguments` is the same object the tool takes when called directly.
+It responds immediately:
+
+```json
+{
+  "status": "ok",
+  "job_id": "job_a1b2c3d4",
+  "tool": "execute_command",
+  "message": "Running in background. Check with job__check, block with job__collect, cancel with job__cancel. You will receive a system_notifications entry on completion. Jobs do not survive coyote exiting."
+}
+```
+
+### Worked example: a long build
+
+1. **Start:** The model backgrounds the slow command and keeps working:
+
+   ```
+   job__start --tool execute_command --arguments {"command": "cargo build --release"}
+   → { "status": "ok", "job_id": "job_a1b2c3d4", ... }
+   ```
+
+2. **Keep working:** The model edits files, runs other tools, or answers questions. An occasional
+   `job__check --id job_a1b2c3d4` shows live progress (`output_tail` is the last chunk of stdout+stderr captured
+   so far), without consuming anything.
+
+3. **Notification arrives:** When the build finishes, the model's *next tool result* carries a
+   `system_notifications` entry (see [Push Notifications](#push-notifications) below) naming the exact collect
+   command.
+
+4. **Collect:** `job__collect --id job_a1b2c3d4` returns the result (status, `result`, `exit_code`, elapsed time,
+   final output tail) and removes the job. Collecting a still-running job simply blocks until it finishes. This is
+   useful when the model has nothing else to do.
+
+## What Can Be Backgrounded
+
+Backgroundable tools:
+
+* **External command tools:** `execute_command` and any custom Bash/Python/JavaScript tool
+  (see [Function Calling](Tools)), including agent-specific tools.
+* **`mcp_invoke_*`:** MCP server tool invocations (see [MCP Servers](MCP-Servers)).
+
+Everything else is rejected with a teaching error explaining what to do instead:
+
+| Attempted tool | Error |
+|---|---|
+| `agent__*`, `job__*` | `'<tool>' is already asynchronous — call it directly. Agents may start jobs, but jobs never start agents or other jobs.` |
+| `user__*` | `'<tool>' is interactive and must run in-turn — a background job cannot touch the terminal. Call it directly.` |
+| `todo__*`, `memory__*`, `skill__*`, `rag__*` | `'<tool>' mutates agent/session state and must run in-turn. Call it directly.` |
+| `fs_*`, `ast_grep` | `'<tool>' is fast — invoke it directly instead of backgrounding it.` |
+| `mcp_search_*`, `mcp_describe_*`, `mcp_read_*`, `mcp_prompt_*` | `'<tool>' is a sub-second call; invoke it directly.` |
+
+Each rejection ends with: `Backgroundable tools: external command tools (e.g. execute_command) and mcp_invoke_* calls.`
+
+Two more gates always apply:
+
+* **Context availability:** `job__start` can only background tools that were actually declared to the model in the
+  current request. A tool filtered out by a role/session/agent/graph-node `enabled_tools` list is rejected with
+  `'<tool>' is not enabled in this context — job__start can only background tools declared to you in this request.
+  Use the exact name of a tool from your current catalog.` Backgrounding is never a way around tool filters.
+* **Capacity:** At the concurrency limit, `job__start` rejects with
+  `At capacity: N/M jobs running. Collect or cancel one first.`
+
+And visibility follows capability: the `job__*` tools are only **declared** where they can do something. A context
+whose declared tools include nothing backgroundable (e.g. a graph `llm` node with `tools: []`) sees no `job__*`
+declarations at all. One carve-out: while a context still owns registered jobs (say a job was started and the tool
+it used was then disabled mid-session), the lifecycle verbs (`job__check`/`collect`/`cancel`/`list`) stay declared
+until the registry drains, only `job__start` disappears. A running job can never become unreachable.
+
+Other errors you may see: `No job '<id>' is registered — it may have already been collected or cancelled. job__list
+shows active jobs.` (unknown/consumed id), and cross-kind teaching errors — `'<id>' is a spawned agent, not a
+background job — use agent__check / agent__collect / agent__cancel` (and the inverse from the `agent__*` tools).
+
+## Push Notifications
+
+When a job (or a spawned agent; completion notifications cover both) finishes, coyote merges a
+`system_notifications` entry onto the **last tool result of the model's next tool batch**:
+
+```json
+{
+  "output": "...the tool's own result...",
+  "system_notifications": [
+    {
+      "event": "job_completed",
+      "id": "job_a1b2c3d4",
+      "tool_or_agent": "execute_command",
+      "status": "success",
+      "next_action": "job__collect --id job_a1b2c3d4 for output"
+    }
+  ],
+  "notification_instruction": "Background tasks have finished; collect each result with its next_action command."
+}
+```
+
+`event` is one of `job_completed`, `job_failed`, `agent_completed`, `agent_failed`. Notifications for jobs that were
+already collected or cancelled are dropped. The model is never pointed at a dead id.
+
+### The turn-end guardrail
+
+If the model tries to end its turn with running or finished-but-uncollected background tasks, coyote injects a
+system reminder instead of ending the turn:
+
+```
+[SYSTEM GUARDRAIL] You attempted to end your turn with 2 unreclaimed background task(s).
+
+Still running (1):
+- job_a1b2c3d4 (job): call `job__collect` (blocks until done, returns output) or `job__cancel` (discards)
+...
+Completed but UNCOLLECTED — collect NOW (1):
+- `job__collect --id job_ffee0011`
+```
+
+After 3 reminders without action, coyote cancels the remaining tasks and discards any uncollected results, then
+lets the turn end. In practice the model collects on the first reminder; the guardrail exists so results are never
+silently abandoned.
+
+## Configuration
+
+```yaml
+# config.yaml (global)
+max_concurrent_jobs: 5      # default: 5; 0 disables background jobs entirely
+```
+
+```yaml
+# agents/<name>/config.yaml (per-agent override)
+max_concurrent_jobs: 2      # this agent gets its own budget; omit to inherit the global value
+```
+
+```yaml
+# agents/<name>/graph.yaml (graph-agent override, agent-level — next to model/temperature)
+max_concurrent_jobs: 2      # budget for the whole graph run; omit to inherit the global value
+```
+
+* Resolution is agent-override → global → default `5`, exactly like `max_tool_result_chars`.
+* Also settable via the `COYOTE_MAX_CONCURRENT_JOBS` environment variable
+  (see [Environment Variables](Environment-Variables)).
+* **`0` disables the feature** for that context: the `job__*` tools and their prompt instructions are simply never
+  offered to the model. It's as if the feature doesn't exist.
+* **Function calling is required.** With a model or configuration that doesn't support function calling, jobs are
+  off for the same reason: no tool declarations ever reach the model.
+* Every job budget is per-context: a sub-agent's jobs register with the sub-agent, not its parent, and each agent
+  resolves its own `max_concurrent_jobs`. Cancelling an agent also cancels its jobs.
+* There is deliberately **no per-node budget** in graph agents: every `llm` node (including parallel branches in
+  the same super-step) draws from the graph run's one shared pool, so the budget belongs to the run as a whole.
+  (Job *ownership* is still node-local; see the fine print below.)
+
+## Behavioral Fine Print
+
+* **Snapshot semantics.** A job runs against a snapshot of the config, environment, and `PATH` taken at
+  `job__start`. Changes made afterwards don't affect a running job. A job only produces output; it never mutates
+  session state.
+* **No persistence.** Jobs die with the coyote process. Quitting the REPL kills every job's process group; there is
+  no reattach-after-restart. Switching agents (`.agent`) also cancels running jobs. Background work belongs to the
+  context that started it.
+* **Two output channels.** While running, stdout+stderr stream into a bounded ring buffer (last 64 KiB). That's the
+  `output_tail` that `job__check` shows, with `output_bytes_captured` counting everything ever written and
+  `tail_truncated` flagging a clipped tail. The *result* returned by `job__collect` is separate (the tool's actual
+  output), capped tail-first at the last 50,000 characters by default — build failures land at the tail, so that's
+  the end that's kept. The truncation header says exactly what was kept and what to do next time. Collecting is
+  consume-once, so decide **before** collecting (`job__check`'s `output_bytes_captured` shows the size):
+  pass `tail_lines: N` to keep only the last N lines, or `full_result: true` to skip the cap and return everything
+  (the session-wide `max_tool_result_chars` limit still applies). For very large outputs, prefer having the command
+  write to a file and paging it with `fs_read`.
+* **Timeouts.** Process jobs honor `COYOTE_TOOL_TIMEOUT` (default 1800s, `0` = unlimited), resolved at
+  `job__start`; on expiry the process group is killed and collect reports the timeout as a tool error. MCP jobs have
+  no timeout; cancel a hung one with `job__cancel`.
+* **Polling ergonomics.** `job__check` (and `job__list`) never trip coyote's tool-call loop detector, so a model can
+  legitimately poll. As a nudge against busy-waiting, once several consecutive `job__check` calls return an unchanged
+  status and output, the result gains a `hint` telling the model to stop polling and rely on the completion
+  notification instead; the counter resets the moment anything changes.
+* **REPL surfaces.** When enabled, the `job__*` tools appear in `.info tools`, but they're built-in infrastructure.
+  `.list tools` and `.tool enable`/`.tool disable` deliberately exclude them, so they can't be individually toggled
+  (`.tool enable job__start` errors with "Unknown tool"). Availability is governed solely by `max_concurrent_jobs`
+  and function-calling support.
+* **Graph LLM nodes.** Jobs are **node-local**: the [graph](Graph-Agents) `llm` node that starts a job must collect
+  or cancel it before the node ends. The turn-end guardrail enforces this on clean paths (an uncollected job burns
+  node iterations and can fail the node at its iteration limit), and anything still registered when the node exits,
+  on any path, including errors and timeouts, is cancelled with its result discarded. There is no handing a job
+  to a downstream node. Parallel branches only ever see (and are nagged about) their own jobs. A node whose
+  `tools:` whitelist declares nothing backgroundable sees no `job__*` tools at all.
+* **Escalations still surface.** If a `job__collect` would block while child agents have pending escalations, it
+  returns early with the escalation summary instead of deadlocking, and tells the model to reply first and collect
+  again.

Graph-Agents.md

@@ -65,6 +65,8 @@ model: anthropic:claude-sonnet-4-6   # default model for llm nodes
 temperature: 0.0                     # default sampling temperature
 top_p: null                          # default sampling top-p
 reasoning_effort: null               # default reasoning effort for llm nodes (model-dependent; e.g. low, medium, high)
+max_concurrent_jobs: null            # optional; graph-wide [background job](Background-Jobs) budget shared by all llm nodes
+                                     # (jobs themselves are node-local); overrides the global setting, 0 disables
 global_tools:                        # global tools available to nodes
   - web_search_coyote.sh
 mcp_servers:                         # MCP servers available to nodes

Home.md

@@ -45,6 +45,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g
 * [Skills](Skills): Modular knowledge or capability packs the LLM can load and unload mid-conversation. Multiple skills compose; instructions stack, tools and MCPs union.
 * [Agents](Agents): Leverage AI agents to perform complex tasks and workflows, including sub-agent spawning, teammate messaging, and user interaction tools.
   * [Graph Agents](Graph-Agents): Define an agent as a declarative, YAML-driven workflow. A directed graph of typed nodes (LLM calls, scripts, approvals, user input, RAG retrieval, sub-agent spawns).
+* [Background Jobs](Background-Jobs): Run long tool calls (builds, test suites, slow MCP calls) in the background with the `job__*` tools while the model keeps working; completion arrives as a push notification.
 * [Todo System](TODO-System): Built-in task tracking for improved agent reliability with smaller models.
 * [Environment Variables](Environment-Variables): Override and customize your Coyote configuration at runtime with environment variables.
 * [Client Configurations](Clients): Configuration instructions for various LLM providers.

_Sidebar.md

@@ -53,6 +53,11 @@
   - [State & Templates](Graph-Agents#state-and-template-syntax)
   - [Structured Output](Graph-Agents#structured-output-output_schema)
   - [Limitations](Graph-Agents#limitations--gotchas)
+- [Background Jobs](Background-Jobs)
+  - [Jobs vs. Sub-Agents](Background-Jobs#jobs-vs-sub-agents)
+  - [The Five Tools](Background-Jobs#the-five-tools)
+  - [Push Notifications](Background-Jobs#push-notifications)
+  - [Configuration](Background-Jobs#configuration)
 
 ## Knowledge & Automation
 - [Memory](Memory)