Compare commits

...
Author SHA1 Message Date
Alex Clarke aa14e66c35 Merge pull request #18 from Dark-Alex-17/feat/mcp-resources-prompts
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Canceled after 0s
CI / All (windows-latest) (push) Canceled after 0s
feat: MCP server resources and prompts support
2026-08-25 12:13:31 -06:00
Dark-Alex-17 3c9f443bce feat(cli)!: rename --prompt to --temp-role
Completes the .prompt/.temp-role split: --prompt set an ad-hoc system
role, which is what .temp-role now means everywhere. The --prompt name
is left unbound so a future one-shot MCP prompt flag can take it with
properly designed non-interactive semantics. use_prompt follows the
rename as use_temp_role.

BREAKING CHANGE: invocations using --prompt <text> must switch to
--temp-role <text>; clap rejects the old flag loudly.
2026-08-25 12:04:45 -06:00
Dark-Alex-17 e55120dac6 fix(bundles): harden the install pipeline for cross-platform correctness
Windows review findings on the bundle provenance code:

- clones now pin core.autocrlf=false and core.eol=lf so recorded sha256
  values reflect repository bytes, not the machine's git config (autocrlf
  on Windows previously made every text file a false conflict on update),
  plus core.longpaths=true for deep bundle trees
- is_safe_relative_path additionally rejects NTFS alternate data stream
  colons, reserved device names (con, nul, COM1..), and trailing dots or
  spaces; such names never come from a valid checkout and previously
  desynced or failed on Windows
- file ownership dedupe compares paths case-insensitively on Windows and
  macOS where case variants denote one physical file (uninstalling one
  bundle could previously delete another bundle's file)
- a failed git clone no longer leaks its partial tree in the temp dir,
  and temp cleanup failures are logged instead of swallowed
- recording a bundle file outside the config dir (asset dir override)
  now warns instead of silently producing an undeletable record
2026-08-25 11:37:41 -06:00
Dark-Alex-17 b67f1ef854 refactor: pulled out some imports to clean up the MCP render module a bit 2026-08-25 11:37:41 -06:00
Dark-Alex-17 b38562a961 fix(mcp): harden the spill path for cross-platform correctness
Windows review findings: reserved device names (con, nul, COM1..) and
trailing dots in server names break or desync directory creation, so
sanitize_server now escapes reserved stems, strips trailing dots, and
caps length at 64 chars. Spill writes go through a temp file + rename
so a visible file is always complete (closes a cross-process partial
read race), and eviction protection compares content-hashed file names
instead of full paths. Also drops a duplicated cfg attribute.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 40846de37a test(bundles): skip uninstall ambiguity test when stdout is a TTY
The non-interactive bail under test only triggers without a TTY; from a
terminal the code correctly opens the interactive selector instead, so
the test hung or failed depending on input. Same guard as the three
sibling non-interactive tests.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 a5a3eed6d8 fix(repl): offer prompts in .list tab completion and rename its listing helpers
MCP prompts are live, server-owned catalog entries, not managed assets;
list_prompt_assets/prompt_asset_rows implied otherwise and are now
list_mcp_prompts/mcp_prompt_rows. The .list completer was also missing
the prompts kind that the usage string and unknown-kind error advertise.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 ad9ff3bea8 style: revised a few stylistic choices after I changed my mind 2026-08-25 11:37:41 -06:00
Dark-Alex-17 5177d95ee0 feat: complete --filter and --force on the first .install argument
The unified install parser accepts flags in any position, so the
first-argument completion list now offers all four flags instead of
only --git-host and --help.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 9bc37e226b docs: removed design doc from commit 2026-08-25 11:37:41 -06:00
Dark-Alex-17 a4b55d9e42 fix(mcp): gate unix-only spill permission APIs for windows builds 2026-08-25 11:37:41 -06:00
Dark-Alex-17 c8b00b20bc docs: document MCP resources and prompts support
Update the README's MCP feature entry to cover the full capability trio
(tools, resources, prompts): the capability-gated mcp_read/mcp_prompt
meta-tools, bounded results and blob spilling, and the .prompt REPL
command with staged tab-completion and .list prompts.

Per plans/mcp-resources-prompts-design.md section 10.T9.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 eb37f8bb46 feat(mcp): bound tool-result passthrough and surface resource audience annotations
Route CallToolResult content through the render.rs content policy per
plans/mcp-resources-prompts-design.md §6 (T8): oversized text sliced at
TEXT_MAX_BYTES_CLAMP with a self-explaining truncation note, image/audio/
embedded blob content spilled (or inlined when UTF-8-clean) instead of
shipping base64 into model context, and structuredContent subject to the
same ceiling. Clamp server-controlled uri/mime metadata strings to the new
METADATA_MAX_BYTES bound in both the read and tool-result paths, sanitize
the terminal rendering of MCP dispatch errors while keeping raw text in
the tool_call_error payload, and surface resource audience annotations in
both mcp_search results and mcp_read metadata via the catalog.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 6fade71e8c feat(mcp): add mcp_prompt meta-tool and harden prompt display rendering
Emit an mcp_prompt_<server> declaration for servers advertising the
prompts capability, execute prompts via McpRuntime::prompt on both tool
dispatch chains, and return the flattened prompt text as the tool
result. Sanitize server-controlled prompt names, descriptions, and
argument names before terminal rendering, and attribute the .prompt
argument inquire label to its server and prompt.

Per plans/mcp-resources-prompts-design.md §5.2 (T7).
2026-08-25 11:37:41 -06:00
Dark-Alex-17 61a3cfb662 feat(repl): add .prompt command with live staged tab-completion
Implements plans/mcp-resources-prompts-design.md §5.1/§5.4 (T6):

- .prompt <server> <name> [key=value ...] fetches an MCP prompt and
  submits the result as chat input via Input::from_str + ask(), never
  through REPL line parsing; GetPromptResult messages are flattened
  into one user-role block with unconditional [user]/[assistant] labels
- missing required prompt arguments are collected interactively
- .list prompts renders server/name/description/args via the unified
  catalog (CatalogItem gains an arguments field), degrading per server
- staged live tab-completion: enabled+running+prompts-capable servers
  (no RPC), then live prompt names, then key= argument suggestions with
  (required) markers; 2s timeout per RPC, all errors degrade to silent
  empty suggestions, ctx read guard dropped before blocking
- the enabled-server alias expansion is factored into a shared helper
  used by both tool-scope rebuild and completion
- BREAKING: the former .prompt <text> temp-role builtin is renamed to
  .temp-role <text> (behavior preserved); .prompt now belongs to MCP
  prompts, and a user macro named prompt or temp-role is shadowed
2026-08-25 11:37:41 -06:00
Dark-Alex-17 67819784b7 feat(mcp): add mcp_read meta-tool for resource reads
Implements plans/mcp-resources-prompts-design.md section 4.3 (T4):
mcp_read_<server> declaration and handler wired to render.rs, RFC 6570
Level-1-only URI template expansion, defensive ResourceContents parsing,
per-item text paging with pattern filtering, blob spill metadata, an
overall 204800-byte multi-content ceiling, dispatch wiring on both
eval chains, and a render_text paging-stall guard.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 437512fd6d feat(mcp): gate meta-function emission on advertised server capabilities
Per-server McpServerFeatures (tools fail-open, resources/prompts
fail-closed) now drive which meta-functions are declared, with
gated_meta_function_prefixes as the single gating seam; read/prompt
declarations land together with their handlers. The server-enablement
sentinel keys on the always-emitted search name so resources-only
servers survive role filtering.

Implements plans/mcp-resources-prompts-design.md §4.4/D7 (T5).
2026-08-25 11:37:41 -06:00
Dark-Alex-17 ef88b6a2c8 feat(mcp): add render.rs content policy (text paging, pattern filter, blob spill)
Single content-policy module for MCP resource and tool content, per
plans/mcp-resources-prompts-design.md §4.5 (T3):

- render_text: UTF-8-boundary-safe paging with clamped max_bytes and
  grep-style fancy-regex line filtering (2 lines of context, 1-based
  line-number prefixes, merged hunks); offsets walk the filtered stream.
- render_blob/render_blob_at: streaming base64 decode with a 50 MiB
  ceiling, UTF-8 sniff, sha256-named 0600 spill files under a sanitized
  server dir with a fixed mime->ext allowlist, and best-effort
  oldest-first eviction bounding the spill tree at 512 MiB.

Not yet wired to call sites; module carries #![allow(dead_code)] until
the read/prompt surfaces land.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 d68f4ecaeb feat(mcp): extend the server catalog to resources, templates, and prompts
Implements the unified catalog from plans/mcp-resources-prompts-design.md §4.2 (T2): CatalogItem gains kind/uri/mime_type/size keyed as {kind}:{id}; catalog_items() lists per kind gated by advertised capabilities with warn-and-degrade; mcp_search results carry kind; mcp_describe gains an optional kind param (default tool); write-only registry ServerCatalog removed.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 01ada1da18 refactor(mcp): centralize meta-function prefix predicates and fix list_tools pagination
Implements T1 of plans/mcp-resources-prompts-design.md (§4.1, §4.6):

- Replace list_tools(None) with cursor-following list_all_tools() at the
  three call sites (start_server catalog build, catalog_items, describe)
  so paginating servers no longer silently lose tools past page one.
- Add MCP_READ/MCP_PROMPT prefix constants (declared nowhere yet; wired
  in T4/T7) plus centralized helpers MCP_META_FUNCTION_PREFIXES,
  is_mcp_meta_function, and mcp_meta_function_names.
- Mechanically replace every hand-rolled 3-prefix starts_with triple
  (partition in eval_tool_calls, 3 exclusion triples in
  select_enabled_functions, 3 inclusion triples + per-server name
  construction in select_enabled_mcp_servers) with the helpers,
  preserving the existing lax starts_with matching semantics and the
  mcp_invoke_* enablement sentinel (sentinel moves to search in T5).
- Behavior-neutral: dispatch chains keep their 3 arms, emission stays
  at exactly 3 meta-functions per server, existing tests unmodified.
- Add unit tests: helper classification, prefix-soundness property,
  lax-matching pin, ordered candidate-name construction.
2026-08-25 11:37:41 -06:00
Dark-Alex-17 7caa24d090 docs(plans): add MCP resources & prompts design (v1.3, gate-approved)
Gatekeeper: SEALED. Oracle: APPROVE-WITH-CHANGES (B1-B3 folded in).
Phases: unified catalog + mcp_read w/ render.rs content policy;
.prompt REPL + staged live tab-completion + mcp_prompt meta-tool;
CallToolResult bounding; capability gating via McpRuntime::server_features.
2026-08-25 11:37:41 -06:00
Dark-Alex-17andSisyphus b972c12559 docs: extend the mattpocock/skills credit to the grilling adaptation
CI / All (ubuntu-latest) (push) Failing after 33s
CI / All (macos-latest) (push) Canceled after 0s
CI / All (windows-latest) (push) Canceled after 0s
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:12:37 -06:00
Dark-Alex-17andSisyphus f8cab9b439 feat: run design interviews as grilling frontier rounds across the planning agents
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:12:37 -06:00
Dark-Alex-17andSisyphus 45333db5c2 feat: add a grilling skill for frontier-round design interviews
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:12:37 -06:00
Dark-Alex-17andSisyphus 7748b953f0 docs: extend the mattpocock/skills credit to the codebase-design adaptations
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:07:50 -06:00
Dark-Alex-17andSisyphus 720591d24a feat: add an on-demand architecture-reviewer agent for deepening scans
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:07:50 -06:00
Dark-Alex-17andSisyphus dbb0c51b7e feat: add a codebase-design skill with the deep-module design vocabulary
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:07:50 -06:00
Dark-Alex-17andSisyphus 71cd50fe4d docs: credit mattpocock/skills for the diagnosing-bugs and smell-baseline adaptations
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:02:12 -06:00
Dark-Alex-17andSisyphus b5863dded0 feat: add a feedback-loop-first diagnosing-bugs skill to the coding suite
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:02:12 -06:00
Dark-Alex-17andSisyphus 03116d2f42 feat: add a Fowler code-smell baseline to the code-review skill
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 15:02:12 -06:00
Dark-Alex-17andSisyphus 9b23247815 feat: flag duplicate helpers in code reviews with a repo-wide DRY check
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 14:46:38 -06:00
Dark-Alex-17andSisyphus 3493b01e9a feat: add a transactional-integrity review skill to the code review gate
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 14:46:38 -06:00
Dark-Alex-17andSisyphus fd989c44d0 feat: add an operational-history prior-art lane to the code-reviewer agent
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-08-24 14:46:13 -06:00
Dark-Alex-17 e7307da6a9 feat: support name=value macro arguments with variable tab completion
Macro invocations (.name and .macro name) accept leading name=value
assignments before positional args: assignments set declared variables
directly so earlier variables can keep their defaults, remaining
positionals fill unassigned variables in declaration order, and the
free text after -- is never scanned for assignments. Identifier-shaped
keys that match no declared variable error with the declared list to
catch typos; non-identifier tokens containing = stay positional.
MacroVariable gains an optional description field, and tab completion
after a macro name offers name= candidates showing each variable's
description and default until the assignment prefix ends.
2026-08-24 14:20:21 -06:00
Dark-Alex-17 b6721d6a15 feat: add --help guides to the .install and .uninstall REPL commands
.install --help and .uninstall --help print a usage guide covering the
owner/repo shorthand, --git-host, --filter, --force, ref pinning, and
the bundle lifecycle; both usage error lines now point at --help. Tab
completion offers --help for both commands and --git-host on the first
.install argument, and the unified install parser accepts flags in any
argument position so completed flags work wherever they are inserted.
The empty .list bundles message now shows the REPL install form
alongside the CLI one.
2026-08-24 13:58:09 -06:00
Alex Clarke 9e72a52b1c Merge pull request #17 from Dark-Alex-17/feat/bundle-provenance
feat: bundle manifest, provenance, and lifecycle for shared configurations
2026-08-24 11:23:25 -06:00
Dark-Alex-17 30c1637dff refactor: Refactored some bundle const locations 2026-08-24 11:15:13 -06:00
Dark-Alex-17 6b5535956d fix: harden the bundle lifecycle per code review
The path-escape guard that uninstall applies to recorded paths now also
covers update's obsolete-file deletion through a shared check, so a
tampered store cannot turn either delete site into an arbitrary file
removal. Updates gain a working non-interactive path: --yes now applies
to --update-bundle (locally modified files, obsolete files, and modified
mcp entries are all kept; everything else refreshes), owned mcp entries
whose recorded hash still matches the local entry take the remote side
without prompting, and the non-TTY conflict bails name the flag that
actually works per surface. An update records its new commit and version
only after files and mcp entries land, so an aborted update cannot claim
content it never wrote. The store gains a version field and rejects
stores from newer builds, the corrupt-store error no longer advises the
removal that would forfeit ownership tracking, and duplicate records
tracking one source abort a rename instead of overwriting a record.
Reinstalling from a source URL reclassifies owned unmodified files as
silent refreshes just like updates. git runs with GIT_TERMINAL_PROMPT=0
and a null stdin so private or mistyped URLs fail instead of hanging.
File comparison fills buffers fully before comparing, deleting an
obsolete file prunes emptied directories, mcp.json backfill uses the
fsynced atomic writer, --list-bundles no longer triggers builtin
backfill, bundle-name completion logs store errors instead of swallowing
them and offers --yes, and REPL .uninstall rejects unknown flags.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 fdfe4ba023 refactor!: drop the .install remote migration hint
'remote' is no longer special-cased anywhere; the token falls through
to the unified .install dispatch like any other value.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 8f02bf1c33 refactor!: drop the --install-from tombstone entirely
The flag no longer exists in any form; --install <GIT_URL|OWNER/REPO>
is the only spelling.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 80b082423c fix: address code review findings on the bundle lifecycle
The user-origin marker on replaced mcp.json entries is now sticky:
re-records and cross-bundle transfers only upgrade replaced to
transferred when the prior record proves bundle origin, so updating a
bundle can no longer make uninstall delete a key the user had before the
bundle replaced it. Canonical source URLs lowercase only the host, since
self-hosted forges treat repository paths as case-sensitive and
collapsing distinct repos misdirects updates and uninstalls. git clone
invocations pass '--' before the URL so a crafted source cannot be
parsed as a git flag. Lifecycle flags (--install, --install-builtins,
--update-bundle, --uninstall) and their companions now conflict
explicitly instead of first-match dispatch silently dropping actions.
--install-from returns as a hidden tombstone that errors with the
replacement instead of feeding the flag to the LLM as prompt text.
--list-bundles dispatches before config load so a pure read no longer
boots MCP servers. write_file_atomic fsyncs before the rename so a crash
cannot persist a truncated store. REPL: .uninstall accepts --yes,
.install rejects trailing tokens after a category, and .install remote
gets a migration hint. Plus polish: host validation rejects '#' and '?',
renamed_to no longer serializes null, derived names get a debug assert
against the validator, completions share DEFAULT_GIT_HOST, README
mentions skills.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 4324d551d6 fix: reserve category names, confirm fork-name collisions, report secrets on uninstall
Bundle names that collide with an asset category (agents, roles, skills,
macros, functions, mcp_config) are now owner-qualified at install time,
whether derived from the repo or declared by a manifest, so no bundle can
shadow a category by name. A manifest name that collides with a bundle
from a different source now prompts for confirmation interactively (a
fork or typo-squat is the likely cause); declining aborts before anything
is written, and non-interactive runs keep the deterministic
owner-qualification. Uninstall summaries now list the vault secrets the
bundle's MCP servers reference, noting they are installed by the bundle
but not removed. Also removes the dead ResolvedBundleName.migrated_from
field.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 1136b1385b docs: removed extra fluff on bundles from the main README 2026-08-24 11:15:13 -06:00
Dark-Alex-17 84b90bfe26 style: remove em-dashes from comments and the uninstall selector 2026-08-24 11:15:13 -06:00
Dark-Alex-17 53ccbda97c feat: expand owner/repo shorthand for --install with a --git-host flag
--install someuser/repo expands to https://github.com/someuser/repo;
--git-host overrides the default host and forces source interpretation
even when the value matches an installed bundle name. Two or more path
segments are accepted so nested GitLab-style groups work, and #ref
pinning applies to shorthand values. --uninstall resolves owner/repo
against recorded sources: a single match uninstalls, multiple matches
prompt an interactive selection showing each bundle's source, and
non-interactive runs bail instead of guessing.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 2d1bf372d8 style: strip narration comments from bundle provenance code
Function docs that restated behavior already evident from names,
signatures, and code are removed; only comments carrying invariants
the code cannot express remain.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 4987d850f9 feat!: remove the deprecated --install-from flag and .install remote form
--install <git-url|name> is the single entry point for remote installs
and updates; the unified .install dispatch likewise replaces
.install remote. Flag completion for .install now applies to the
unified form.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 9541a094d8 fix: make bundle provenance portable to Windows
Provenance records stored OS-native path separators, making
installed-bundles.yaml non-portable; slug derivation treated a Windows
drive letter as an scp host and swallowed the whole path into one
sanitized segment. Store paths are now always forward-slashed and
backslashes normalize before URL parsing. Test fixture repos commit a
'* -text' .gitattributes so clone-side autocrlf cannot rewrite content
assertions.
2026-08-24 11:15:13 -06:00
Dark-Alex-17 89df8ec1ca docs: document bundle lifecycle and manifest for sharing configurations 2026-08-24 11:15:13 -06:00
Dark-Alex-17 bca85a4017 feat: rename install flags and unify .install dispatch
--install now takes a git URL or an installed bundle name: categories
are redirected to the new --install-builtins, installed names become
implicit updates, and source-shaped values install remotely. The old
--install-from keeps its exact behavior as a hidden deprecated alias.
The REPL's .install gains the same unified dispatch while keeping
.install <category> and .install remote <url> back-compat.
2026-08-24 11:15:12 -06:00
Dark-Alex-17 0e5d85f2ff feat: add --uninstall and .uninstall for installed bundles 2026-08-24 11:15:12 -06:00
Dark-Alex-17 0a806da8d2 feat: add --update-bundle with provenance-aware conflict handling
Updates re-clone a bundle's recorded source (honoring a recorded commit
pin unless a #<ref> override moves it), silently refresh files the bundle
owns that the user never modified, and fall back to the normal conflict
prompts for modified or unowned files. Files the remote no longer ships
are offered for deletion (kept by default non-interactively, staying
owned). The record is refreshed with the new commit, version, and
metadata, and stamped with an updated_at timestamp on success.
2026-08-24 11:15:12 -06:00
Dark-Alex-17 2790a823b0 feat: add --list-bundles and .list bundles with drift detection 2026-08-24 11:15:12 -06:00
Dark-Alex-17 88acf2362f feat: record bundle provenance when installing from remote repos 2026-08-24 11:15:12 -06:00
Dark-Alex-17 bfcc762ec9 feat: add bundle provenance store 2026-08-24 11:15:12 -06:00
Dark-Alex-17 b21699b749 feat: parse bundle manifests and capture resolved SHAs for remote installs 2026-08-24 11:15:12 -06:00
Dark-Alex-17 5f23e2403f fix: user__ask should have been renamed to user__select in graph agent user interaction invocations 2026-08-21 14:49:27 -06:00
Dark-Alex-17 2af6fe64d4 Merge branch 'feat/comment-discipline' 2026-08-21 14:02:55 -06:00
Dark-Alex-17 da640f3dcd feat: created a new comment-discipline skill for the built in sisyphus suite 2026-08-21 14:02:39 -06:00
Dark-Alex-17 79ec2d87c7 fix: support tab completions for graph-based agents with variables as well as standard agents 2026-08-21 12:47:38 -06:00
Dark-Alex-17 faf9dd581f feat: created a dedicated git_command tool to tighten tool calling permissions in the git-master skill 2026-08-21 12:33:28 -06:00
Dark-Alex-17 873deef7c7 feat: Added a new security review step to the code writing quality gates 2026-08-21 12:30:53 -06:00
Alex Clarke f518c8a6fc Merge pull request #16 from Dark-Alex-17/feat/macros-as-commands
feat: macros as first-class custom commands
2026-08-21 12:29:24 -06:00
Dark-Alex-17 aea5f3d615 test: updated embedded macro tests to expect descriptions for all built-in macros 2026-08-21 12:22:48 -06:00
Dark-Alex-17 bda37d9f38 docs: Added a description to the built-in generate-commit-message macro 2026-08-21 12:15:31 -06:00
Dark-Alex-17 96e5390621 test: use collision-proof temp dirs in macro_policy tests
The with_macro_dirs fixture derived its temp-dir name from a wall-clock
nanosecond timestamp, so parallel tests starting in the same clock tick
shared a directory and saw each other's macro files (flaky on CI
runners with coarse tick granularity). A process id + atomic counter
makes the name unique by construction.
2026-08-21 12:03:11 -06:00
Dark-Alex-17 b43acac8ee fix: surface macro parse errors on top-level invocation
An invalid installed macro invoked as a top-level command fell through
to the generic unknown-command error, while .macro <name> reported the
parse/validation failure. Both paths now surface the reason.
2026-08-21 11:55:14 -06:00
Dark-Alex-17 03687c8981 feat: addressed review comments 2026-08-21 11:55:14 -06:00
Dark-Alex-17 95dd31e24b docs: cleaned up docs 2026-08-21 11:55:14 -06:00
Dark-Alex-17 e1b5562888 refactor: render .list agents and .list skills as comfy-tables
Long agent/skill descriptions wrapped badly in the bullet-list format.
Extract a shared asset_table helper (UTF8_FULL + dynamic arrangement,
same style as the markdown renderer and .list macros) and use it for
the agents, skills, and macros listings. The skills loaded marker
keeps its color; comfy-table's custom_styling feature accounts for
ANSI sequences in column widths.
2026-08-21 11:55:14 -06:00
Dark-Alex-17 fba040c668 fix: render .list macros as a comfy-table instead of fixed-width columns
Hand-rolled {:<24} padding broke alignment as soon as a macro name
exceeded the column width. Reuse the comfy-table UTF8_FULL preset with
dynamic content arrangement, matching the markdown renderer's tables.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 f61a8f7afd style: updated styles across macro implementation 2026-08-21 11:55:13 -06:00
Dark-Alex-17 91328ca7e1 docs: document macros as first-class custom commands
Covers top-level .name invocation, the new description/isolated macro
fields (with the non-isolation persistence, fail-fast, nested-macro, and
.exit caveats), workspace .coyote/macros/ + --no-workspace-macros,
enabled_macros scoping at global/role/agent/session levels, and
.macro enable|disable across the README and every example config.
graph.example.yaml gains a note that enabled_macros is ignored in graph
configs.

CHANGELOG intentionally untouched: it is generated by commitizen at
release time from the conventional commit subjects.

Implements plans/custom-commands-design.md §8.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 125360033d docs(plans): record implementation-verified correction from T6
no_workspace_mcp (and therefore no_workspace_macros, per the exact-mirror
ruling) is CLI-flag-only: AppConfig field exists but there is no
Config-struct key and no env arm, so a config.yaml entry is non-functional.
Pre-existing issue: config.example.yaml:140 documents the dead
no_workspace_mcp key — recorded as follow-up, out of scope.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 4323d4823c feat: add --no-workspace-macros opt-out for workspace macro loading
Mirrors --no-workspace-mcp exactly: a CLI-only flag backed by an
AppConfig field (default false) that disables .coyote/macros in both
the resolved macro policy and Macro::load's workspace-then-global
preference, so the two always agree (custom-commands design §5).
2026-08-21 11:55:13 -06:00
Dark-Alex-17 5478c5a239 feat: execute non-isolated macros on the live REPL context
Make the macro isolated field live (design §3, §9 step 5):
isolated: false now runs the interpolated steps via run_repl_command
on the live RequestContext — session-recorded, conversation-visible,
with mutating steps persisting by design — while isolated: true keeps
the forked execution byte-for-byte unchanged.

- Add macro_non_isolated companion field beside macro_flag; both
  fork-propagation sites mirror it verbatim.
- RAII MacroModeGuard wraps the whole &mut RequestContext (DerefMut
  passthrough) and restores flag+mode on every exit path, including a
  failing step; steps remain fail-fast.
- Reject nested macro invocation when the current mode is non-isolated
  ("nested macros not allowed in non-isolated mode"); an isolated
  macro's step may still run a non-isolated macro inline on its fork.
- use_agent now suppresses the agent's default session only for
  isolated macros; a non-isolated .agent step engages it as if typed.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 e94bd450cd feat: surface macros as first-class custom commands in the REPL
Implements the invocation and management surfaces from
plans/custom-commands-design.md §4 and §6:

- Top-level dispatch: an enabled macro <name> now runs as ".<name> [args]"
  from the command catch-all; runtime-disabled macros point at
  ".macro enable <name>", locked macros name the owning config, and
  unknown commands keep the existing error verbatim
- .macro enable|disable <name>: runtime toggles over the in-memory
  global-level enabled_macros list (disable with no list materializes
  all-active-minus-name); toggles error when a role/agent/session
  allowlist owns the field
- .set enabled_macros <csv|null> with workspace-then-global existence
  validation; .set key completion gains enabled_macros and the
  previously missing enabled_skills
- Dynamic completion: enabled macros (with descriptions) join built-ins
  on ".<TAB>" without touching the static command registry;
  ".macro <TAB>" lists invocable macros (incl. built-in-shadowed ones)
  plus the enable/disable subcommands; second-arg completion offers
  toggle-eligible names
- .list macros: enriched table (name, source, isolated, state,
  description) covering every resolver state incl. missing and
  shadowed rows; .help gains a custom-commands section
- Session info/render and sysinfo display enabled_macros; Macro::load
  resolves workspace-then-global; enable/disable rejected as macro
  names in the creator
2026-08-21 11:55:13 -06:00
Dark-Alex-17 e8ddb61518 feat: add lazy macro resolver with two-dir discovery and per-macro states
Adds src/config/macro_policy.rs: MacroPolicy::effective computes the
visible macro set on demand from the discovered definition files, the
four-level enabled_macros allowlists, and the built-in command names.

- Discovery scans workspace (.coyote/macros/) then global macros dirs on
  every resolution; workspace shadows global by name, and the shadowed
  global entry is retained and flagged so both stay listable (plan
  custom-commands-design.md §5). Workspace scanning is gated on a bool
  parameter so the future --no-workspace-macros flag wires in one line.
- Allowlist precedence is session > agent > role > global, first Some
  wins, no merging; None falls through, an empty list is an explicit
  zero, all-None enables everything (mirrors SkillPolicy).
- Per-macro states per plan §6: enabled, disabled (runtime, global-level
  exclusions only), locked (role/agent/session exclusions, recording the
  owning level), missing (unknown allowlist names warn instead of
  bailing — deliberate divergence from skills), shadowed (built-in name
  collisions), and invalid (parse failures and the reserved names
  enable/disable). Invalid beats allowlist exclusion beats shadowing.
- Adds enabled_macros() accessors on Role, Session, and Agent alongside
  their enabled_skills() counterparts, plus paths::workspace_macros_dir.
- 37 tests: state matrix, pairwise precedence, explicit-zero pinned at
  every level, workspace shadowing, reserved names, builtin collisions,
  missing rows, invalid YAML, and env-gated discovery (#[serial]).
2026-08-21 11:55:13 -06:00
Dark-Alex-17 f39381aa9d feat: add enabled_macros config field at global, role, agent, and session levels
Mirrors the enabled_skills plumbing per plans/custom-commands-design.md §5:
- global: Config + AppConfig structs, from_config copy, and the
  COYOTE_ENABLED_MACROS env-override arm (csv_to_vec parsing)
- role: frontmatter via parse_string_or_array (list or csv string),
  plus the export() mirror so Role::save round-trips the field
- agent (non-graph): plain serde on AgentConfig; graph.yaml silently
  ignores the key (pinned by test, no field on Graph by design)
- session: plain serde with csv-or-vec deserializer

Empty list/string deserializes to Some([]) (explicit zero), distinct
from absent/null (None) — pinned by tests at every level, including
the env arm (serial-fenced against the from_config tests, which read
the process env via load_envs).
2026-08-21 11:55:13 -06:00
Dark-Alex-17 e8b55bba15 feat: add description and isolated fields to Macro struct
description (optional, default None) will surface in listings and
completion; isolated (default true) preserves today's forked-context
execution behavior exactly. Both fields use plain serde defaults so
every existing macro YAML deserializes unchanged, and unknown fields
in newer files remain tolerated by older binaries.

Adds Serialize to Macro/MacroVariable (None description skipped) and
back-compat, round-trip, and embedded-asset deserialization tests.

Per plans/custom-commands-design.md §3 / §9 step 1.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 9863c7a5f3 docs(plans): add macros-as-custom-commands design (gate-approved)
Design doc for macros as first-class custom commands: top-level .name
invocation, description/isolated fields, enabled_macros scoping,
workspace-local macros, completion integration.

Gatekeeper: sealed (3 findings fixed). Oracle plan-review: approved.
2026-08-21 11:55:13 -06:00
Dark-Alex-17 f44722df04 lint: Removed accidental session file
CI / All (ubuntu-latest) (push) Failing after 32s
CI / All (macos-latest) (push) Canceled after 0s
CI / All (windows-latest) (push) Canceled after 0s
2026-08-19 12:59:50 -06:00
Dark-Alex-17 3eaae0e652 feat: Dynamically detect RAG embedding model dimension for any given model 2026-08-18 19:57:11 -06:00
Dark-Alex-17 b12829db39 fix: drain crossterm characters in zellij in kitty contexts to prevent DA1 responses from entering prompt 2026-08-18 15:30:07 -06:00
Dark-Alex-17 dffaf6b9db fix: drain tty input when displaying inquire prompts to prevent unintentional escapes 2026-08-18 15:10:30 -06:00
Dark-Alex-17 a9a4ccca88 fix: breaking iwe MCP server changes with newest version 2026-08-18 11:29:36 -06:00
Dark-Alex-17 7cb7d66575 fix: improved handling of non service-specific secrets using sbx custom-secrets
CI / All (ubuntu-latest) (push) Failing after 30s
CI / All (macos-latest) (push) Canceled after 0s
CI / All (windows-latest) (push) Canceled after 0s
2026-08-17 18:01:21 -06:00
Dark-Alex-17 400b50fbd0 feat: Support auto confirmation for gatekeeper agents 2026-08-17 15:48:36 -06:00
Dark-Alex-17 7eeff2a226 fix: include tool output to LLM_OUTPUT in errors as well as stderr 2026-08-17 11:38:21 -06:00
Dark-Alex-17 374e8c0bf7 fix: prevent tmp-overwriting
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Canceled after 0s
CI / All (windows-latest) (push) Canceled after 0s
2026-08-15 19:35:12 -06:00
Dark-Alex-17 ede87df960 fix: Corrected a rare edge case on how tool files are generated during parallel executions 2026-08-15 18:41:13 -06:00
Dark-Alex-17 95a8c3df44 fix: cosmetic fix after improved bash tool handling 2026-08-15 17:31:32 -06:00
Dark-Alex-17 f57bd21ee4 fix: Improved subagent escalation handling 2026-08-14 16:53:37 -06:00
Dark-Alex-17 e1b6e3f8c6 fix: latent parsing bugs in fs_patch and argc 2026-08-14 16:12:41 -06:00
Dark-Alex-17 644d899f78 fix: Prevent infinite hangs in coder agent and implement timeouts for LLM API calls and interactive tools 2026-08-14 15:22:25 -06:00
Dark-Alex-17 2596194417 feat: retry LLM API calls once after 401 by force-refreshing the OAuth token
The Client trait's default chat_completions, chat_completions_streaming,
and embeddings methods now classify failures via ApiStatusError: on a
401 with a cached OAuth token, the token is distrusted (identity-aware
marker) and the call retried exactly once — the retry's prepare step
sees the marker and force-refreshes. Streaming retries only while the
SSE handler has received no content, preventing duplicate rendering.
A second 401 propagates the original error; other retry errors
propagate as-is. API-key clients never retry. No backoff by design:
cost is bounded to one refresh + one retry per failing request.
2026-08-14 13:17:56 -06:00
Dark-Alex-17 684f19250a feat: identity-aware rejected-token marker for LLM OAuth cache
distrust_access_token compare-and-invalidates the in-memory entry only
when the cached token equals the rejected one, so a concurrent refresh
is never clobbered. is_valid_access_token and both expiry checks in
prepare_oauth_access_token treat marked tokens as expired, forcing a
refresh of provider-rejected tokens that are still locally unexpired.
The marker is cleared after every completed refresh, including ones
that return the same token.
2026-08-14 13:07:51 -06:00
Dark-Alex-17 f3d59ade11 feat: typed ApiStatusError carrying HTTP status through LLM client errors
catch_error and sse_stream now bail with ApiStatusError{status, message}
instead of bare anyhow strings, preserving every existing Display output
byte-for-byte. Enables structural status classification (e.g. 401
detection) via downcast through anyhow context chains.
2026-08-14 13:01:08 -06:00
Dark-Alex-17 e1604c58ea feat: per-request OAuth token injection with mid-session refresh for HTTP MCP servers
Replace the spawn-time static Authorization header for OAuth-managed HTTP
MCP servers with McpOAuthClient, a custom implementation of rmcp's
StreamableHttpClient trait that resolves the bearer token on every
request via load_or_refresh_mcp_token. Tokens that expire mid-session
now refresh transparently instead of failing tool calls until restart.

On a 401 for an injected token, the wrapper force-refreshes (identity-
aware: a still-unexpired copy of the rejected token is not trusted) and
retries exactly once, matching Claude Code / official SDK semantics.
Both *_with_max_sse_event_size trait methods are overridden to preserve
the inner client's SSE size enforcement, and the inner reqwest client
mirrors rmcp's default (pool_max_idle_per_host(0), no redirects).

SSE, stdio, and static-header HTTP paths are unchanged; startup
warning semantics (McpAuthRequired reasons) are preserved. Verified
live: mid-session backdated token refreshed transparently during an
active atlassian session.
2026-08-14 12:33:46 -06:00
Dark-Alex-17 dcacb3a962 chore: upgrade rmcp 1.8.0 -> 3.1.2
Compile-clean upgrade verified: zero source changes needed, full test
suite green, clippy clean. Coyote's rmcp API surface (14 items) dodges
all 2.0/3.0 breaking changes; the "Auth required" error string matched
by is_auth_required_error is intact in 3.1.2.
2026-08-14 11:10:45 -06:00
Dark-Alex-17 d791098e51 feat: reason-specific warnings for MCP servers that fail OAuth at startup
Distinguish why an OAuth MCP server was not started: never authenticated
(no stored credentials), stored token expired and refresh failed, or the
server rejected a token that looked valid. McpTokenStatus replaces the
Option<String> return of load_or_refresh_mcp_token, and McpAuthRequired
carries the reason across the error boundary via anyhow context.
2026-08-14 11:07:56 -06:00
Dark-Alex-17 d31110cd67 fix: allow nested italics inside bold spans in markdown renderer
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-13 16:18:50 -06:00
Dark-Alex-17 0f35e03a85 fix: properly handle OAuth refreshes
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-13 16:00:42 -06:00
Dark-Alex-17 c2b0c120d7 chore: Added grok4.6 to models.yaml
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-13 14:20:35 -06:00
Dark-Alex-17 65c9be36b2 fix: correct newline removal from fs_write and fs_patch 2026-08-13 14:18:45 -06:00
83 changed files with 19355 additions and 945 deletions
+4 -1
View File
@@ -5,4 +5,7 @@
.idea/ .idea/
/coyote.iml /coyote.iml
/.idea/ /.idea/
.coyote .coyote/**
.sisyphus/**
.coyote-project.json
.coyote/memory/
+11
View File
@@ -1,5 +1,16 @@
# Credits # Credits
## Matt Pocock's Skills
The bundled `diagnosing-bugs`, `codebase-design`, and `grilling` skills, the
`architecture-reviewer` agent, and the code smell baseline in the bundled
`code-review` skill are adapted from
[mattpocock/skills](https://github.com/mattpocock/skills) by Matt Pocock,
licensed under the MIT License. The smell definitions trace back to Martin
Fowler's *Refactoring* (ch. 3); the deep-module vocabulary builds on John
Ousterhout's *A Philosophy of Software Design* and Michael Feathers'
*Working Effectively with Legacy Code*.
## AIChat ## AIChat
Coyote originally started as a fork of the fantastic Coyote originally started as a fork of the fantastic
[AIChat CLI](https://github.com/sigoden/aichat). The initial goal was simply [AIChat CLI](https://github.com/sigoden/aichat). The initial goal was simply
Generated
+43 -8
View File
@@ -1962,6 +1962,16 @@ dependencies = [
"darling_macro 0.23.0", "darling_macro 0.23.0",
] ]
[[package]]
name = "darling"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "88490bf1b990d87eaaa7ac8aa887f629a08e7359765b4911faf63c3763347d23"
dependencies = [
"darling_core 0.24.0",
"darling_macro 0.24.0",
]
[[package]] [[package]]
name = "darling_core" name = "darling_core"
version = "0.20.11" version = "0.20.11"
@@ -1989,6 +1999,19 @@ dependencies = [
"syn 2.0.119", "syn 2.0.119",
] ]
[[package]]
name = "darling_core"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "084e274f91c482280130e1e34e0b8d6e66776a060d7b6de7b84289ca778868c4"
dependencies = [
"ident_case",
"proc-macro2",
"quote",
"strsim",
"syn 3.0.3",
]
[[package]] [[package]]
name = "darling_macro" name = "darling_macro"
version = "0.20.11" version = "0.20.11"
@@ -2011,6 +2034,17 @@ dependencies = [
"syn 2.0.119", "syn 2.0.119",
] ]
[[package]]
name = "darling_macro"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68f5792fa0d41cd2325ce0ffa64f0a340eaebd4971a3a0c5e1ffd2cc488a355e"
dependencies = [
"darling_core 0.24.0",
"quote",
"syn 3.0.3",
]
[[package]] [[package]]
name = "defmt" name = "defmt"
version = "1.1.1" version = "1.1.1"
@@ -5303,12 +5337,12 @@ dependencies = [
[[package]] [[package]]
name = "rmcp" name = "rmcp"
version = "1.8.0" version = "3.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d1f571c72940a19d9532fe52dbea8bc9912bf1d766c2970bb824056b86f3f59" checksum = "c8dddc5b1924b9a59fba420166160ca2c4663a4e01803e52eda33070f56d63c8"
dependencies = [ dependencies = [
"async-trait", "base64 0.23.1",
"base64 0.22.1", "bytes",
"chrono", "chrono",
"futures", "futures",
"http 1.5.0", "http 1.5.0",
@@ -5326,19 +5360,20 @@ dependencies = [
"tokio-stream", "tokio-stream",
"tokio-util", "tokio-util",
"tracing", "tracing",
"uuid",
] ]
[[package]] [[package]]
name = "rmcp-macros" name = "rmcp-macros"
version = "1.8.0" version = "3.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1aad0035b69380782d78ea95b508327e6deaa2235909053e596eea8f27b5e1d5" checksum = "6898e24cd16342b59bfa8a53c2c04b9cf62fc8a2cfea57b9c038b09984bfc521"
dependencies = [ dependencies = [
"darling 0.23.0", "darling 0.24.0",
"proc-macro2", "proc-macro2",
"quote", "quote",
"serde_json", "serde_json",
"syn 2.0.119", "syn 3.0.3",
] ]
[[package]] [[package]]
+2 -1
View File
@@ -84,7 +84,7 @@ duct = "1.0.0"
argc = "1.23.0" argc = "1.23.0"
strum_macros = "0.27.2" strum_macros = "0.27.2"
indoc = "2.0.6" indoc = "2.0.6"
rmcp = { version = "1.5.0", features = [ rmcp = { version = "3.1.2", features = [
"client", "client",
"transport-child-process", "transport-child-process",
"transport-streamable-http-client-reqwest", "transport-streamable-http-client-reqwest",
@@ -141,6 +141,7 @@ arboard = { version = "3.3.0", default-features = false }
[dev-dependencies] [dev-dependencies]
pretty_assertions = "1.4.0" pretty_assertions = "1.4.0"
rmcp = { version = "3.1.2", features = ["server"] }
serial_test = "3" serial_test = "3"
[[bin]] [[bin]]
+9 -3
View File
@@ -23,7 +23,7 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g
* [AIChat Migration Guide](https://github.com/Dark-Alex-17/coyote/wiki/AIChat-Migration): Coming from AIChat? Follow the migration guide to get started. * [AIChat Migration Guide](https://github.com/Dark-Alex-17/coyote/wiki/AIChat-Migration): Coming from AIChat? Follow the migration guide to get started.
* [Installation](#install): Install Coyote * [Installation](#install): Install Coyote
* [Getting Started](#getting-started): Get started with Coyote by doing first-run setup steps. * [Getting Started](#getting-started): Get started with Coyote by doing first-run setup steps.
* [Sharing Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Sharing-Configurations): Install bundles of agents, roles, macros, tools, and MCP servers from any git repo, and share your own. * [Sharing Configurations](https://github.com/Dark-Alex-17/coyote/wiki/Sharing-Configurations): Install bundles of agents, roles, skills, macros, tools, and MCP servers from any git repo, and share your own. Bundles are Coyote's equivalent of plugins in other CLI agents.
* [REPL](https://github.com/Dark-Alex-17/coyote/wiki/REPL): Interactive Read-Eval-Print Loop for conversational interactions with LLMs and Coyote. * [REPL](https://github.com/Dark-Alex-17/coyote/wiki/REPL): Interactive Read-Eval-Print Loop for conversational interactions with LLMs and Coyote.
* [Custom REPL Prompt](https://github.com/Dark-Alex-17/coyote/wiki/REPL-Prompt): Customize the REPL prompt to provide useful contextual information. * [Custom REPL Prompt](https://github.com/Dark-Alex-17/coyote/wiki/REPL-Prompt): Customize the REPL prompt to provide useful contextual information.
* [Vault](https://github.com/Dark-Alex-17/coyote/wiki/Vault): Securely store and manage sensitive information such as API keys and credentials. * [Vault](https://github.com/Dark-Alex-17/coyote/wiki/Vault): Securely store and manage sensitive information such as API keys and credentials.
@@ -35,8 +35,14 @@ Coming from [AIChat](https://github.com/sigoden/aichat)? Follow the [migration g
* [Create Custom TypeScript Tools](https://github.com/Dark-Alex-17/coyote/wiki/Custom-Tools#custom-typescript-based-tools) * [Create Custom TypeScript Tools](https://github.com/Dark-Alex-17/coyote/wiki/Custom-Tools#custom-typescript-based-tools)
* [Create Custom Bash Tools](https://github.com/Dark-Alex-17/coyote/wiki/Custom-Bash-Tools) * [Create Custom Bash Tools](https://github.com/Dark-Alex-17/coyote/wiki/Custom-Bash-Tools)
* [Bash Prompt Utilities](https://github.com/Dark-Alex-17/coyote/wiki/Bash-Prompt-Helpers) * [Bash Prompt Utilities](https://github.com/Dark-Alex-17/coyote/wiki/Bash-Prompt-Helpers)
* [First-Class MCP Server Support](https://github.com/Dark-Alex-17/coyote/wiki/MCP-Servers): Easily connect and interact with MCP servers for advanced functionality. * [First-Class MCP Server Support](https://github.com/Dark-Alex-17/coyote/wiki/MCP-Servers): Easily connect and interact with MCP servers for advanced functionality. Coyote supports all three MCP capabilities: tools, resources, and prompts.
* [Macros](https://github.com/Dark-Alex-17/coyote/wiki/Macros): Automate repetitive tasks and workflows with Coyote "scripts" (macros). * Models interact with each server through a compact set of capability-gated meta-tools: `mcp_search`/`mcp_describe` for discovery across tools, resources, and prompts, `mcp_invoke` for tool calls, `mcp_read` for paged and regex-filterable resource reads, and `mcp_prompt` for server-defined prompts. Binary content is spilled to disk instead of inlined, and oversized tool results are bounded before they reach the model.
* Invoke server prompts yourself with `.prompt <server> <name> [key=value ...]` in the REPL, with live staged tab-completion (servers, then prompt names, then `key=` arguments), and discover them with `.list prompts`.
* [Macros](https://github.com/Dark-Alex-17/coyote/wiki/Macros): Automate repetitive tasks and workflows with Coyote "scripts" (macros). Macros are Coyote's custom commands: invoke any macro directly by name (e.g. `.review main`), with tab-completion, right alongside the built-in REPL commands.
* Give a macro a `description` (shown in `.list macros` and completions) and set `isolated: false` to run its steps on the live session, exactly as if you typed them. Note that non-isolated steps are recorded in the session, and mutating steps (`.role`, `.model`, ...) persist after the macro ends — by design. Steps are fail-fast: an error aborts the remaining steps, but completed steps' effects remain. A non-isolated macro step cannot invoke another macro, and a `.exit` step never exits the REPL.
* Commit project-specific macros to `.coyote/macros/` in your repo — they shadow same-named global macros (opt out with `--no-workspace-macros`).
* Pass variables positionally or by name: leading `name=value` args set declared variables directly (letting earlier variables keep their defaults), and remaining args fill the rest in order. Tab completion after a macro name lists each variable with its description and default.
* Scope which macros are invocable with `enabled_macros` in the global config, a role, an agent, or a session (most specific wins; an empty list disables all macros), and toggle at runtime with `.macro enable|disable <name>`.
* [RAG](https://github.com/Dark-Alex-17/coyote/wiki/RAG): Retrieval-Augmented Generation for enhanced information retrieval and generation. * [RAG](https://github.com/Dark-Alex-17/coyote/wiki/RAG): Retrieval-Augmented Generation for enhanced information retrieval and generation.
* [Sessions](https://github.com/Dark-Alex-17/coyote/wiki/Sessions): Manage and persist conversational contexts and settings across multiple interactions. * [Sessions](https://github.com/Dark-Alex-17/coyote/wiki/Sessions): Manage and persist conversational contexts and settings across multiple interactions.
* [Memory](https://github.com/Dark-Alex-17/coyote/wiki/Memory): Persistent file-based memory that survives across sessions. Bootstrap with `coyote --init-memory [global|workspace]`. * [Memory](https://github.com/Dark-Alex-17/coyote/wiki/Memory): Persistent file-based memory that survives across sessions. Bootstrap with `coyote --init-memory [global|workspace]`.
+6 -2
View File
@@ -5,7 +5,7 @@ description: |
spawns one Sisyphus per task on a single run branch, verifies each with an adversarial spawns one Sisyphus per task on a single run branch, verifies each with an adversarial
plan-conformance check, and finishes with ONE draft PR (CI checks watched to green) plus tracked plan-conformance check, and finishes with ONE draft PR (CI checks watched to green) plus tracked
follow-up tasks. Task state lives on disk in a plans directory, so runs survive context compression. follow-up tasks. Task state lives on disk in a plans directory, so runs survive context compression.
version: 2.0.0 version: 2.1.0
agent_session: temp agent_session: temp
auto_continue: true auto_continue: true
max_auto_continues: 100 max_auto_continues: 100
@@ -27,6 +27,7 @@ summarization_threshold: 100000
skills_enabled: true skills_enabled: true
enabled_skills: enabled_skills:
- design-session - design-session
- grilling
- task-tracking - task-tracking
- plan-authoring - plan-authoring
- delegation-protocol - delegation-protocol
@@ -126,7 +127,10 @@ instructions: |
### Phase B — Design decomposition ### Phase B — Design decomposition
Load and follow the `design-session` skill against the design doc. This produces Load and follow the `design-session` skill against the design doc. When running
interactively, also load `grilling` and put the open design decisions to the user as
frontier rounds (numbered questions, each with a recommended answer) instead of ad-hoc
one-at-a-time questions. This produces
`{{plans_dir}}/PLAN-<slug>.md` with Problem, Scope, Approach, Alternatives, Constraints/risks, `{{plans_dir}}/PLAN-<slug>.md` with Problem, Scope, Approach, Alternatives, Constraints/risks,
Open questions, and a **Task breakdown** where **each task is sized to ~1 engineer-day** (decompose Open questions, and a **Task breakdown** where **each task is sized to ~1 engineer-day** (decompose
anything bigger NOW). anything bigger NOW).
@@ -0,0 +1,67 @@
# Architecture Reviewer
An **on-demand architecture improvement scout**. It scans a codebase for **deepening
opportunities** — refactors that turn shallow modules into deep ones — presents them as a visual
report, then refines the candidate you pick into a concrete, implementation-ready interface
proposal.
Two things it is deliberately **not**:
1. **Not a completion gate.** The review stack ([`code-reviewer`](../code-reviewer/README.md),
[`adversary`](../adversary/README.md), [`security-reviewer`](../security-reviewer/README.md))
judges *changes* before a task finishes. This agent is invoked on demand, when you want the
codebase itself made deeper, more testable, and easier to navigate. A "cleanup gate" would
produce noisy, opinionated churn on every diff; a cleanup *tool* produces focused proposals
when you ask for them.
2. **Not an implementer.** It proposes; you (or a `coder` you delegate to) implement. Its only
write is the report file in the OS temp directory — repository files are never touched.
## How it works
Driven by the [`codebase-design`](../../skills/codebase-design/SKILL.md) skill — the shared
deep-module vocabulary (**module**, **interface**, **depth**, **seam**, **adapter**, **leverage**,
**locality**) and its principles (the deletion test, "the interface is the test surface", "one
adapter = hypothetical seam, two = real").
1. **Scope by git history (YAGNI).** Deepening pays off where code keeps changing, so hot spots
from the commit log rank first — unless you name a direction.
2. **Explore for friction.** Fans out `explore` agents hunting shallow modules, leaked seams,
concept-bouncing, and code that's hard to test through its current interface; every suspect
gets the deletion test.
3. **Report candidates.** 3-6 cards (problem / solution / leverage-and-locality benefits /
before-after visual / `Strong`-`Worth exploring`-`Speculative` badge), as a self-contained
Tailwind+Mermaid HTML file in your temp dir (default) or inline markdown
(`report_format: markdown`). Ends with a top recommendation, then stops and asks which
candidate to pursue.
4. **Refine via design-it-twice.** For the chosen candidate: frame the constraints and dependency
categories, produce 2-3 radically different interface designs (optionally spawning `oracle`
for an independent alternative), compare on depth/locality/seam placement, and hand off ONE
opinionated, implementation-ready proposal including the testing strategy ("replace, don't
layer").
## Usage
```sh
# Scan the current repo, HTML report
coyote -a architecture-reviewer "Find deepening opportunities"
# Aim it at a pain point, inline report
coyote -a architecture-reviewer --agent-variable report_format markdown \
"The billing/entitlements code is painful to test - what should be deepened?"
```
Also spawnable from `sisyphus` when a request is explicitly architecture-scale ("improve the
architecture of X", "make this module easier to test").
## Related
- [`codebase-design`](../../skills/codebase-design/SKILL.md) — the vocabulary and principles it runs on.
- [`oracle`](../oracle/README.md) — advisory design review; also loads `codebase-design` for the shared vocabulary.
- [`explore`](../explore/README.md) — the codebase walkers it fans out.
## Credits
Adapted from the `codebase-design` and `improve-codebase-architecture` skills in
[mattpocock/skills](https://github.com/mattpocock/skills) (MIT), which build on ideas from John
Ousterhout's *A Philosophy of Software Design* and Michael Feathers' *Working Effectively with
Legacy Code*.
@@ -0,0 +1,158 @@
name: architecture-reviewer
description: On-demand architecture improvement scout - scans a codebase for deepening opportunities (shallow modules, leaked seams, missing locality) weighted by git-history hot spots, presents candidates as a visual report, then refines the chosen candidate into a concrete interface proposal via design-it-twice. Proposes, never implements. NOT a completion gate - invoke it when you want the codebase made deeper, more testable, and easier to navigate.
version: 1.1.0
agent_session: temp
auto_continue: true
max_auto_continues: 20
inject_todo_instructions: true
can_spawn_agents: true
spawnable_agents:
- explore
- oracle
max_concurrent_agents: 4
max_agent_depth: 2
inject_spawn_instructions: true
skills_enabled: true
enabled_skills:
- codebase-design
- delegation-protocol
- grilling
- parallel-research
variables:
- name: project_dir
description: Project directory to scan
default: '.'
- name: report_format
description: Candidate report format - 'html' (self-contained file in the OS temp dir, opened for the user) or 'markdown' (inline in chat)
default: html
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools:
- ast_grep.sh
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
- fs_glob.sh
- fs_ls.sh
- fs_write.sh
- execute_command.sh
instructions: |
You are an architecture improvement scout. You surface **deepening opportunities** — refactors
that turn shallow modules into deep ones — and refine the one the user picks into a concrete
interface proposal. The aim is testability, locality, and AI-navigability.
Two things you are NOT:
1. **Not a completion gate.** The review stack (`code-reviewer`/`adversary`/`security-reviewer`)
judges changes; you are invoked on demand to improve what already exists.
2. **Not an implementer.** You produce candidates and interface proposals; the user (or a coder
they delegate to) owns the code change. You never modify repository files — your only writes
are the report file in the OS temp directory.
## Step 0: Load the skill
Before anything else, `skill__load` `codebase-design`. It is your source of truth for the
vocabulary (**module**, **interface**, **depth**, **seam**, **adapter**, **leverage**,
**locality**), the principles (the deletion test, "the interface is the test surface", "one
adapter = hypothetical seam, two = real"), the dependency categories for safe deepening, and the
design-it-twice pattern. Use those terms EXACTLY in every finding — no "component", "service",
or "boundary". Load `delegation-protocol` and `parallel-research` before spawning sub-agents.
## Phase 1: Scope, then explore
**Scope before you scan — YAGNI.** Deepening pays off where code keeps changing:
- If the user named a direction (a module, subsystem, or pain point), take it and skip inference.
- Otherwise mine the history for hot spots:
`execute_command --command "git -C {{project_dir}} log --oneline --name-only -100"` (or
similar) and let the files that keep recurring pull your attention. Scattered changes with no
hot spot → widen the net.
Read the workspace instructions (`COYOTE.md`/`AGENTS.md`) if present — documented conventions and
recorded decisions are constraints, not candidates; don't re-litigate them.
Then spawn 1-3 `explore` agents (per `delegation-protocol`, in parallel per `parallel-research`)
to walk the scoped area. Brief them to report friction, not metrics:
- Where does understanding one concept require bouncing between many small modules?
- Where are modules shallow — an interface nearly as complex as the implementation?
- Where were pure functions extracted "for testability" while the real bugs hide in how they're
called (no locality)?
- Where do tightly-coupled modules leak across their seams?
- What is untested, or hard to test through its current interface?
Apply the **deletion test** yourself to every suspect the explorers return: would deleting it
concentrate complexity (real candidate) or just move it (pass-through)?
## Phase 2: Present candidates
Produce 3-6 candidates, each with:
- **Files**: the modules involved
- **Problem**: the friction the current shape causes, in skill vocabulary
- **Solution**: plain-English description of the deepening (no interface design yet)
- **Benefits**: stated as leverage and locality gains, and how tests improve
- **Recommendation strength**: `Strong` / `Worth exploring` / `Speculative`
- **Before/after sketch**: for `html`, a visual per candidate; for `markdown`, a compact
ASCII/mermaid sketch
**Report delivery** (per `report_format`, currently: {{report_format}}):
- `html` — write ONE self-contained file to the OS temp dir (`$TMPDIR`, falling back to `/tmp`)
named `architecture-review-<timestamp>.html`. Use Tailwind via CDN for layout and Mermaid via
CDN for graph-shaped structure (call graphs, dependencies); hand-built divs/SVG for editorial
visuals (mass diagrams, collapse animations). One card per candidate with a side-by-side
before/after diagram. Open it for the user (`open` on macOS, `xdg-open` on Linux, `start` on
Windows) and print the absolute path. Nothing lands in the repo.
- `markdown` — render the same cards inline in your response.
End the report with a **Top recommendation**: which candidate you'd tackle first and why.
Then STOP and ask which candidate to explore. Do NOT propose interfaces yet.
## Phase 3: Refine the chosen candidate
1. **Frame the problem space**: the constraints any new interface must satisfy, the dependencies
and their category (in-process / local-substitutable / remote-but-owned / true external, per
the skill), and a rough illustrative sketch to make the constraints concrete. Show the user.
When the candidate carries open decisions (what sits behind the seam, which callers to
optimise for, what tests must survive), load `grilling` and walk them as frontier rounds —
recommended answer per question, facts fetched by you, decisions made by the user.
2. **Design it twice**: produce 2-3 radically different interface designs per the skill's
pattern (different constraint each: minimal interface / maximal flexibility / optimise the
common caller). For a candidate worth the budget, spawn `oracle` to independently design or
critique one alternative. Each design: interface (with invariants, ordering, error modes),
caller example, what hides behind the seam, adapter strategy, trade-offs.
3. **Compare and recommend**: contrast on depth, locality, and seam placement; give ONE
opinionated recommendation or a justified hybrid.
4. **Hand off**: summarize the chosen design as an implementation-ready proposal — files to
change, the target interface, the testing strategy ("replace, don't layer": new tests at the
deepened interface, old shallow-module tests deleted). Note that implementation belongs to
the caller, not you.
## Rules
1. **Never modify repository files.** The temp-dir report is your only write.
2. **Skill vocabulary, exactly.** Findings that say "service" or "boundary" get rewritten.
3. **Friction over dogma.** A shallow module that never changes and confuses no one is not a
candidate. Recent-change hot spots rank first.
4. **Candidates are judgment calls.** Frame every problem as observed friction with evidence
(file:line, test absence, change-history churn), not as rule violations.
5. **Respect recorded decisions.** If a candidate contradicts a documented convention or
decision, surface it only when the friction justifies revisiting — and mark the conflict
clearly in the card.
## Context
- Project: {{project_dir}}
- Report format: {{report_format}}
- CWD: {{__cwd__}}
- Shell: {{__shell__}}
## Available Tools
{{__tools__}}
+25
View File
@@ -12,6 +12,31 @@ agents while handling coordination and final reporting.
- 🔄 **Cross-File Context**: Broadcasts sibling rosters so reviewers can alert each other about cross-cutting changes. - 🔄 **Cross-File Context**: Broadcasts sibling rosters so reviewers can alert each other about cross-cutting changes.
- 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels. - 📊 **Unified Reporting**: Synthesizes findings into a structured, easy-to-read summary with severity levels.
-**Parallel Execution**: Runs reviews concurrently for maximum speed. -**Parallel Execution**: Runs reviews concurrently for maximum speed.
- 🚨 **Operational History (optional)**: Checks the change against past production incidents via the [`incident-prior-art`](../../skills/incident-prior-art/SKILL.md) skill.
## Operational History Lane
Code review answers "is this code good?" — this lane answers "did we already get burned by this?"
When the diff touches operationally-relevant surface (error handling, retries, timeouts, alerting,
config controlling any of these), the orchestrator:
1. **Git archaeology** (always available): blames the lines the diff deletes or weakens. A guard
that originated in an incident-fix commit and is being removed is a 🔴 CRITICAL finding — the
change reintroduces a known production failure mode.
2. **Prior-art delegation** (opt-in): if the `prior_art_agent` variable names an agent that can
search your incident record (Slack, Jira, postmortems, handoff docs), it is spawned in REVIEW
MODE with symptom-vocabulary search keys extracted from the diff (error strings, metric/alert
names, config keys — the vocabulary operators actually use).
The lane is disabled by default (`prior_art_agent: ''`) and findings fold into the standard
severity taxonomy under an "Operational history" report section — no separate verdict. Wire it up
in a bundle or your local config:
```yaml
variables:
- name: prior_art_agent
default: 'oncall-historian' # any spawnable agent that can search your incident record
```
## Pro-Tip: Use an IDE MCP Server for Improved Performance ## Pro-Tip: Use an IDE MCP Server for Improved Performance
Many modern IDEs now include MCP servers that let LLMs perform operations within the IDE itself and use IDE tools. Using Many modern IDEs now include MCP servers that let LLMs perform operations within the IDE itself and use IDE tools. Using
+14 -5
View File
@@ -1,6 +1,6 @@
name: code-reviewer name: code-reviewer
description: CodeRabbit-style code reviewer - spawns per-file reviewers, synthesizes findings description: CodeRabbit-style code reviewer - spawns per-file reviewers, synthesizes findings
version: 2.0.0 version: 2.2.0
auto_continue: true auto_continue: true
max_auto_continues: 20 max_auto_continues: 20
@@ -14,11 +14,15 @@ skills_enabled: true
enabled_skills: enabled_skills:
- delegation-protocol - delegation-protocol
- parallel-research - parallel-research
- incident-prior-art
variables: variables:
- name: project_dir - name: project_dir
description: Project directory to review description: Project directory to review
default: '.' default: '.'
- name: prior_art_agent
description: Optional agent that can search the incident record (Slack/Jira/postmortems) for operational prior art. Empty disables the delegation lane; git archaeology still runs.
default: ''
- name: auto_confirm - name: auto_confirm
description: Auto-confirm command execution description: Auto-confirm command execution
default: '1' default: '1'
@@ -46,11 +50,12 @@ instructions: |
1. **Get the diff:** Run `get_diff` to get the git diff (defaults to staged changes, falls back to unstaged) 1. **Get the diff:** Run `get_diff` to get the git diff (defaults to staged changes, falls back to unstaged)
2. **Parse changed files:** Extract the list of files from the diff 2. **Parse changed files:** Extract the list of files from the diff
3. **Create todos:** One todo per phase (get diff, spawn reviewers, collect results, synthesize report) 3. **Create todos:** One todo per phase (get diff, spawn reviewers, operational-history lane, collect results, synthesize report)
4. **Spawn file-reviewers:** One `file-reviewer` agent per changed file, in parallel. Apply the `delegation-protocol` structured prompt format. 4. **Spawn file-reviewers:** One `file-reviewer` agent per changed file, in parallel. Apply the `delegation-protocol` structured prompt format.
5. **Broadcast sibling roster:** Send each file-reviewer a message with all sibling IDs and their file assignments 5. **Broadcast sibling roster:** Send each file-reviewer a message with all sibling IDs and their file assignments
6. **Collect all results:** Per `parallel-research`, do not poll. End your response after spawns + roster; the system will notify you when agents complete. 6. **Operational-history lane (conditional):** Load `incident-prior-art` and follow it. If the diff touches operationally-relevant surface (per the skill's trigger list), run its git-archaeology pass yourself, and — if `prior_art_agent` is set (currently: '{{prior_art_agent}}') — spawn that agent in REVIEW MODE alongside the file-reviewers using the skill's prompt template. If the surface is not operationally relevant, skip with a one-line note.
7. **Synthesize:** Combine all findings into a CodeRabbit-style report 7. **Collect all results:** Per `parallel-research`, do not poll. End your response after spawns + roster; the system will notify you when agents complete.
8. **Synthesize:** Combine all findings into a CodeRabbit-style report. Prior-art findings go under an "Operational history" section using the skill's severity folding (reintroduction of a past incident's failure mode = CRITICAL).
## Spawning File Reviewers ## Spawning File Reviewers
@@ -70,7 +75,8 @@ instructions: |
## MUST DO ## MUST DO
- Load `code-review` and `ai-slop-remover` skills before reading any code - Load `code-review` and `ai-slop-remover` skills before reading any code
- Apply both skill checklists to the diff - Load `transactional-integrity` as well if this file's diff touches state-changing code (DB writes, transactions, queue/webhook/job handlers, retries, external side effects)
- Apply all loaded skill checklists to the diff
- Use targeted fs_read with offset/limit; max 5 file reads - Use targeted fs_read with offset/limit; max 5 file reads
- End with REVIEW_COMPLETE - End with REVIEW_COMPLETE
@@ -138,6 +144,9 @@ instructions: |
## Cross-File Concerns ## Cross-File Concerns
<any cross-cutting issues identified by the teammate pattern> <any cross-cutting issues identified by the teammate pattern>
## Operational history
<only when the lane ran: archaeology + prior-art findings with incident/commit references, or "no relevant incident history found">
--- ---
*Reviewed N files, found X critical, Y warnings, Z suggestions, W nitpicks* *Reviewed N files, found X critical, Y warnings, Z suggestions, W nitpicks*
``` ```
+22 -5
View File
@@ -15,6 +15,8 @@ skills_enabled: true
enabled_skills: enabled_skills:
- ai-slop-remover - ai-slop-remover
- code-review - code-review
- comment-discipline
- diagnosing-bugs
- git-master - git-master
- frontend-ui-ux - frontend-ui-ux
- verification-gates - verification-gates
@@ -31,7 +33,7 @@ settings:
max_loop_iterations: 20 max_loop_iterations: 20
log_state_snapshots: true log_state_snapshots: true
validate_before_run: true validate_before_run: true
timeout: 1800 timeout: 14400
initial_state: initial_state:
project_dir: '' project_dir: ''
@@ -90,6 +92,7 @@ nodes:
Project directory: {{project_dir}} Project directory: {{project_dir}}
prompt: '{{initial_prompt}}' prompt: '{{initial_prompt}}'
tools: [] tools: []
timeout: 300
output_schema: output_schema:
type: object type: object
properties: properties:
@@ -166,6 +169,8 @@ nodes:
enabled_skills: enabled_skills:
- ai-slop-remover - ai-slop-remover
- code-review - code-review
- comment-discipline
- diagnosing-bugs
- git-master - git-master
- frontend-ui-ux - frontend-ui-ux
- verification-gates - verification-gates
@@ -176,9 +181,10 @@ nodes:
## Skills ## Skills
Use `skill__list` to see what's available, then `skill__load` the ones Use `skill__list` to see what's available, then `skill__load` the ones
that fit the work: `ai-slop-remover` always, `frontend-ui-ux` when that fit the work: `ai-slop-remover` and `comment-discipline` always,
touching UI, `git-master` when touching history, `verification-gates` `frontend-ui-ux` when touching UI, `git-master` when touching history,
to remember what evidence is required. Unload when a phase ends. `verification-gates` to remember what evidence is required. Unload when
a phase ends.
## Writing code ## Writing code
@@ -211,7 +217,11 @@ nodes:
Before writing ANY file: Before writing ANY file:
1. Find a similar existing file (grep, then read). 1. Find a similar existing file (grep, then read).
2. Match its style: imports, naming, structure, error handling. 2. Match its style: imports, naming, structure, error handling.
3. Follow the same patterns exactly. Do not invent new ones. 3. While reading it, note the repo's comment register per
`comment-discipline` (self-documenting / api-documented /
comment-heavy) and write comments to match. When the signal is
weak, write NO comment.
4. Follow the same patterns exactly. Do not invent new ones.
## Fix loop ## Fix loop
@@ -219,6 +229,11 @@ nodes:
the previous attempt failed verification. Read the error, identify the previous attempt failed verification. Read the error, identify
the minimal fix, apply it. Do not refactor while fixing. the minimal fix, apply it. Do not refactor while fixing.
If the fix is not obvious from the error, or a previous fix attempt
for the SAME failure did not stick, `skill__load` `diagnosing-bugs`
and follow it: build a red-capable reproduction loop before forming
any hypothesis. Do not spend a second attempt on a blind retry.
## Rules ## Rules
1. Match existing patterns - read examples first. 1. Match existing patterns - read examples first.
@@ -254,6 +269,7 @@ nodes:
- fs_patch - fs_patch
- execute_command - execute_command
max_iterations: 100 max_iterations: 100
timeout: 1800
state_updates: state_updates:
last_node_output: '{{output}}' last_node_output: '{{output}}'
fallback: end_failure fallback: end_failure
@@ -327,6 +343,7 @@ nodes:
- fs_ls - fs_ls
- execute_command - execute_command
max_iterations: 15 max_iterations: 15
timeout: 600
output_schema: output_schema:
type: object type: object
properties: properties:
+5 -2
View File
@@ -1,11 +1,12 @@
name: file-reviewer name: file-reviewer
description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns description: Reviews a single file's diff for bugs, style issues, and cross-cutting concerns
version: 2.0.0 version: 2.1.0
skills_enabled: true skills_enabled: true
enabled_skills: enabled_skills:
- code-review - code-review
- ai-slop-remover - ai-slop-remover
- transactional-integrity
variables: variables:
- name: project_dir - name: project_dir
@@ -29,7 +30,9 @@ instructions: |
Before reading any code, call `skill__load` for `code-review` and `ai-slop-remover`. They carry your detailed review methodology — the categories to check (correctness, tests, clarity, coupling, footguns), the investigation workflow (how to use the fs tools to build context before reviewing), the slop checklist (useless comments, dishonest naming, defensive handling of impossible cases), and the standard for when to flag vs. skip. Before reading any code, call `skill__load` for `code-review` and `ai-slop-remover`. They carry your detailed review methodology — the categories to check (correctness, tests, clarity, coupling, footguns), the investigation workflow (how to use the fs tools to build context before reviewing), the slop checklist (useless comments, dishonest naming, defensive handling of impossible cases), and the standard for when to flag vs. skip.
Apply BOTH checklists in every review. Skill bodies are your source of truth for what to flag; this agent's instructions handle workflow and output shape. Additionally load `transactional-integrity` when the diff touches state-changing code — database writes, transaction blocks, queue/webhook/job handlers, retry logic, or calls to external state-holding systems. It carries the atomicity/race/idempotency/dual-write checklist that generic correctness review misses. Skip it for pure reads, UI, and stateless computation.
Apply every loaded checklist in every review. Skill bodies are your source of truth for what to flag; this agent's instructions handle workflow and output shape.
## Your Mission ## Your Mission
+3
View File
@@ -14,6 +14,9 @@ variables:
- name: project_dir - name: project_dir
description: Absolute path to the project the plan targets - the ground truth for pointer verification description: Absolute path to the project the plan targets - the ground truth for pointer verification
default: '.' default: '.'
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools: global_tools:
- ast_grep.sh - ast_grep.sh
+3 -1
View File
@@ -1,11 +1,12 @@
name: oracle name: oracle
description: High-IQ advisor for architecture, debugging, and complex decisions. Blocking by design - the orchestrator is waiting on you. description: High-IQ advisor for architecture, debugging, and complex decisions. Blocking by design - the orchestrator is waiting on you.
version: 2.1.0 version: 2.2.0
skills_enabled: true skills_enabled: true
enabled_skills: enabled_skills:
- code-review - code-review
- ai-slop-remover - ai-slop-remover
- codebase-design
- plan-review - plan-review
- plan-authoring - plan-authoring
- iwe-knowledge-base - iwe-knowledge-base
@@ -61,6 +62,7 @@ instructions: |
- `skill__load code-review` — when reviewing a diff or existing code; gives you a focused review checklist. - `skill__load code-review` — when reviewing a diff or existing code; gives you a focused review checklist.
- `skill__load ai-slop-remover` — when judging code quality (especially for advising on cleanups). - `skill__load ai-slop-remover` — when judging code quality (especially for advising on cleanups).
- `skill__load codebase-design` — when advising on module/interface design, seam placement, testability, or refactoring structure; gives you the deep-module vocabulary (module, interface, depth, seam, adapter, leverage, locality) and its principles. Use those terms exactly.
- `skill__load plan-review` — when asked to review an implementation plan; adversarial checklist plus the PLAN_REVIEW verdict format. Load `plan-authoring` alongside it — it defines the plan schema you are checking against. - `skill__load plan-review` — when asked to review an implementation plan; adversarial checklist plus the PLAN_REVIEW verdict format. Load `plan-authoring` alongside it — it defines the plan schema you are checking against.
- `skill__load iwe-knowledge-base` — when the plans live in a large markdown corpus; navigate it structurally instead of globbing. - `skill__load iwe-knowledge-base` — when the plans live in a large markdown corpus; navigate it structurally instead of globbing.
+125
View File
@@ -0,0 +1,125 @@
# Security Reviewer
A **security analyst** for code changes. Where [`code-reviewer`](../code-reviewer/README.md) asks
*"is this code good?"* and [`adversary`](../adversary/README.md) asks *"is this the code the plan
asked for?"*, `security-reviewer` asks the third orthogonal question:
> **"Can this code be abused?"**
It traces untrusted data from sources (CLI args, HTTP input, file contents, LLM outputs) to
dangerous sinks (shell, SQL, file paths, deserializers, network) and hunts the classic classes:
injection, committed secrets, missing authn/authz, path traversal, SSRF, unsafe deserialization,
supply-chain hazards, weak crypto, and sensitive-data exposure.
## Why it's a third reviewer
| | `code-reviewer` | `adversary` | `security-reviewer` |
|---|---|---|---|
| Question | Is the code correct/clean? | Does the code match the plan? | Can the code be abused? |
| Unit of analysis | per-file diffs (fan-out) | criteria ↔ diff mapping | **data flows across files** |
| Blind spot it covers | slop, bugs, coupling | skipped criteria, scope drift | source→sink paths, secrets, authz gaps |
| Output | severity-tagged findings | `CONFORMS` / `DIVERGES` | `PASS` / `FAIL` (posture-gated) |
Security flaws live in the path between an input in one file and a sink in another —
exactly what a per-file review fans out past, and what acceptance criteria almost never mention.
## Posture-gated blocking
Not every project needs production strictness — a POC shouldn't be blocked on missing rate
limiting. The `security_posture` variable (or an explicit posture in the spawn prompt) sets the
blocking threshold:
| Posture | Blocks (FAIL) | Intended for |
|---|---|---|
| `prototype` | 🔴 Critical only | POCs, spikes, demos, localhost-only tools |
| `standard` (default) | 🔴 Critical + 🟠 High | Anything deployed, shared, or built upon |
| `hardened` | 🔴 + 🟠 + 🟡 Medium | Auth, payments, secrets handling, public-facing, multi-tenant |
Two invariants that do not bend with posture:
1. **Critical always blocks.** A committed secret is Critical in a prototype too — git history
outlives the prototype. Same for code that endangers the host machine or third-party systems.
2. **Posture gates the verdict, not the report.** Non-blocking findings are still listed; the
posture only decides PASS/FAIL.
Severity itself is calibrated by **reachability × blast radius**, not vulnerability class: SQL
injection in a localhost-only debug script is not High, and a "small" secret in a repo is Critical.
## Verdict (blocking on FAIL)
The agent ends every review with one sentinel:
```
SECURITY_REVIEW: PASS
Posture: standard. Findings: 0 critical, 0 high, 2 medium, 1 low (none at or above the blocking threshold).
```
```
SECURITY_REVIEW: FAIL
Posture: standard. Findings: 0 critical, 1 high, 1 medium, 0 low.
Blocking findings:
1. 🟠 Path traversal — export.rs:88 — 'name' from the HTTP body is joined into the output path with no canonicalization; '../../.ssh/authorized_keys' escapes the export root — canonicalize and verify the prefix before writing
Non-blocking findings:
1. 🟡 Sensitive data in logs — auth.rs:41 — bearer token logged at debug level — redact before logging
```
A `FAIL` verdict **blocks** completion, exactly like adversary's `DIVERGES`. The caller
(sisyphus/architect) resumes the SAME coder session with the blocking findings pasted verbatim,
then re-runs `security-reviewer` ONCE to confirm the fix.
Every finding cites `file:line` and articulates the concrete attack path. Vague findings are not
emitted.
## How it reviews
Driven by the [`security-review`](../../skills/security-review/SKILL.md) skill:
1. **Source→sink tracing** per hunk: where does untrusted data enter, what does it reach, and is
the mediation between them real (read the sanitizer, don't trust its name)?
2. **Ground-truth with read-only tools** (`fs_grep`/`fs_read`/`ast_grep`): confirm the vulnerable
path is reachable, confirm callers can deliver untrusted data, compare sibling code for the
security controls the new code should have mirrored.
3. **Posture gating**: severities assigned by exploitability, verdict decided by the threshold.
It is **read-only** — it produces a verdict, never a fix.
## Usage
Typically spawned by `sisyphus` alongside `code-reviewer`/`adversary`. The spawn prompt IS its
entire context, so include the diff (or a base ref), the posture, and any deployment context:
```sh
agent__spawn --agent security-reviewer --prompt "
## TASK
Security-review the recent changes. Return PASS/FAIL.
## POSTURE
standard # or: prototype (this is a throwaway POC) / hardened (this touches auth)
## DIFF
Run get_diff (or --base main), or: <paste diff>
## DEPLOYMENT CONTEXT
<what this code is for, who can reach it, whether it will be deployed/shared>
"
```
Direct invocation for ad-hoc use:
```sh
coyote -a security-reviewer --agent-variable security_posture prototype \
--agent-variable project_dir /path/to/repo \
"Review the staged changes. This is a localhost-only spike."
```
### Tools
- `get_diff [--base <ref>]` — staged → unstaged → `HEAD~1` fallback (or an explicit base/PR branch).
- `get_changed_files [--base <ref>]` — quick map of the attack surface.
- Plus read-only `fs_*` and `ast_grep` for ground-truth checks.
## Related
- [`security-review`](../../skills/security-review/SKILL.md) — the methodology it runs on.
- [`code-reviewer`](../code-reviewer/README.md) — the quality reviewer it runs alongside.
- [`adversary`](../adversary/README.md) — the plan-conformance reviewer it runs alongside.
+121
View File
@@ -0,0 +1,121 @@
name: security-reviewer
description: Security analyst - hunts exploitable flaws in a code change (injection, secrets, authz gaps, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Returns a posture-gated PASS/FAIL verdict so POCs aren't held to production strictness. Complements code-reviewer (quality) and adversary (plan conformance). Designed to be delegated to by sisyphus.
version: 1.0.0
auto_continue: true
max_auto_continues: 15
inject_todo_instructions: true
skills_enabled: true
enabled_skills:
- security-review
variables:
- name: project_dir
description: Project directory containing the changes under review
default: '.'
- name: security_posture
description: Blocking threshold - prototype (Critical only), standard (Critical+High), hardened (Critical+High+Medium)
default: standard
- name: auto_confirm
description: Auto-confirm command execution
default: '1'
global_tools:
- ast_grep.sh
- fs_read.sh
- fs_cat.sh
- fs_grep.sh
- fs_glob.sh
- fs_ls.sh
- execute_command.sh
instructions: |
You are a security reviewer. You answer ONE question: **can this code be abused?** You are NOT
the code-quality reviewer (that is `code-reviewer`/`file-reviewer`) and NOT the plan-conformance
reviewer (that is `adversary`). You hunt exploitable flaws in the CHANGE: injection, committed
secrets, missing auth, path traversal, SSRF, unsafe deserialization, supply-chain hazards.
Your value is attacker mindset applied to fresh code with zero stake in the implementation. The
implementer thought about the happy path; you think about the input that lies.
## Step 0: Load the skill
Before anything else, `skill__load` `security-review`. It carries your methodology: the
source-to-sink tracing discipline, the severity model (calibrated by reachability and blast
radius, not vulnerability class), the posture gating table, the hunt checklist, and the exact
verdict format. The skill body is your source of truth for HOW to review and WHAT blocks; these
instructions handle workflow and I/O.
## Input (the spawn prompt IS your entire context)
You are given:
1. **The diff** — pasted inline, or run `get_diff` (optionally `--base <ref>`) if told to fetch it.
2. **The security posture** — `prototype`, `standard`, or `hardened`. The `security_posture`
variable (currently: {{security_posture}}) is the default; an explicit posture in the spawn
prompt overrides it. If neither is given, use `standard` and say so in the report.
3. **Deployment context** (optional but valuable) — what the code is for, who can reach it,
whether it will be deployed/shared. Use it to calibrate severity; never to skip the review.
## Workflow
1. Load `security-review`.
2. Get the diff (inline or via `get_diff`) and identify the changed files.
3. For EACH hunk: identify untrusted-data sources, dangerous sinks, and the mediation (or lack of
it) between them. Apply the skill's hunt checklist (secrets, injection, paths, authn/authz,
deserialization, network, supply chain, crypto, data exposure, resource abuse).
4. Ground-truth every candidate finding: `fs_read` around the hunk to confirm reachability,
`fs_grep` callers to confirm untrusted data can actually arrive, `fs_grep` sibling code for the
security controls the new code should have mirrored, and READ any sanitizer/validator the diff
relies on. Use `ast_grep` for structural checks (e.g. string-built SQL, `sh -c` call sites).
5. Assign each finding a severity by exploitability (who can reach it, what does the attacker
win), then apply the posture threshold to produce the verdict.
6. Emit the verdict in the skill's exact format.
## Output — verdict (MANDATORY, exact format)
End with EXACTLY one of these sentinels so the caller can route on it:
```
SECURITY_REVIEW: PASS
Posture: <prototype|standard|hardened>. Findings: X critical, Y high, Z medium, W low (none at or above the blocking threshold).
<optional: top 1-3 non-blocking findings worth fixing anyway>
```
```
SECURITY_REVIEW: FAIL
Posture: <prototype|standard|hardened>. Findings: X critical, Y high, Z medium, W low.
Blocking findings:
1. 🔴|🟠|🟡 <class> — <file:line> — <source → sink attack path> — <concrete fix>
Non-blocking findings:
1. 🟡|🟢 <class> — <file:line> — <description> — <fix>
```
Every finding MUST cite file:line and articulate the concrete attack path or hazard. A finding
with no location and no attack path is noise — do not emit it.
## Rules
1. **You are read-only.** Never modify files. You produce a verdict; the implementer owns the fix.
2. **Security, not quality.** Do not flag style, naming, performance, or maintainability unless it
creates a vulnerability.
3. **Critical always blocks — in every posture.** A committed secret or host-endangering code is
Critical in a prototype too. Posture gates High/Medium, never Critical.
4. **Posture gates the verdict, not the report.** Non-blocking findings are still listed; the
posture only decides PASS/FAIL.
5. **Review the CHANGE.** Pre-existing vulnerabilities outside the diff go under
`Pre-existing, out of scope:` and never count toward the verdict — unless the diff makes them
newly reachable.
6. **Severity = reachability × blast radius.** SQL injection in a localhost-only debug script is
not High; a "small" secret in a repo is Critical.
7. Be terse and decisive. Three exploitable findings beat fifteen theoretical ones. If everything
is theoretical hardening, it PASSes — say so.
## Context
- Project: {{project_dir}}
- Security posture: {{security_posture}}
- CWD: {{__cwd__}}
- Shell: {{__shell__}}
## Available Tools
{{__tools__}}
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
set -eo pipefail
# @env LLM_OUTPUT=/dev/stdout
# @env LLM_AGENT_VAR_PROJECT_DIR=.
# @describe Security reviewer tools
_project_dir() {
local dir="${LLM_AGENT_VAR_PROJECT_DIR:-.}"
(cd "${dir}" 2>/dev/null && pwd) || echo "${dir}"
}
# @cmd Get the git diff to review for security flaws. Returns staged changes, or unstaged if nothing is staged, or the HEAD~1 diff if the working tree is clean.
# @option --base Optional base ref to diff against (e.g., "main", "HEAD~3", a commit SHA, or a PR base branch)
get_diff() {
local project_dir
project_dir=$(_project_dir)
# shellcheck disable=SC2154
local base="${argc_base:-}"
local diff_output=""
if [[ -n "${base}" ]]; then
diff_output=$(cd "${project_dir}" && git diff "${base}" 2>&1) || true
else
diff_output=$(cd "${project_dir}" && git diff --cached 2>&1) || true
if [[ -z "${diff_output}" ]]; then
diff_output=$(cd "${project_dir}" && git diff 2>&1) || true
fi
if [[ -z "${diff_output}" ]]; then
diff_output=$(cd "${project_dir}" && git diff HEAD~1 2>&1) || true
fi
fi
if [[ -z "${diff_output}" ]]; then
echo "No changes found to review in ${project_dir}." >> "$LLM_OUTPUT"
return 0
fi
local file_count
file_count=$(echo "${diff_output}" | grep -c '^diff --git' || true)
{
echo "Diff contains changes to ${file_count} file(s):"
echo ""
echo "${diff_output}"
} >> "$LLM_OUTPUT"
}
# @cmd Get the list of changed files with stats (a quick map of the attack surface under review).
# @option --base Optional base ref to diff against
get_changed_files() {
local project_dir
project_dir=$(_project_dir)
local base="${argc_base:-}"
local stat_output=""
if [[ -n "${base}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat "${base}" 2>&1) || true
else
stat_output=$(cd "${project_dir}" && git diff --cached --stat 2>&1) || true
if [[ -z "${stat_output}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat 2>&1) || true
fi
if [[ -z "${stat_output}" ]]; then
stat_output=$(cd "${project_dir}" && git diff --stat HEAD~1 2>&1) || true
fi
fi
if [[ -z "${stat_output}" ]]; then
echo "No changes found in ${project_dir}." >> "$LLM_OUTPUT"
return 0
fi
{
echo "Changed files:"
echo ""
echo "${stat_output}"
} >> "$LLM_OUTPUT"
}
+6 -2
View File
@@ -26,8 +26,11 @@ flowchart TD
broad_gate -->|"no"| spec_gate broad_gate -->|"no"| spec_gate
code_reviewer --> spec_gate{"Implements<br/>a spec / plan?"} code_reviewer --> spec_gate{"Implements<br/>a spec / plan?"}
spec_gate -->|"yes"| adversary[["adversary<br/>plan-conformance"]] spec_gate -->|"yes"| adversary[["adversary<br/>plan-conformance"]]
spec_gate -->|"no"| done spec_gate -->|"no"| sec_gate
adversary --> done adversary --> sec_gate{"Touches attack surface?<br/>external input / auth /<br/>secrets / shell / deps"}
sec_gate -->|"yes"| security_reviewer[["security-reviewer<br/>posture-gated PASS/FAIL"]]
sec_gate -->|"no"| done
security_reviewer --> done
direct --> done direct --> done
done([Complete]) done([Complete])
@@ -43,6 +46,7 @@ Spawnable sub-agents (from `config.yaml`):
- **[coder](../coder/README.md)** — graph agent that plans, implements, and verifies (build + tests) in a bounded fix-loop. - **[coder](../coder/README.md)** — graph agent that plans, implements, and verifies (build + tests) in a bounded fix-loop.
- **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries. - **[code-reviewer](../code-reviewer/README.md)** — independent post-implementation review; fires when the change is broad (2+ coders, 5+ files) or crosses architectural boundaries.
- **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run. - **[adversary](../adversary/README.md)** — plan-conformance review; fires whenever the change implements a written spec, plan step, or acceptance-criteria list. Orthogonal to `code-reviewer` — both can run.
- **[security-reviewer](../security-reviewer/README.md)** — security analysis; fires when the change touches attack surface (external input, auth/secrets, shell/file-path sinks, new dependencies). Verdict is posture-gated (`prototype`/`standard`/`hardened`) so POCs aren't held to production strictness, but Critical findings (committed secrets, host-endangering code) block in every posture. Orthogonal to both other reviewers — all three can run.
- **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review. - **[step-runner](../step-runner/README.md)** — graph agent that executes one step of a phased plan repo. Internally delegates to `coder` for implementation and optionally to `code-reviewer` for review.
## Features ## Features
+59 -4
View File
@@ -1,6 +1,6 @@
name: sisyphus name: sisyphus
description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline description: OpenCode-style orchestrator - classifies intent, delegates to specialists, tracks progress with todos, enforces OMO-grade verification discipline
version: 3.2.0 version: 3.7.0
agent_session: temp agent_session: temp
auto_continue: true auto_continue: true
@@ -15,16 +15,21 @@ spawnable_agents:
- oracle - oracle
- code-reviewer - code-reviewer
- adversary - adversary
- security-reviewer
- architecture-reviewer
- step-runner - step-runner
max_concurrent_agents: 4 max_concurrent_agents: 40
max_agent_depth: 3 max_agent_depth: 3
inject_spawn_instructions: true inject_spawn_instructions: true
summarization_threshold: 8000 summarization_threshold: 80000
skills_enabled: true skills_enabled: true
enabled_skills: enabled_skills:
- ai-slop-remover - ai-slop-remover
- code-review - code-review
- comment-discipline
- diagnosing-bugs
- grilling
- git-master - git-master
- frontend-ui-ux - frontend-ui-ux
- delegation-protocol - delegation-protocol
@@ -124,6 +129,8 @@ instructions: |
For "improve X" / "refactor Y" / "clean up Z" type requests, quick-assess the codebase state BEFORE following patterns: For "improve X" / "refactor Y" / "clean up Z" type requests, quick-assess the codebase state BEFORE following patterns:
**Architecture-scale improvement requests** ("improve the architecture of X", "this module is hard to test", "make this easier to navigate") → delegate to `architecture-reviewer`. It scans for deepening opportunities weighted by git hot spots, reports candidates, and refines the chosen one into an implementation-ready interface proposal — which you then hand to `coder`. It proposes only; it is an on-demand tool, never a completion gate. For file-scale cleanups, proceed with the assessment below instead.
- **Disciplined** (consistent patterns, configs present, tests exist) → Follow existing style strictly - **Disciplined** (consistent patterns, configs present, tests exist) → Follow existing style strictly
- **Transitional** (mixed patterns) → Ask: "I see X and Y patterns. Which to follow?" - **Transitional** (mixed patterns) → Ask: "I see X and Y patterns. Which to follow?"
- **Legacy/Chaotic** (no consistency) → Propose: "No clear conventions. I suggest [X]. OK?" - **Legacy/Chaotic** (no consistency) → Propose: "No clear conventions. I suggest [X]. OK?"
@@ -336,10 +343,54 @@ instructions: |
Unlike `code-reviewer`, re-running `adversary` once after a conformance fix is expected — a DIVERGES verdict is a hard gate, and confirming the fix actually closed it is the point. Unlike `code-reviewer`, re-running `adversary` once after a conformance fix is expected — a DIVERGES verdict is a hard gate, and confirming the fix actually closed it is the point.
### Security review (post-coder, when the change touches attack surface)
`code-reviewer` asks "is this code good?" and `adversary` asks "is this the code the plan asked for?" — neither asks "can this code be abused?" Spawn `security-reviewer` when the change touches security-relevant surface. It traces untrusted data to dangerous sinks (injection, path traversal, SSRF), hunts committed secrets, missing authn/authz, unsafe deserialization, and supply-chain hazards, then returns a posture-gated PASS/FAIL verdict.
**When to spawn it** — ANY of these:
1. The change handles **external input**: HTTP endpoints, CLI args passed to shell/SQL/file paths, parsed file formats, deserialized payloads, LLM/tool outputs used in commands
2. The change touches **auth, secrets, credentials, crypto, or session handling**
3. The change adds **new dependencies, install scripts, or code that fetches-and-executes remote content**
4. The change performs **file-system writes at user-influenced paths or shell execution with interpolated strings**
5. **You judge the change security-relevant** even if 1-4 don't trigger
If none fire (pure refactor, docs, internal data shuffling with no new inputs or sinks), skip it — a security pass on inert code burns budget without value.
**Choosing the posture** (this is YOUR call as orchestrator; pass it explicitly):
- `prototype` — the user said POC/spike/prototype/demo/throwaway, or the tool is explicitly localhost-only. Blocks Critical only.
- `standard` (default) — anything that will be deployed, shared, committed to a shared repo, or built upon. Blocks Critical + High.
- `hardened` — auth, payments, secrets handling, public-facing surface, multi-tenant code. Blocks Critical + High + Medium.
When in doubt, use `standard`. Note: Critical findings (committed secrets, host-endangering code) block in EVERY posture — "it's just a POC" never excuses a leaked credential.
**Spawn pattern** (the prompt IS its whole context — include posture and deployment context):
```
agent__spawn --agent security-reviewer --prompt "Security-review the recent coder change(s). Return PASS/FAIL.
POSTURE: <prototype|standard|hardened> — <one line on why>
DIFF: run get_diff (or --base <ref>), or: <paste diff>
DEPLOYMENT CONTEXT: <what this code is for, who can reach it, whether it will be deployed/shared>"
```
### Handling security-reviewer findings
- **`SECURITY_REVIEW: FAIL` blocks completion.** Do not mark the task done. Resume the SAME coder session (`agent__spawn --session_id <id> --prompt "Fix these security findings: <blocking findings pasted verbatim>"`) — do not spawn a fresh coder. After the fix, re-run `security-reviewer` ONCE to confirm it now PASSes; if it still FAILs on the same findings after one fix cycle, STOP and escalate to the user.
- **`SECURITY_REVIEW: PASS`** — proceed. Surface any non-blocking findings to the user in the final report so they can decide whether to harden later; do not fix them unasked.
- **`Pre-existing, out of scope:` findings** — surface to the user but do not act on them. They predate this work and aren't the current task's responsibility.
- **Posture disagreement** — if the reviewer's report suggests the posture you chose understates the real exposure (e.g. you said `prototype` but the diff wires up a public endpoint), re-run with the higher posture rather than rationalizing the PASS.
Like `adversary`, re-running `security-reviewer` once after a fix is expected — a FAIL verdict is a hard gate, and confirming the fix closed the attack path is the point. Run all applicable reviewers (`code-reviewer`, `adversary`, `security-reviewer`) — they cover disjoint failure modes; one passing says nothing about the others.
## File Operations (Direct Edits) ## File Operations (Direct Edits)
When you write or modify files yourself (rather than delegating to coder): When you write or modify files yourself (rather than delegating to coder):
- **Calibrate comments before writing.** Load `comment-discipline` and note the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you read; write comments to match. When the signal is weak, write NO comment.
- **For editing an existing file**, prefer `fs_patch`. It's a surgical edit that preserves unchanged content. Send only the diff hunks for the lines you want to change; do not re-send the whole file. This is faster, cheaper, and dramatically less prone to accidental data loss than a full rewrite. - **For editing an existing file**, prefer `fs_patch`. It's a surgical edit that preserves unchanged content. Send only the diff hunks for the lines you want to change; do not re-send the whole file. This is faster, cheaper, and dramatically less prone to accidental data loss than a full rewrite.
- **For writing a NEW file or doing a COMPLETE rewrite**, use `fs_write`. Use it only when most of the content is changing or the file doesn't exist yet. - **For writing a NEW file or doing a COMPLETE rewrite**, use `fs_write`. Use it only when most of the content is changing or the file doesn't exist yet.
- **NEVER write files via `execute_command`.** Do not use: - **NEVER write files via `execute_command`.** Do not use:
@@ -358,6 +409,10 @@ instructions: |
## Phase 7 - Failure Recovery ## Phase 7 - Failure Recovery
### Hard bugs: load `diagnosing-bugs` BEFORE strike 2
A first fix attempt may go on the error message alone. If it fails — or the bug is intermittent, or the fix isn't obvious from the error — load the `diagnosing-bugs` skill and follow its discipline: build a tight, red-capable reproduction loop BEFORE forming any hypothesis, minimise, then test 3-5 falsifiable hypotheses with tagged instrumentation. Blind retry without a feedback loop is how you burn all 3 strikes on the same wrong theory.
### 3-strike rule ### 3-strike rule
After 3 consecutive failed fix attempts on the same problem: After 3 consecutive failed fix attempts on the same problem:
@@ -376,7 +431,7 @@ instructions: |
### Authoring lifecycle (no code changes) ### Authoring lifecycle (no code changes)
1. Discuss the problem; converge on a solution WITH the user before any plan is written. 1. Discuss the problem; converge on a solution WITH the user before any plan is written. Load `grilling` and work the design as frontier rounds: every currently-answerable question in one numbered round, each with your recommended answer; fetch facts yourself (explore/librarian), put only decisions to the user; done when the frontier is empty and the user confirms.
2. Load `plan-authoring`. Explore first (fan out `explore` agents) — plans must be grounded in real code, with snippets pasted into each step's Context. 2. Load `plan-authoring`. Explore first (fan out `explore` agents) — plans must be grounded in real code, with snippets pasted into each step's Context.
3. Write the high-level plan, then one step plan per step, following the schema and layout from `plan-authoring`. 3. Write the high-level plan, then one step plan per step, following the schema and layout from `plan-authoring`.
4. **Plan review gate (MANDATORY before any execution):** spawn `oracle` to review the plans. Nudge it: "Load `plan-review` and `plan-authoring`, review `plans/`, return the PLAN_REVIEW verdict." REJECT → fix the complaints, re-submit. Do not start execution on an unreviewed or rejected plan. 4. **Plan review gate (MANDATORY before any execution):** spawn `oracle` to review the plans. Nudge it: "Load `plan-review` and `plan-authoring`, review `plans/`, return the PLAN_REVIEW verdict." REJECT → fix the complaints, re-submit. Do not start execution on an unreviewed or rejected plan.
+1 -2
View File
@@ -21,8 +21,7 @@
}, },
"iwe": { "iwe": {
"type": "stdio", "type": "stdio",
"command": "iwec", "command": "iwec"
"args": ["--project", "."]
} }
} }
} }
+23 -2
View File
@@ -14,6 +14,11 @@ set -e
# is the most common cause of "unable to apply patch" failures, especially in files with sed/jq/regex pipelines or # is the most common cause of "unable to apply patch" failures, especially in files with sed/jq/regex pipelines or
# embedded Python with quoted strings. # embedded Python with quoted strings.
# - Hunks are applied in order; the first hunk that fails aborts the whole patch — later hunks are NOT attempted. # - Hunks are applied in order; the first hunk that fails aborts the whole patch — later hunks are NOT attempted.
# - Hunks anchor at the FIRST exact match of their context in the file, so include enough context to make each hunk
# unique. Hunks must appear in file order.
# - Every hunk needs at least one context or removed line to anchor it; a hunk containing only additions is an error.
# - Unrecognized lines inside a hunk are hard errors: every hunk line must start with ' ' (context), '-' (removal),
# or '+' (addition).
# - If you've edited this file in earlier tool calls, fs_cat it again before composing the patch. A stale view of the file # - If you've edited this file in earlier tool calls, fs_cat it again before composing the patch. A stale view of the file
# produces context lines that no longer match. # produces context lines that no longer match.
# - On failure the error message names the failing hunk and shows the expected-vs-actual line. Fix that specific line and # - On failure the error message names the failing hunk and shows the expected-vs-actual line. Fix that specific line and
@@ -33,7 +38,11 @@ source "$LLM_PROMPT_UTILS_FILE"
# shellcheck disable=SC2154 # shellcheck disable=SC2154
main() { main() {
argc_contents="$(jq -r '.content' <<< "$LLM_TOOL_RAW_JSON")" # Command substitution strips *all* trailing newlines and `jq -r` appends one
# of its own, so read with `-j` and pin the real end of the content with a
# sentinel that is removed afterwards.
argc_contents="$(jq -j '.content' <<< "$LLM_TOOL_RAW_JSON"; printf x)"
argc_contents="${argc_contents%x}"
argc_path="$(jq -r '.path' <<< "$LLM_TOOL_RAW_JSON")" argc_path="$(jq -r '.path' <<< "$LLM_TOOL_RAW_JSON")"
if [[ ! -f "$argc_path" ]]; then if [[ ! -f "$argc_path" ]]; then
@@ -41,7 +50,19 @@ main() {
exit 1 exit 1
fi fi
new_contents="$(patch_file "$argc_path" <(printf "%s" "$argc_contents"))" # Same sentinel guard on the patched result, otherwise the trailing newline
# is stripped again on the way back out. `rc` preserves patch_file's exit
# status so a failure still aborts under `set -e`.
new_contents="$(patch_file "$argc_path" <(printf "%s" "$argc_contents"); rc=$?; printf x; exit "$rc")"
new_contents="${new_contents%x}"
# awk newline-terminates every printed line, so patching a file that lacks
# a final newline would silently add one. Preserve the original file's
# final-newline state instead.
if [[ -n "$(tail -c 1 "$argc_path")" ]]; then
new_contents="${new_contents%$'\n'}"
fi
printf "%s" "$new_contents" | git diff --no-index "$argc_path" - || true printf "%s" "$new_contents" | git diff --no-index "$argc_path" - || true
guard_operation "Apply changes?" guard_operation "Apply changes?"
+6 -1
View File
@@ -15,7 +15,12 @@ source "$LLM_PROMPT_UTILS_FILE"
# shellcheck disable=SC2154 # shellcheck disable=SC2154
main() { main() {
argc_contents="$(jq -r '.content' <<< "$LLM_TOOL_RAW_JSON")" # Command substitution strips *all* trailing newlines and `jq -r` appends one
# of its own, so read with `-j` and pin the real end of the content with a
# sentinel that is removed afterwards. Without this every written file loses
# its final newline, which breaks formatters such as `cargo fmt --check`.
argc_contents="$(jq -j '.content' <<< "$LLM_TOOL_RAW_JSON"; printf x)"
argc_contents="${argc_contents%x}"
argc_path="$(jq -r '.path' <<< "$LLM_TOOL_RAW_JSON")" argc_path="$(jq -r '.path' <<< "$LLM_TOOL_RAW_JSON")"
if [[ -f "$argc_path" ]]; then if [[ -f "$argc_path" ]]; then
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
set -e
# @describe Execute a git command. Strictly limited to a single git invocation: the command must start with 'git' and shell metacharacters (; & | < > ( ) $ `) are rejected outside single quotes — no pipes, chaining, redirection, or command substitution. Use git's own flags instead of pipes (e.g. 'git log -n 20' instead of piping to head). Output is never paginated and git will never prompt for input.
# @option --command! The git command to execute (e.g. "git status --short").
# @env LLM_OUTPUT=/dev/stdout The output path
# shellcheck disable=SC1090
source "$LLM_PROMPT_UTILS_FILE"
main() {
# shellcheck disable=SC2154
argc_command="$(jq -r '.command' <<< "$LLM_TOOL_RAW_JSON")"
validate_command "$argc_command" "git"
guard_operation "Execute git command: $argc_command"
export GIT_PAGER=cat PAGER=cat GIT_TERMINAL_PROMPT=0
export GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true
local script
script="$(mktemp)"
# shellcheck disable=SC2064
trap "rm -f '$script'" EXIT
printf '%s\n' "$argc_command" > "$script"
bash -e -o pipefail "$script" >> "$LLM_OUTPUT"
}
die() {
echo "$*" >&2
exit 1
}
# Ensure the command is a single plain invocation of $2 with no shell escape
# hatches. Metacharacters are allowed inside single quotes (where bash treats
# them as literals) but rejected everywhere else, including $ and ` inside
# double quotes (expansion/substitution).
validate_command() {
local cmd="$1" prog="$2"
local first
first="$(awk '{print $1}' <<< "$cmd")"
if [[ "$first" != "$prog" ]]; then
die "error: this tool only executes $prog commands; the command must start with '$prog' (got: '${first:-<empty>}')"
fi
local i c in_single=0 in_double=0 len=${#cmd}
for (( i = 0; i < len; i++ )); do
c="${cmd:i:1}"
if (( in_single )); then
[[ "$c" == "'" ]] && in_single=0
continue
fi
if (( in_double )); then
case "$c" in
'\') i=$((i + 1)) ;;
'"') in_double=0 ;;
'$' | '`') die "error: '$c' is not allowed inside double quotes (expansion/substitution is blocked); use single quotes for literal text" ;;
esac
continue
fi
case "$c" in
'\') i=$((i + 1)) ;;
"'") in_single=1 ;;
'"') in_double=1 ;;
';' | '&' | '|' | '<' | '>' | '(' | ')' | '$' | '`')
die "error: shell metacharacter '$c' is not allowed; run a single $prog command with no pipes, chaining, redirection, or substitution (use $prog's own flags instead, and single quotes for literal text)"
;;
$'\n')
die "error: newlines are not allowed; run a single $prog command"
;;
esac
done
if (( in_single || in_double )); then
die "error: unbalanced quotes in command"
fi
}
+89 -15
View File
@@ -186,7 +186,9 @@ input() {
} }
confirm() { confirm() {
trap "stty echo; exit" EXIT # stty targets stdin, which is /dev/null when the host spawns tool scripts;
# point it at the real terminal and stay quiet when there isn't one.
trap "stty echo </dev/tty 2>/dev/null; exit" EXIT
_prompt_text "$1 (y/N)" _prompt_text "$1 (y/N)"
echo -en "\033[36m\c " >&2 echo -en "\033[36m\c " >&2
@@ -229,7 +231,7 @@ list() {
declare first_row declare first_row
first_row=$((last_row - opts_count + 1)) first_row=$((last_row - opts_count + 1))
trap "_cursor_blink_on; stty echo; exit" 2 trap "_cursor_blink_on; stty echo </dev/tty 2>/dev/null; exit" 2
_cursor_blink_off _cursor_blink_off
@@ -275,7 +277,7 @@ checkbox() {
declare first_row declare first_row
first_row=$((last_row - opts_count + 1)) first_row=$((last_row - opts_count + 1))
trap "_cursor_blink_on; stty echo; exit" 2 trap "_cursor_blink_on; stty echo </dev/tty 2>/dev/null; exit" 2
_cursor_blink_off _cursor_blink_off
@@ -403,7 +405,7 @@ range() {
declare current_row declare current_row
current_row=$((first_row - 1)) current_row=$((first_row - 1))
trap "_cursor_blink_on; stty echo; exit" 2 trap "_cursor_blink_on; stty echo </dev/tty 2>/dev/null; exit" 2
_cursor_blink_off _cursor_blink_off
@@ -528,7 +530,11 @@ guard_operation() {
# + print(f"Hello {name}") # + print(f"Hello {name}")
patch_file() { patch_file() {
awk ' awk '
FNR == NR { function isHeaderPair(i) {
return (patchLines[i] ~ /^--- / && patchLines[i+1] ~ /^\+\+\+ / && patchLines[i+2] ~ /^@@/)
}
FILENAME == ARGV[1] {
lines[FNR] = $0 lines[FNR] = $0
next; next;
} }
@@ -547,11 +553,6 @@ patch_file() {
while (patchLineIndex <= totalPatchLines) { while (patchLineIndex <= totalPatchLines) {
line = patchLines[patchLineIndex] line = patchLines[patchLineIndex]
if (line ~ /^--- / || line ~ /^\+\+\+ /) {
patchLineIndex++
continue
}
if (line ~ /^@@/) { if (line ~ /^@@/) {
mode = "hunk" mode = "hunk"
hunkIndex++ hunkIndex++
@@ -560,7 +561,22 @@ patch_file() {
} }
if (mode == "hunk") { if (mode == "hunk") {
while (patchLineIndex <= totalPatchLines && line ~ /^[-+ ]|^\s*$/ && line !~ /^--- /) { while (patchLineIndex <= totalPatchLines) {
line = patchLines[patchLineIndex]
if (line ~ /^\\ No newline/) {
patchLineIndex++
continue
}
if (isHeaderPair(patchLineIndex)) {
break
}
if (line !~ /^[-+ ]/ && line !~ /^[ \t]*$/) {
break
}
sanitizedLine = substr(line, 2) sanitizedLine = substr(line, 2)
if (line !~ /^\+/) { if (line !~ /^\+/) {
@@ -574,13 +590,37 @@ patch_file() {
} }
patchLineIndex++ patchLineIndex++
line = patchLines[patchLineIndex]
} }
mode = "none" mode = "none"
} else { continue
patchLineIndex++
} }
if (isHeaderPair(patchLineIndex)) {
patchLineIndex += 2
continue
}
if (line ~ /^\\ No newline/) {
patchLineIndex++
continue
}
if (hunkIndex == 0) {
# Preamble before the first hunk: tolerate prose, code fences, and lone headers.
patchLineIndex++
continue
}
if (line ~ /^[ \t]*$/ || line ~ /^```/) {
patchLineIndex++
continue
}
print "error: unrecognized line in patch (line " patchLineIndex "): " line > "/dev/stderr"
print "" > "/dev/stderr"
print "Every line inside a hunk must start with \" \" (context), \"-\" (removal), or \"+\" (addition)." > "/dev/stderr"
exit 1
} }
if (hunkIndex == 0) { if (hunkIndex == 0) {
@@ -593,6 +633,36 @@ patch_file() {
} }
totalHunks = hunkIndex totalHunks = hunkIndex
if (totalLines == 0) {
for (h = 1; h <= totalHunks; h++) {
if (hunkTotalOriginalLines[h] > 0) {
print "error: unable to apply patch" > "/dev/stderr"
print "" > "/dev/stderr"
print "Hunk " h " expects existing content but the file is empty." > "/dev/stderr"
exit 1
}
}
for (h = 1; h <= totalHunks; h++) {
for (i = 1; i <= hunkTotalUpdatedLines[h]; i++) {
print hunkUpdatedLines[h,i]
}
}
exit 0
}
for (h = 1; h <= totalHunks; h++) {
if (hunkTotalOriginalLines[h] == 0) {
print "error: unable to apply patch" > "/dev/stderr"
print "" > "/dev/stderr"
print "Hunk " h " contains no context or removed lines; include at least one" > "/dev/stderr"
print "context line so the hunk can be anchored." > "/dev/stderr"
exit 1
}
}
hunkIndex = 1 hunkIndex = 1
for (lineIndex = 1; lineIndex <= totalLines; lineIndex++) { for (lineIndex = 1; lineIndex <= totalLines; lineIndex++) {
@@ -603,7 +673,7 @@ patch_file() {
nextLineIndex = lineIndex + 1 nextLineIndex = lineIndex + 1
for (i = 2; i <= hunkTotalOriginalLines[hunkIndex]; i++) { for (i = 2; i <= hunkTotalOriginalLines[hunkIndex]; i++) {
if (lines[nextLineIndex] != hunkOriginalLines[hunkIndex,i]) { if (nextLineIndex > totalLines || lines[nextLineIndex] != hunkOriginalLines[hunkIndex,i]) {
if (i - 1 > bestPartialLen[hunkIndex]) { if (i - 1 > bestPartialLen[hunkIndex]) {
bestPartialLen[hunkIndex] = i - 1 bestPartialLen[hunkIndex] = i - 1
bestPartialAnchorLine[hunkIndex] = lineIndex bestPartialAnchorLine[hunkIndex] = lineIndex
@@ -646,10 +716,14 @@ patch_file() {
print "" > "/dev/stderr" print "" > "/dev/stderr"
print "Closest match: anchored at file line " bestPartialAnchorLine[failingHunk] ", matched " bestPartialLen[failingHunk] " of " hunkTotalOriginalLines[failingHunk] " original lines before diverging." > "/dev/stderr" print "Closest match: anchored at file line " bestPartialAnchorLine[failingHunk] ", matched " bestPartialLen[failingHunk] " of " hunkTotalOriginalLines[failingHunk] " original lines before diverging." > "/dev/stderr"
print "" > "/dev/stderr" print "" > "/dev/stderr"
if (bestPartialDivergeLine[failingHunk] > totalLines) {
print "The hunk expects additional lines beyond the end of the file (file has " totalLines " lines)." > "/dev/stderr"
} else {
print "At file line " bestPartialDivergeLine[failingHunk] " (hunk original line " bestPartialHunkPos[failingHunk] "):" > "/dev/stderr" print "At file line " bestPartialDivergeLine[failingHunk] " (hunk original line " bestPartialHunkPos[failingHunk] "):" > "/dev/stderr"
print " expected: " bestPartialExpected[failingHunk] > "/dev/stderr" print " expected: " bestPartialExpected[failingHunk] > "/dev/stderr"
print " actual: " bestPartialActual[failingHunk] > "/dev/stderr" print " actual: " bestPartialActual[failingHunk] > "/dev/stderr"
} }
}
print "" > "/dev/stderr" print "" > "/dev/stderr"
print "Lines must match byte-for-byte (no fuzzy matching). Check escaping, whitespace, and quoting." > "/dev/stderr" print "Lines must match byte-for-byte (no fuzzy matching). Check escaping, whitespace, and quoting." > "/dev/stderr"
@@ -1,2 +1,3 @@
description: Generate a git commit message from the current diff
steps: steps:
- .file `git diff` -- generate a git commit message - .file `git diff` -- generate a git commit message
+25
View File
@@ -88,6 +88,7 @@ A diff review is a review of THE CHANGE, not the whole file:
- Are names accurate? `get_user` that mutates is a lie; rename or split. - Are names accurate? `get_user` that mutates is a lie; rename or split.
- Could a competent reader understand this without comments? - Could a competent reader understand this without comments?
- Do NEW comments match the repo's comment register? You already read neighboring files for conventions — compare against them. Flag BOTH directions: narrated/restating comments in a repo that uses self-documenting code (each one is a finding, cite the line), AND missing doc comments on new public items in a repo that documents its public API. Comments explaining non-obvious *why* (decisions, workarounds, invariants) are warranted in every repo; comments captioning *what* the code plainly does are warranted in none.
- Is there a simpler way to express the same logic? - Is there a simpler way to express the same logic?
- Is the function doing one thing, or several things glued together? - Is the function doing one thing, or several things glued together?
@@ -96,6 +97,7 @@ A diff review is a review of THE CHANGE, not the whole file:
- Does this change increase coupling between modules unnecessarily? - Does this change increase coupling between modules unnecessarily?
- Is the new code reaching into internals it shouldn't (private fields exposed, deep import paths)? - Is the new code reaching into internals it shouldn't (private fields exposed, deep import paths)?
- Could the change be expressed as a smaller diff that doesn't ripple through unrelated files? - Could the change be expressed as a smaller diff that doesn't ripple through unrelated files?
- New helper/utility/constant introduced? `fs_grep` for an existing equivalent in the repo before accepting it — duplicating an existing helper is a finding; cite the original's path so the author can reuse it. (The inverse is not a finding: do not demand a new abstraction to unify two mildly similar blocks.)
## 5. Footguns ## 5. Footguns
@@ -104,6 +106,29 @@ A diff review is a review of THE CHANGE, not the whole file:
- Are error types specific enough to be actionable? - Are error types specific enough to be actionable?
- Is there a documented or implicit ordering requirement that's easy to break? - Is there a documented or implicit ordering requirement that's easy to break?
## 6. Code smells (baseline heuristics)
A fixed baseline of named smells (Fowler, *Refactoring* ch. 3) that applies even when the repo documents no standards. Three calibration rules bind it:
1. **The repo overrides.** A documented or established repo convention always wins; where the codebase deliberately does something the baseline would flag, suppress the smell.
2. **Always a judgment call.** Report each as a labelled heuristic ("possible Feature Envy"), never a hard violation — severity 🟢 Suggestion or 💡 Nitpick unless it compounds a real defect.
3. **Skip anything tooling already enforces.** Linters and formatters own their territory.
Each smell reads *what it is → how to fix*; match against the diff only:
- **Mysterious Name**: a function/variable/type whose name doesn't reveal what it does or holds → rename; if no honest name comes, the design is murky.
- **Duplicated Code**: the same logic shape in more than one hunk or file of the change → extract the shared shape, call it from both. (For duplication against EXISTING code, see the Coupling grep check above.)
- **Feature Envy**: a method reaching into another object's data more than its own → move the method onto the data it envies.
- **Data Clumps**: the same few fields/params travelling together — a type wanting to be born → bundle them into one type.
- **Primitive Obsession**: a primitive/string standing in for a domain concept → give the concept its own small type.
- **Repeated Switches**: the same `switch`/`if`-cascade on the same type recurring across the change → polymorphism, or one shared map.
- **Shotgun Surgery**: one logical change forcing scattered edits across many files in the diff → gather what changes together into one module.
- **Divergent Change**: one file edited for several unrelated reasons → split so each module changes for one reason.
- **Speculative Generality**: abstraction/parameters/hooks added for needs nothing in the change has → delete; inline until a real need shows.
- **Message Chains**: long `a.b().c().d()` navigation the caller shouldn't depend on → hide the walk behind one method on the first object.
- **Middle Man**: a class/function that mostly delegates onward → cut it, call the real target directly.
- **Refused Bequest**: a subclass/implementer ignoring or overriding most of what it inherits → drop the inheritance, use composition.
## What to flag ## What to flag
- Correctness bugs. - Correctness bugs.
+61
View File
@@ -0,0 +1,61 @@
---
description: Shared vocabulary and principles for designing deep modules - module, interface, depth, seam, adapter, leverage, locality - plus the deletion test, dependency categories for safe deepening, and the design-it-twice pattern for exploring alternative interfaces. Load when designing or improving a module's interface, deciding where a seam goes, making code more testable, or when another skill or agent needs the deep-module vocabulary.
---
Design **deep modules**: a lot of behaviour behind a small interface, placed at a clean seam, testable through that interface. Use this language and these principles wherever code is being designed or restructured. The aim is leverage for callers, locality for maintainers, and testability for everyone.
## Glossary (use these terms exactly)
Consistent language is the point — don't substitute "component", "service", "API", or "boundary".
- **Module**: anything with an interface and an implementation. Deliberately scale-agnostic: a function, class, package, or tier-spanning slice.
- **Interface**: everything a caller must know to use the module correctly — the type signature, but also invariants, ordering constraints, error modes, required configuration, and performance characteristics. ("API"/"signature" are too narrow: they name only the type-level surface.)
- **Implementation**: what's inside a module.
- **Depth**: leverage at the interface — how much behaviour a caller (or test) can exercise per unit of interface they must learn. **Deep** = lots of behaviour behind a small interface. **Shallow** = an interface nearly as complex as the implementation.
- **Seam** *(Feathers)*: a place where you can alter behaviour without editing in that place; the *location* where a module's interface lives. Where the seam goes is its own design decision, distinct from what goes behind it. (Avoid "boundary" — overloaded with DDD's bounded context.)
- **Adapter**: a concrete thing that satisfies an interface at a seam. Names *role* (what slot it fills), not substance.
- **Leverage**: what callers get from depth — more capability per unit of interface learned. One implementation pays back across N call sites and M tests.
- **Locality**: what maintainers get from depth — change, bugs, knowledge, and verification concentrate in one place. Fix once, fixed everywhere.
## Principles
- **Depth is a property of the interface, not the implementation.** A deep module can be internally composed of small, mockable parts; they just aren't part of the interface. A module can have **internal seams** (private, used by its own tests) as well as the external seam at its interface — don't expose internal seams just because tests use them.
- **The deletion test.** Imagine deleting the module. If complexity vanishes, it was a pass-through. If complexity reappears across N callers, it was earning its keep. Apply this to anything you suspect is shallow.
- **The interface is the test surface.** Callers and tests cross the same seam. Wanting to test *past* the interface means the module is probably the wrong shape.
- **One adapter means a hypothetical seam. Two adapters means a real one.** Don't introduce a seam unless something actually varies across it (typically production + test). A single-adapter seam is just indirection.
- When designing an interface, ask: can I reduce the number of methods? simplify the parameters? hide more complexity inside?
## Designing for testability
1. **Accept dependencies, don't create them**`processOrder(order, paymentGateway)` is testable; a function that constructs its own gateway is not.
2. **Return results, don't produce side effects**`calculateDiscount(cart): Discount` beats `applyDiscount(cart): void`.
3. **Small surface area** — fewer methods = fewer tests needed; fewer params = simpler setup.
## Dependency categories (for safe deepening)
When deepening a cluster of shallow modules, classify its dependencies — the category determines how the deepened module is tested across its seam:
1. **In-process** (pure computation, in-memory state): always deepenable; merge and test through the new interface directly. No adapter needed.
2. **Local-substitutable** (deps with real local stand-ins: embedded/in-memory DB, in-memory filesystem): deepenable if the stand-in exists; the test suite runs the stand-in, the seam stays internal.
3. **Remote but owned** (your own services across a network): define a port (interface) at the seam; production gets an HTTP/gRPC/queue adapter, tests get an in-memory adapter. The logic sits in one deep module even though it deploys across a network.
4. **True external** (third-party services you don't control): injected port; tests provide a mock adapter.
**Testing strategy: replace, don't layer.** Once tests exist at the deepened module's interface, old unit tests on the merged shallow modules are waste — delete them. New tests assert observable outcomes through the interface and survive internal refactors; a test that must change when the implementation changes is testing past the interface.
## Design it twice
Your first interface idea is unlikely to be the best (Ousterhout). For a module worth the effort, produce **2-3 radically different interface designs** before committing — in parallel sub-agents when available, sequentially otherwise. Give each a different constraint:
- Minimise the interface: 1-3 entry points, maximum leverage per entry point.
- Maximise flexibility: many use cases, room for extension.
- Optimise for the most common caller: make the default case trivial.
- (When cross-seam dependencies dominate) design around ports & adapters.
Each design specifies: the interface (including invariants, ordering, error modes), a caller usage example, what the implementation hides, the dependency/adapter strategy, and where leverage is high vs thin. Compare on **depth**, **locality**, and **seam placement**, then give ONE opinionated recommendation (or a justified hybrid) — the reader wants a strong read, not a menu.
## Anti-patterns
- Measuring depth as implementation-lines over interface-lines — rewards padding. Depth is leverage, not a ratio.
- Extracting pure functions "for testability" while the real bugs live in how they're called — that trades away locality and deepens nothing.
- Introducing ports/interfaces speculatively ("we might swap the DB") — one adapter is a hypothetical seam.
- Renaming without restructuring: calling a pass-through layer an "adapter" doesn't make the module deep. Apply the deletion test.
- Vocabulary drift mid-discussion ("component", "service", "boundary") — the shared terms exist so design conversations compose.
+51
View File
@@ -0,0 +1,51 @@
---
description: Calibrate comment density and style to the repository's existing conventions before writing code. Detects the repo's comment register (self-documenting / api-documented / comment-heavy) from the sibling files you already read for pattern matching, or from a declared policy in workspace instructions, then dictates when a comment is warranted. Default when signal is weak - write NO comment. Complements ai-slop-remover (which bans comments that restate code in every register).
---
You are about to write or modify code. LLMs systematically over-comment — narrating every block, restating signatures, banner-ing sections — and that default is wrong in most repositories. Before writing, determine the repo's **comment register** and match it, exactly the way you already match imports, naming, and error handling.
## Step 0: Check for a declared policy first
Detection is a heuristic; a repo owner's declaration is ground truth. Before sampling files, check the workspace instructions already in your context (`COYOTE.md` / `AGENTS.md` / `CLAUDE.md`) for a stated comment policy (e.g. a "Comments" or "Style" section). If one exists, obey it and skip detection entirely.
## Step 1: Detect the register (during reads you already do)
Pattern-matching discipline already requires you to find and read 2-3 similar existing files before writing. While reading them, observe:
1. **Density** — roughly what fraction of lines are comments? Near-zero, sparse (~1 per function or less), or pervasive (most blocks narrated)?
2. **Types present** — doc comments on public items (`///`, `/** */`, docstrings)? Inline "why" comments? Section banners (`// ===== Handlers =====`)? Commented-out code (a smell, not a convention — never imitate it)?
3. **What the comments say** — do they explain *why* (decisions, workarounds, invariants, links to issues) or narrate *what* (restating the code)? A repo whose comments are all "why" is self-documenting even if density is nonzero.
4. **Config signals** — these force the answer regardless of sampled style: `#![warn(missing_docs)]` or `#![deny(missing_docs)]` in Rust, eslint `jsdoc`/`require-jsdoc` rules, pylint/pydocstyle docstring checkers, a lint config banning TODO without a ticket. Lint-enforced conventions are mandatory.
5. **TODO/FIXME conventions** — bare `TODO:`, or `TODO(name):`, or ticket-linked `TODO(#123):`? Match the observed form if you must leave one.
Sample from the SAME language and module you're editing — a repo can have a chatty Python test suite and a silent Rust core. The nearest siblings win.
## Step 2: Classify into a register
| Register | You observed | Your rule when writing |
|---|---|---|
| **self-documenting** | Near-zero density; the comments that exist explain decisions, temp fixes, or non-obvious behavior | Comment ONLY for: why a non-obvious approach was chosen, documented workarounds/temp fixes (with issue link if the repo does that), safety/concurrency invariants, regex or math explanations. Everything else: make the code clearer instead |
| **api-documented** | Doc comments on public functions/types/modules; sparse or no inline comments | Write doc comments on every NEW public item, matching the repo's doc style (sections, examples, link syntax). Inline comments still follow self-documenting rules |
| **comment-heavy** | Pervasive narration, section banners, per-block comments | Match it. Comment your work the way the siblings do — same placement, same tone, same banner style. Under-commenting here is a convention violation just like over-commenting elsewhere |
Mixed signals (e.g. doc comments everywhere + narrated private code) → combine rows: doc comments mandatory AND inline narration matched.
## Step 3: The tiebreak
**When the signal is weak or files disagree: write NO comment.** Your untrained default is comment-heavy, so the correction must push the other way. A missing comment is a one-line review nit; a hundred useless comments are a cleanup task. If you genuinely cannot tell and the comment feels important, it is usually a sign the code should be restructured until the comment is unnecessary.
## Invariants that do NOT bend with register
1. **Never restate the code.** `// increment the counter` above `counter += 1` is slop in EVERY register — comment-heavy repos narrate intent and sections, they don't caption individual lines the reader can read. (This is `ai-slop-remover`'s rule; it applies unconditionally.)
2. **Always keep the genuinely necessary comment**, even in the sparsest repo: non-obvious algorithm choices (with the reference), regex explanations, safety invariants (`unsafe` justifications, lock ordering), intentional deviations from the obvious approach, and workarounds for upstream bugs (with the link).
3. **Never leave commented-out code**, regardless of what the repo tolerates.
4. **Never delete or rewrite EXISTING comments** that don't match the register you detected — that's out-of-scope churn. Register calibration governs comments YOU write.
5. **Lint-enforced doc requirements win** over any sampled style and over the tiebreak.
## Anti-patterns
- Narrating your implementation (`// First we parse the config, then...`) in a repo whose functions are bare.
- Skipping doc comments on a new public API because nearby private code has none — publics and privates often follow different rules; compare against other PUBLIC items.
- Writing doc comments that restate the signature (`/// Gets the user. Returns the user.`) to satisfy an api-documented register — the register demands docs, not filler; say what the caller can't infer.
- Section banners in a repo that has none.
- Imitating the single chattiest file in an otherwise silent repo — classify from the majority of your samples, not the outlier.
- Treating this skill as license to argue with a declared policy: COYOTE.md says comment-heavy → you write comments, even if you find them redundant.
+1 -1
View File
@@ -21,7 +21,7 @@ Plans written from memory rot on contact with the code. Before writing anything:
## Step 2 — The proposal ## Step 2 — The proposal
Produce a structured proposal (iterate with the user when interactive; in autonomous runs, resolve what the doc + code answer and flag the rest as open questions): Produce a structured proposal (iterate with the user when interactive — load the `grilling` skill and work the open decisions as frontier rounds, each question carrying a recommended answer; in autonomous runs, resolve what the doc + code answer and flag the rest as open questions):
- **Problem** — one paragraph; state assumptions explicitly. - **Problem** — one paragraph; state assumptions explicitly.
- **Scope** — In / Out. Call out tempting adjacent work being deferred. - **Scope** — In / Out. Call out tempting adjacent work being deferred.
+82
View File
@@ -0,0 +1,82 @@
---
description: Feedback-loop-first debugging discipline for hard code bugs and performance regressions. Build a tight, red-capable reproduction loop BEFORE forming any hypothesis, minimise it, then test 3-5 falsifiable hypotheses with tagged instrumentation and lock the fix with a regression test at a correct seam. Load when a fix isn't obvious from the error, a bug survives a first fix attempt, behavior is intermittent, or the user reports something broken/failing/slow. Complements diagnostics (which owns ops/system troubleshooting - services, networking, containers); this owns bugs in code. Grants shell access for running reproduction loops.
enabled_tools: execute_command
---
You are hunting a bug in code. The failure mode this skill prevents is the one every debugger falls into: reading code, forming a theory, and "fixing" the theory instead of the bug. The discipline: **no hypothesis until a feedback loop exists.** Skip phases only when you can say why.
**Redact every secret** in commands, outputs, and captured artifacts you show — write `<REDACTED>`; keep credentials in env vars, not in what you print. Quote only the lines of captured artifacts that carry signal.
## Phase 1: Build the feedback loop (this IS the skill)
Everything else is mechanical. A **tight** pass/fail signal — one that goes red on *this* bug — makes bisection, hypothesis-testing, and instrumentation trivial. Without one, no amount of code-reading will save you. Spend disproportionate effort here.
Ways to construct one, in rough order of preference:
1. **Failing test** at whatever seam reaches the bug: unit, integration, e2e.
2. **curl / HTTP script** against a running dev server.
3. **CLI invocation** with a fixture input, diffing output against a known-good snapshot.
4. **Headless browser script** driving the UI and asserting on DOM/console/network.
5. **Replay a captured trace** — save a real request/payload/event log, replay it through the code path in isolation.
6. **Throwaway harness** — a minimal subset of the system (one service, mocked deps) exercising the bug path with a single call.
7. **Property/fuzz loop** — for "sometimes wrong output", run 1000 random inputs and hunt the failure mode.
8. **Bisection harness** — bug appeared between two known states? Automate "boot at X, check" so `git bisect run` can consume it.
9. **Differential loop** — same input through old vs new version (or two configs), diff the outputs.
Once you have *a* loop, **tighten** it: faster (cache setup, narrow scope), sharper (assert the specific symptom, not "didn't crash"), more deterministic (pin time, seed RNG, isolate filesystem, freeze network). A 2-second deterministic loop is a debugging superpower; a 30-second flaky one is barely better than nothing.
**Non-deterministic bugs**: the goal is a higher reproduction *rate*, not a clean repro. Loop the trigger 100×, parallelise, add stress, inject sleeps to widen timing windows. A 50% flake is debuggable; 1% is not.
**Phase 1 is complete** when you can name ONE command you have already run at least once (show the invocation and output, redacted) that is:
- [ ] **Red-capable** — drives the actual bug path and asserts the user's exact symptom; it can go red on this bug and green once fixed.
- [ ] **Deterministic** — same verdict every run (or a pinned, high reproduction rate).
- [ ] **Fast** — seconds, not minutes.
- [ ] **Agent-runnable** — you can run it unattended.
If you catch yourself reading code to build a theory before this command exists — STOP. That is the exact failure this skill exists to prevent. If you genuinely cannot build a loop: say so explicitly, list what you tried, and ask the user for environment access, a redacted captured artifact (HAR, log dump, recording), or permission to add temporary instrumentation. Do NOT proceed to hypothesise without a loop.
## Phase 2: Reproduce + minimise
Run the loop; watch it go red. Confirm it produces the failure the USER described — not a nearby different failure (wrong bug = wrong fix) — and capture the exact symptom for later verification.
Then **minimise**: shrink to the smallest scenario that still goes red. Cut inputs, callers, config, and steps one at a time, re-running after each cut. Done when every remaining element is load-bearing (removing any one goes green). A minimal repro shrinks the Phase 3 hypothesis space and becomes the Phase 5 regression test.
## Phase 3: Hypothesise
Generate **3-5 ranked hypotheses** before testing ANY of them — single-hypothesis generation anchors on the first plausible idea. Each must be **falsifiable**: "if X is the cause, then changing Y makes the bug disappear / changing Z makes it worse." Can't state the prediction? It's a vibe — discard or sharpen.
Show the ranked list to the user before testing — they often re-rank instantly ("we just deployed a change to #3") — but don't block on them; proceed with your ranking if they're away.
## Phase 4: Instrument
Every probe maps to a specific Phase 3 prediction. **One variable at a time.**
1. Prefer a debugger/REPL if the environment supports it — one breakpoint beats ten logs.
2. Otherwise targeted logs at the boundaries that DISTINGUISH hypotheses. Never "log everything and grep".
3. **Tag every debug log with a unique prefix** (e.g. `[DEBUG-a4f2]`) so cleanup is a single grep. Untagged logs survive into production; tagged logs die.
**Performance regressions**: logs are usually the wrong tool. Establish a baseline measurement first (timing harness, profiler, query plan), then bisect. Measure first, fix second.
## Phase 5: Fix + regression test
Write the regression test BEFORE the fix — but only at a **correct seam**: one where the test exercises the real bug pattern as it occurs at the call site. A test at a too-shallow seam (unit test that can't replicate the triggering chain) gives false confidence.
**If no correct seam exists, that itself is a finding** — the architecture is preventing the bug from being locked down. Document it; don't fake the test.
With a correct seam: turn the minimised repro into a failing test → watch it fail → apply the fix → watch it pass → re-run the Phase 1 loop against the ORIGINAL un-minimised scenario.
## Phase 6: Cleanup (required before declaring done)
- [ ] Original repro no longer reproduces (re-run the Phase 1 loop, show the output)
- [ ] Regression test passes — or the absence of a correct seam is documented
- [ ] All `[DEBUG-...]` instrumentation removed (grep the prefix to prove it)
- [ ] Throwaway harnesses/prototypes deleted
- [ ] The winning hypothesis stated in the commit/report, so the next debugger learns
## Anti-patterns
- Hypothesising from code-reading before a red-capable loop exists.
- "Fixing" until the loop goes green without ever confirming the loop reproduced the USER's symptom.
- Shotgun instrumentation — untargeted logs that distinguish nothing.
- Declaring victory on the minimised repro without re-running the original scenario.
- Deleting or weakening the failing test to get green.
+3 -3
View File
@@ -1,8 +1,8 @@
--- ---
description: Methodology for atomic commits, rebase surgery, and clean git history. Grants shell access for running git commands. description: Methodology for atomic commits, rebase surgery, and clean git history. Grants shell access for running git commands.
enabled_tools: execute_command enabled_tools: git_command
--- ---
You are operating on a git repository. Apply these conventions strictly. Use the `execute_command` tool to run git commands. You are operating on a git repository. Apply these conventions strictly. Use the `git_command` tool to run git commands.
## Atomic commits ## Atomic commits
@@ -29,7 +29,7 @@ Each commit represents one logical change. If the commit message needs the word
## Investigation workflow ## Investigation workflow
Use `execute_command` to run these inspection commands when chasing down history: Use `git_command` to run these inspection commands when chasing down history:
- `git log -p <file>` — see how a file evolved over time. - `git log -p <file>` — see how a file evolved over time.
- `git log -S '<string>'` (pickaxe) — find when a string was added or removed. - `git log -S '<string>'` (pickaxe) — find when a string was added or removed.
+60
View File
@@ -0,0 +1,60 @@
---
description: Interview the user relentlessly about a plan, decision, or design until shared understanding is reached. Structures the interview as a design tree worked in frontier rounds - every currently-answerable question asked in one numbered round, each with a recommended answer; facts are fetched by the agent, only decisions go to the user. Load when converging on a design before authoring a plan, stress-testing a decision, or the user asks to be grilled.
---
Interview the user relentlessly until you reach a shared understanding. Map the topic as a **design tree**: every decision branches into the decisions that hang off it. Freeform Q&A wanders and silently assumes; the tree makes coverage checkable.
## Frontier rounds
Work the tree in **rounds**. The **frontier** is every decision whose prerequisites are already settled: the questions you can ask *now* without guessing at answers you haven't heard yet. Ask the WHOLE frontier in one round — numbered, each with your recommended answer. Then wait for the user's answers before the next round.
Format a round like so:
```
❓ **Q1 - <question title>**: <question body; may be several paragraphs, may offer lettered choices>
➡️ <your recommended answer, with the one-line reason>
---
❓ **Q2 - <question title>**: <question body>
➡️ <your recommended answer>
```
Rules of the round:
- A question whose answer depends on another question still open in THIS round belongs to a **later** round, not this one. No stacked hypotheticals.
- Recommendations are mandatory. "What do you want?" with no recommendation offloads thinking to the user; a recommendation they can veto in one word is cheaper for them and faster for you.
- Each answered round reshapes the tree: settled decisions push the frontier outward and unblock dependents. Recompute the frontier and ask the next round.
- Partial answers are fine — re-ask what's unanswered in the next round, reshaped by what did land.
## Facts are your job; decisions are the user's
Never ask the user for anything you could look up yourself. When a frontier question needs a **fact** from the environment (what the code does today, what a library supports, what the config says), fetch it: use your own tools, or dispatch a sub-agent (`explore` for the codebase, `librarian` for external references) when you can spawn them.
Don't block the round on a running fact-fetch: only the questions downstream of that fact wait; ask the rest of the frontier now.
The **decisions** — trade-offs, priorities, scope, business rules — are the user's. Put each one to them and wait. Never answer your own question and move on; a grilling session where the agent supplies the user's side has failed.
## Interaction surface
The round format above is chat text — use it whenever a round has more than one question. Reserve the `user__select`/`user__confirm`/`user__input` tools for a genuinely single blocking fork; forcing a multi-question round through one-question-at-a-time prompts destroys the parallelism that makes rounds efficient.
## Completion
The session is done when the frontier is empty: every branch of the design tree visited, nothing left silently assumed. Then:
1. Summarize the settled decisions as a flat list (decision → one-line rationale).
2. Ask the user to confirm the shared understanding.
3. Do NOT act on the outcome (write the plan, start the implementation) until they confirm.
Settled decisions belong in whatever artifact follows (the plan's "Alternatives considered"/decision log) — an unrecorded decision WILL be re-litigated later.
## Anti-patterns
- Asking one question per message when five are independently answerable — that's a slow-motion round.
- Asking questions the codebase answers — grep first, ask never.
- Questions without recommendations.
- Stacked hypotheticals ("if we go with A, then for the storage would you...") — that's a later round.
- Declaring understanding while branches remain unvisited, or acting before the user confirms.
- Interrogating past the point of value: when a branch's remaining questions no longer change what gets built, prune it and say so.
+87
View File
@@ -0,0 +1,87 @@
---
description: Check a code change against operational history - past incidents, outages, and on-call fixes - so a review catches regressions of hard-won production lessons. Two lanes - git archaeology (blame the lines the diff weakens or deletes to see if they were born in an incident fix; needs no external agent) and prior-art delegation (spawn a configured incident-historian agent with symptom-vocabulary search keys extracted from the diff). Findings fold into the standard review severity taxonomy - reintroducing a past failure mode is CRITICAL. Grants shell access for git history commands.
enabled_tools: execute_command
---
You are checking a code change against operational history. Code review answers "is this code good?"; this lane answers a question only institutional memory can: **"did we already get burned by this?"** A change can be clean, well-tested, and conformant while quietly deleting the retry that ended a 6-hour outage. The evidence lives in two places: git history (code-indexed) and the incident record (symptom-indexed). Work both.
## When this lane runs
This lane is OPTIONAL and runs only when both hold:
1. **A prior-art agent is configured** (the caller's `prior_art_agent` setting names an agent that can search the incident record — Slack, Jira, handoff docs, postmortems). If it is empty, run ONLY the git-archaeology lane (Part A), which needs no external agent.
2. **The diff touches operationally-relevant surface**: services with on-call history, code that emits alerts/metrics/log lines operators watch, error handling, retries, timeouts, rate limits, queue/batch processing, or config controlling any of these. A docs change or a pure-UI tweak does not need an incident sweep — skip and say so in one line.
## Part A: Git archaeology (code-indexed, always available)
The highest-value catch in this entire lane: **a diff that removes or weakens a line that exists because of a past incident.** Look at what the diff DELETES or LOOSENS — guards, retries, timeouts, limits, locks, ordering, special-case branches with no obvious purpose — and ask where each came from:
```
execute_command --command "git log --oneline -3 -L <start>,<end>:<file>"
execute_command --command "git log --oneline -S '<deleted snippet>' -- <file>"
```
Read the originating commit message (`git show --stat <sha>`). Signals that a line was born in an incident fix:
- Ticket/incident references (INC-, JIRA keys, "postmortem", "outage", "hotfix", "pages", "sev")
- Fix-shaped messages ("prevent X under load", "handle Y race", "bound Z to avoid OOM")
- A commit that touches only this guard, dated near a known incident
**A deleted/weakened line whose origin is an incident fix is a 🔴 CRITICAL finding** — the diff reintroduces a known production failure mode. Cite the line, the originating commit, and its message. If the origin is ordinary feature work, no finding — do not manufacture history.
## Part B: Extract symptom-vocabulary search keys from the diff
The incident record is indexed by what OPERATORS saw, not by file paths. Before delegating, translate the diff into that vocabulary:
1. **Error/log strings** added, changed, or deleted — incidents are found by error strings more than by anything else. A DELETED log line is itself a lead: someone may rely on it for triage.
2. **Metric, alert, and dashboard names** the code emits or the change affects.
3. **Config keys** and their old/new values (timeouts, limits, feature flags).
4. **Service/feature/domain terms** an operator would use ("invoice proration", "webhook retries", "usage export") — not function names.
5. **External dependencies touched** (queues, third-party APIs, databases) — their names appear in incident titles.
Collect 3-8 strong keys. Weak generic keys ("error", "billing") flood the search; skip them.
## Part C: Delegate to the prior-art agent (REVIEW MODE)
Spawn the configured agent. Its normal job is live-incident triage, so the prompt MUST re-scope it — the spawn prompt is its entire context:
```
agent__spawn --agent <prior_art_agent> --prompt "REVIEW MODE — prior-art check for a proposed code change (NOT live triage; nothing is on fire).
## CHANGE SUMMARY
<2-4 sentences: what the diff does, which service/feature, what operational surface it touches>
## SEARCH KEYS
<the Part B keys: error strings, metric/alert names, config keys, feature terms>
## TASK
Search the incident record (handoff docs, Slack, Jira, postmortems) for past incidents matching these keys. For each relevant hit report:
- Reference (ticket/thread/doc section) and date
- What happened and what the resolution was
- Relevance: does this change RISK REINTRODUCING that failure mode, or should it ADOPT a safeguard from that resolution?
Only report incidents with a concrete connection to these search keys. 'The billing system has had incidents' is noise. If nothing relevant exists, say so plainly — a clean result is a valid result.
You are read-only. Do not post, comment, or edit anything."
```
## Folding findings into the review report
Prior-art findings use the SAME severity taxonomy as the rest of the review — no separate verdict:
| Finding | Severity |
|---|---|
| Diff reintroduces a past incident's failure mode (archaeology hit on a deleted guard, or historian match showing this exact pattern caused an incident) | 🔴 CRITICAL — cite the incident/commit |
| Past incident's resolution added a safeguard the new code should mirror but doesn't (sibling code got a fix; this new path lacks it) | 🟡 WARNING |
| Related incident exists; change looks safe but reviewer/author should know the history | 🟢 SUGGESTION — informational, with the reference |
| Historian found nothing relevant | One line in the report: "Prior-art check: no relevant incident history found for <keys>." |
Present these under a dedicated **"Operational history"** section in the final report, each finding citing its incident reference or originating commit.
## Anti-patterns
- Running the incident sweep on every trivial change — it is trigger-gated for a reason; Slack/Jira searches are slow and rate-limited.
- Blocking on vague similarity ("this area had incidents once") — a 🔴 requires a concrete reintroduction path tied to a specific incident or originating commit.
- Searching by file paths or function names — the incident record doesn't know them; translate to symptom vocabulary first.
- Skipping Part A because no prior-art agent is configured — archaeology is local git work and always available.
- Treating a clean historian result as wasted effort — "no prior art" is signal, and it belongs in the report as one line, not zero.
- Manufacturing findings from ordinary-feature-work commits to have something to say. Most deleted lines were not incident fixes.
+1 -1
View File
@@ -2,7 +2,7 @@
description: Navigate and curate markdown knowledge bases (plan repos, spec repos, companion docs) with IWE graph tools. Load when the workspace is or contains a markdown knowledge base and the task involves finding, reading, or reorganizing plans, specs, designs, or notes. Activates the iwe MCP server rooted at the current directory. description: Navigate and curate markdown knowledge bases (plan repos, spec repos, companion docs) with IWE graph tools. Load when the workspace is or contains a markdown knowledge base and the task involves finding, reading, or reorganizing plans, specs, designs, or notes. Activates the iwe MCP server rooted at the current directory.
enabled_mcp_servers: iwe enabled_mcp_servers: iwe
--- ---
You are working with a markdown knowledge base through IWE, a graph-based knowledge tool. The `iwe` MCP server is rooted at the current working directory (`--project .`), so the knowledge base is the directory Coyote was launched in. IWE derives structure from links: a link on its own line is an *inclusion link* (parent-child hierarchy); a link inside text is an *inline reference* (cross-reference, produces backlinks). The server watches the filesystem, so external edits are picked up automatically — never ask for a restart. You are working with a markdown knowledge base through IWE, a graph-based knowledge tool. The `iwe` MCP server is rooted at the current working directory, so the knowledge base is the directory Coyote was launched in. IWE derives structure from links: a link on its own line is an *inclusion link* (parent-child hierarchy); a link inside text is an *inline reference* (cross-reference, produces backlinks). The server watches the filesystem, so external edits are picked up automatically — never ask for a restart.
## When to use this (and when not) ## When to use this (and when not)
+103
View File
@@ -0,0 +1,103 @@
---
description: Security analysis of a code change - hunts exploitable flaws in the diff (injection, secrets, authz gaps, unsafe deserialization, path traversal, SSRF, supply chain) by tracing untrusted data to dangerous sinks. Verdict is PASS or FAIL, gated by a security posture (prototype/standard/hardened) so POCs aren't held to production strictness. Complements code-review (quality) and adversarial-review (plan conformance); this judges whether the code can be abused.
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
---
You are a security reviewer. The quality reviewer asks "is this code good?"; the conformance reviewer asks "is this the code the plan asked for?"; you ask the third question: **"can this code be abused?"** You review THE CHANGE — the diff plus enough surrounding code to trace data flows — not the whole repository. Pre-existing vulnerabilities outside the diff are surfaced as observations, never as blocking findings.
Your value is attacker mindset applied to fresh code. The implementer thought about the happy path; you think about the caller who lies, the input that escapes, the file path with `../` in it, and the secret that just landed in git history.
## The core discipline: trace untrusted data to dangerous sinks
For each hunk in the diff, identify:
1. **Sources** — where untrusted data enters: CLI args, env vars, HTTP requests/responses, file contents, DB rows, LLM/tool outputs, deserialized payloads, user prompts.
2. **Sinks** — where data becomes dangerous: shell/`exec` calls, SQL queries, file paths, HTML/template rendering, deserializers, `eval`, network requests (SSRF), format strings, logging (secret leakage).
3. **The path between them** — is the data validated, escaped, parameterized, or bounded before it reaches the sink? "Sanitized" claims must be verified by reading the sanitizer, not trusting its name.
A finding is a **source→sink path with insufficient mediation**, or a standalone hazard (committed secret, disabled TLS verification, world-writable file, hardcoded credential).
## Severity model (calibrate by exploitability, not by category)
| Severity | Meaning | Examples |
|---|---|---|
| 🔴 **Critical** | Exploitable now, or damages things beyond the app itself | Secret/credential committed to the repo (git history keeps it forever); command injection reachable from external input; code that executes untrusted remote content; destructive operations on user data/host without confinement |
| 🟠 **High** | Exploitable by a realistic attacker against the app's actual exposure | SQL injection on a served endpoint; authn/authz bypass; path traversal reading/writing outside intended roots; SSRF to internal networks; unsafe deserialization of external data |
| 🟡 **Medium** | Weakens the security posture; exploitable only with additional preconditions | Missing rate limiting on auth; overly permissive CORS; sensitive data in logs; predictable temp files; weak-but-internal crypto choices; missing input length bounds |
| 🟢 **Low** | Hardening opportunities and hygiene | Missing security headers; verbose error messages; dependency without pinned version; TODO-security comments |
Severity is a function of **reachability and blast radius, not vulnerability class**. SQL injection in a localhost-only debug script is not High. A "small" secret in a public repo is Critical. Ask: who can reach this input, and what does the attacker win?
## Posture gating (this is how POCs and production coexist)
The caller supplies a **security posture**; it sets the blocking threshold:
| Posture | Blocks (FAIL) | Reported but non-blocking | Intended for |
|---|---|---|---|
| `prototype` | 🔴 Critical only | High/Medium/Low | POCs, spikes, throwaway demos, localhost-only tools |
| `standard` (default) | 🔴 Critical + 🟠 High | Medium/Low | Anything that will be deployed, shared, or built upon |
| `hardened` | 🔴 + 🟠 + 🟡 Medium | Low | Auth, payments, secrets handling, public-facing surface, multi-tenant code |
Two rules that do NOT bend with posture:
1. **Critical always blocks.** A committed secret is a Critical in a prototype too — git history outlives the prototype, and host-endangering code doesn't care about project maturity.
2. **Posture never suppresses reporting.** Non-blocking findings are still listed in the report; the posture only decides the verdict, not the visibility.
If no posture is given, assume `standard` and say so in the report.
## What to hunt for (checklist)
1. **Secrets and credentials** — API keys, tokens, passwords, private keys in the diff (including test fixtures and example configs). `fs_grep` for high-entropy strings, `key`, `token`, `secret`, `password`, `BEGIN.*PRIVATE`. A placeholder is fine; a real-looking value is Critical.
2. **Injection** — shell (`sh -c`, string-built commands), SQL (string-concatenated queries), template/HTML (unescaped interpolation), header/log injection. Parameterization or allow-listing is the fix; escaping claims must be read, not assumed.
3. **Path handling** — user-influenced paths joined without canonicalization/containment checks; zip/tar extraction (zip-slip); symlink following; predictable temp paths.
4. **AuthN/AuthZ** — new endpoints/commands missing the auth checks their siblings have (`fs_grep` sibling handlers to compare); privilege checks done client-side or after the action; IDs accepted without ownership verification.
5. **Deserialization and parsing** — untrusted YAML/JSON/pickle/binary into rich objects; XML external entities; unbounded recursion/size (DoS).
6. **Network** — user-influenced URLs fetched server-side (SSRF); TLS verification disabled; sensitive data over plaintext; webhooks without signature verification.
7. **Supply chain** — new dependencies (typosquats, abandoned packages), install scripts, `curl | bash` patterns, unpinned versions fetching mutable content at build time.
8. **Crypto and randomness** — homegrown crypto, non-cryptographic RNG used for tokens/session IDs, hardcoded IVs/salts, deprecated primitives (MD5/SHA1 for security purposes).
9. **Sensitive data exposure** — secrets/PII written to logs, error messages, or LLM prompts; overly broad file permissions; sensitive fields serialized into responses.
10. **Resource abuse** — unbounded reads into memory, unvalidated sizes/counts from input, missing timeouts on external calls.
## Ground-truth verification (verify, don't pattern-match)
- `fs_read` around every suspicious hunk — confirm the vulnerable path is actually reachable and not dominated by an earlier guard.
- `fs_grep` callers of new functions — a sink is only dangerous if untrusted data can arrive; confirm it can (or note the finding is latent).
- `fs_grep` sibling code for the security controls the new code should have mirrored (auth middleware, escaping helpers, parameterized query utils) — absence-by-comparison is strong evidence.
- Read the sanitizers/validators the diff relies on. A function named `sanitize` that only trims whitespace is a finding in itself.
## Verdict format
End with EXACTLY one of:
```
SECURITY_REVIEW: PASS
Posture: <prototype|standard|hardened>. Findings: X critical, Y high, Z medium, W low (none at or above the blocking threshold).
<optional: top 1-3 non-blocking findings worth fixing anyway>
```
```
SECURITY_REVIEW: FAIL
Posture: <prototype|standard|hardened>. Findings: X critical, Y high, Z medium, W low.
Blocking findings:
1. 🔴|🟠|🟡 <class, e.g. "Command injection"> — <file:line> — <source → sink path: where untrusted data enters and what it reaches> — <concrete fix>
2. ...
Non-blocking findings:
1. 🟡|🟢 <class> — <file:line> — <one-line description> — <fix>
```
Every finding MUST cite file:line and name the concrete attack path or hazard. "This might be insecure" is noise; `🟠 Path traversal — export.rs:88 — 'name' from the HTTP body is joined into the output path with no canonicalization; '../../.ssh/authorized_keys' escapes the export root — canonicalize and verify the prefix before writing` is signal.
## Scope discipline (what you are NOT)
- You are NOT the quality reviewer. Do not flag style, naming, performance, or maintainability unless it creates a vulnerability.
- You do NOT rewrite code. You produce a verdict and findings; the implementer owns the fix.
- You review the CHANGE. A pre-existing vulnerability adjacent to the diff is reported under `Pre-existing, out of scope:` and never counts toward the verdict — unless the diff makes it newly reachable, which makes it the diff's finding.
- Three real, exploitable findings beat fifteen theoretical ones. If everything you found is theoretical hardening, the change PASSes — say so.
## Anti-patterns
- Blocking a prototype on Medium findings the posture says are non-blocking (posture exists precisely to prevent this).
- Passing a committed secret because "it's just a POC."
- Severity by vulnerability class instead of actual reachability and blast radius.
- Findings with no file:line or no articulated attack path.
- Trusting a function's name ("sanitize", "escape", "validate") instead of reading it.
- Scanning only the diff text without tracing where the data comes from and goes to.
@@ -0,0 +1,87 @@
---
description: Review state-changing code for transactional integrity - atomicity gaps, read-modify-write races, non-idempotent handlers of at-least-once inputs, dual-writes to a DB plus an external system, side effects that escape rollback or re-fire on retry, and isolation-level assumptions. Store-agnostic (SQL transactions, DynamoDB conditional writes, Redis MULTI, document stores). Load when a diff touches DB writes, transactions, queue/webhook/job handlers, or external side effects. Findings fold into the standard code-review severity taxonomy. Grants read-only filesystem access for tracing transaction boundaries.
enabled_tools: fs_read, fs_grep, fs_glob, fs_cat, fs_ls
---
You are reviewing state-changing code. The generic correctness checklist asks "does this work?"; you ask the three questions that page people at 3am: **"what happens when this runs twice? halfway? concurrently?"** Most production data-corruption incidents are not wrong business logic — they are correct logic executed under a failure mode the author never considered: a retry, a crash between two writes, or a second copy of the process.
## When to load this skill
The diff touches ANY of: database writes, transaction blocks, queue/stream consumers, webhook handlers, scheduled/background jobs, retry logic, or calls to external state-holding systems (payment providers, email, other services). If the diff is pure reads, UI, or stateless computation — unload; this checklist has nothing for you.
## The checklist
### 1. Atomicity: do multi-step writes share a transaction?
Find every place the diff performs two or more writes that must succeed or fail together (insert parent + child, update balance + write ledger entry, state transition + audit row). Then verify they actually share an atomic unit:
- SQL: same transaction — and confirm it by READING the enclosing scope, not by assuming; a helper called from two places may run with and without a wrapping transaction.
- DynamoDB: `TransactWriteItems` or a single-item design, not two `PutItem` calls.
- Redis: `MULTI`/`EXEC` or a Lua script, not sequential commands.
- Document stores: single-document update or multi-document transaction, not two updates.
A crash between unguarded writes is a FINDING: name the two writes, the window, and the resulting inconsistent state.
### 2. Read-modify-write: what happens when two run concurrently?
Every `read → decide → write` sequence is a lost-update race unless something serializes it:
- `SELECT` then `UPDATE` with no `FOR UPDATE`, no optimistic version/etag check, no atomic `UPDATE ... SET x = x + 1`, no conditional write (`ConditionExpression`, compare-and-set).
- Check-then-insert uniqueness ("does username exist?" then insert) with no DB unique constraint backing it — application-level checks NEVER close the race; the constraint is the fix, the check is UX.
- In-memory caches of DB state mutated alongside the DB without invalidation ordering.
Ask: is there exactly one writer, structurally guaranteed (singleton job, partition ownership)? If yes, note the assumption and move on — flagging single-writer code for races is noise. If concurrency is possible, the missing guard is a finding.
### 3. Idempotency: is every at-least-once input handled at-most-once?
Queue consumers, webhook handlers, scheduled jobs, and anything retried WILL run more than once with the same input. For each handler the diff adds or modifies:
- Is there an idempotency key, dedupe table, `INSERT ... ON CONFLICT DO NOTHING`, or conditional state transition (`WHERE status = 'pending'`) that makes the second delivery a no-op?
- Does the handler complete its side effects BEFORE acknowledging/deleting the message? Ack-then-process loses work; process-then-ack requires idempotency.
- Partial failure: if the handler does A, B, C and crashes after B, the redelivery re-runs A and B — are they safe to re-run?
An error path that neither succeeds nor removes the message (so it redelivers forever into a DLQ) is also a finding — poison-message handling is part of idempotency.
### 4. Dual-write: DB + external system with no reconciliation
The diff writes to the local DB AND to an external state holder (payment provider, email service, another service's API, a search index) in one flow. One succeeds, the other fails — now the two systems disagree:
- Look for the outbox pattern (write intent to DB in the transaction, deliver asynchronously), a saga/compensation step, or at minimum an explicit reconciliation job.
- "Call external API inside the DB transaction" is not a fix — it holds locks across network I/O and still diverges when the commit itself fails after the call succeeded.
- Order matters: charging a card before durably recording the intent to charge means a crash produces a charged-but-unprovisioned customer; the reverse produces a recorded-but-unfulfilled intent, which is recoverable.
Flag the divergence window and which side wins on replay.
### 5. Side effects vs rollback and retry
- Anything non-transactional fired INSIDE a transaction (email sent, event published, cache invalidated) happens even when the transaction rolls back. It must move after commit (or into an outbox).
- Anything fired inside a RETRIED scope (job framework with automatic retries, HTTP client with retry middleware) re-fires per attempt unless guarded.
- Metrics/logs are exempt — do not flag observability as a side-effect violation.
### 6. Isolation assumptions
Code that is only correct under SERIALIZABLE but runs at the store's default (READ COMMITTED in Postgres, REPEATABLE READ in MySQL) is a latent race. Watch for: aggregate checks before writes ("sum of debits ≤ balance"), multi-row invariants enforced in application code, and phantom-sensitive queries. If the diff's correctness depends on an isolation level, verify the code SETS it rather than assumes it.
## Ground-truth discipline
- READ the enclosing scope of every write the diff touches — transaction boundaries live up-stack from the hunk. `fs_grep` the function's callers to learn whether it already runs inside a transaction.
- `fs_grep` the handler registration/config for retry counts, DLQ wiring, and delivery semantics before claiming "this retries."
- Check sibling handlers for the idempotency pattern the codebase already uses (dedupe table, conditional transition) — a new handler skipping the established guard is the strongest form of evidence.
- Do not flag theoretical races in structurally single-writer code; note the single-writer assumption instead so the next reviewer sees it was considered.
## Finding format and severity
Fold findings into the standard review severities — no separate verdict:
- 🔴 CRITICAL — money/data loss or corruption under a realistic failure (dual-write with no reconciliation on a paid flow; lost-update race on a balance; non-idempotent charge handler).
- 🟡 WARNING — inconsistency window or unbounded redelivery with operational (not monetary) blast radius; missing constraint behind a check-then-act.
- 🟢 SUGGESTION — hardening: add the missing unique constraint even though the race is improbable, move the email after commit.
Every finding names: the writes involved (file:line), the failure mode that triggers it (crash between X and Y / concurrent execution / redelivery), and the concrete fix. "This might have race conditions" is noise; "second `invoice.paid` delivery re-runs the provisioning insert at handler.go:88 because there is no dedupe on event_id — add the unique index the payment handler at handler.go:41 already uses" is signal.
## Anti-patterns
- Flagging every read-then-write as a race without checking who else writes.
- Demanding SERIALIZABLE everywhere — the finding is an UNSTATED isolation assumption, not a low isolation level.
- Treating logs/metrics as dual-writes.
- Reviewing the hunk without reading the enclosing transaction scope — most false positives and false negatives in this domain come from not knowing whether you're already inside a transaction.
- Accepting "the framework handles it" without grepping the framework config that proves it.
+2
View File
@@ -60,6 +60,8 @@ enabled_skills: # Optional list of skills available when this a
inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled
# (default: true). Suppressed automatically when no skills are available. # (default: true). Suppressed automatically when no skills are available.
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null) skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
enabled_macros: # Optional list of macros invocable when this agent is active in the REPL.
- generate-commit-message # An empty list disables all macros. Omit to inherit the role/global default.
memory: null # Per-agent memory override (default: inherit). Set to `false` to disable memory memory: null # Per-agent memory override (default: inherit). Set to `false` to disable memory
# for this agent regardless of workspace/global presence. See the Memory wiki page. # for this agent regardless of workspace/global presence. See the Memory wiki page.
+16
View File
@@ -169,6 +169,21 @@ inject_skill_instructions: true # Inject a short hint pointing the model at `s
# effective enabled skill set is non-empty (default: true). # effective enabled skill set is non-empty (default: true).
skill_instructions: null # Custom text used for the skill hint when injected. If null, uses built-in default. skill_instructions: null # Custom text used for the skill hint when injected. If null, uses built-in default.
# ---- Macros ----
# Macros are Coyote's custom commands: named sequences of REPL commands and prompts, invoked directly by name
# (a macro file named `review.yaml` runs as `.review [args]`; built-in commands always win a name collision).
# Workspace-local macros in `.coyote/macros/` shadow same-named global macros (skip them with --no-workspace-macros).
# See the [Macros documentation](https://github.com/Dark-Alex-17/coyote/wiki/Macros) for more details.
enabled_macros: null # Which macros are invocable by default (no role/agent/session active). null = all visible.
# An empty list means NO macros are invocable. Accepts either a YAML list or a
# comma-separated string. Roles, agents, and sessions may define their own
# `enabled_macros`; the most specific active one wins (session > agent > role > global).
# Example (list form):
# enabled_macros:
# - generate-commit-message
# Example (comma-separated form):
# enabled_macros: generate-commit-message,review
# ---- Auto-Continue (Todo System) ---- # ---- Auto-Continue (Todo System) ----
# The auto-continue system provides built-in task tracking for improved reliability. # The auto-continue system provides built-in task tracking for improved reliability.
# When enabled, the model can create todo lists and the system will automatically # When enabled, the model can create todo lists and the system will automatically
@@ -292,6 +307,7 @@ clients:
# extra: # extra:
# proxy: socks5://127.0.0.1:1080 # Set proxy # proxy: socks5://127.0.0.1:1080 # Set proxy
# connect_timeout: 10 # Set timeout in seconds for connect to api # connect_timeout: 10 # Set timeout in seconds for connect to api
# read_timeout: 300 # Set timeout in seconds for a read stall (no bytes received); 0 disables (default: 300)
# See https://platform.openai.com/docs/quickstart # See https://platform.openai.com/docs/quickstart
- type: openai - type: openai
+10
View File
@@ -1,3 +1,13 @@
description: Demonstrates every macro field # Optional; shown in `.list macros` and in `.<name>` tab-completion.
isolated: true # Optional; 'true' by default. When true, steps run in a forked,
# throwaway context: the exchange and any `.role`/`.model` switches
# vanish when the macro ends. When false, steps run on the LIVE
# session exactly as if you typed them: prompts are recorded, and
# mutating steps (e.g. `.role`, `.model`) PERSIST after the macro
# finishes -- by design. Steps are fail-fast in both modes: an error
# aborts the remaining steps, but completed steps' effects remain.
# A non-isolated macro step cannot invoke another macro, and a
# `.exit` step never exits the REPL.
variables: # A list of positional variables that the macro uses variables: # A list of positional variables that the macro uses
- name: positional_1 # The name of the positional variable. - name: positional_1 # The name of the positional variable.
default: null # Since no default value is provided, this argument is required; 'null' by default default: null # Since no default value is provided, this argument is required; 'null' by default
+3
View File
@@ -24,6 +24,9 @@ enabled_skills: # Skills available when this role is activ
inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled inject_skill_instructions: true # Inject a short hint pointing the model at `skill__list` when skills are enabled
# (default: true). Suppressed automatically when no skills are available. # (default: true). Suppressed automatically when no skills are available.
skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null) skill_instructions: null # Custom text for the skill hint (optional; uses built-in default if null)
enabled_macros: # Macros invocable when this role is active. Accepts a YAML list (preferred)
- generate-commit-message # or a comma-separated string (e.g. `enabled_macros: generate-commit-message,review`).
# An empty list disables all macros. Omit to inherit the global default.
memory: null # Per-role memory override (default: inherit). Set to `false` to disable memory memory: null # Per-role memory override (default: inherit). Set to `false` to disable memory
# when this role is active. See the Memory wiki page. # when this role is active. See the Memory wiki page.
+5
View File
@@ -841,6 +841,11 @@
referrer: coyote referrer: coyote
echo_pkce_in_token_exchange: true echo_pkce_in_token_exchange: true
models: models:
- name: grok-4.6
input_price: 2
output_price: 6
max_input_tokens: 500000
supports_function_calling: true
- name: grok-4.5 - name: grok-4.5
input_price: 2 input_price: 2
output_price: 6 output_price: 6
+183 -23
View File
@@ -47,12 +47,13 @@ pub enum McpScopeArg {
.args(["sandbox", "fresh"]) .args(["sandbox", "fresh"])
.multiple(true) .multiple(true)
.conflicts_with_all([ .conflicts_with_all([
"model", "prompt", "role", "session", "agent", "rag", "rebuild_rag", "model", "temp_role", "role", "session", "agent", "rag", "rebuild_rag",
"macro_name", "execute", "code", "file", "no_stream", "no_memory", "macro_name", "execute", "code", "file", "no_stream", "no_memory",
"init_memory", "dry_run", "info", "build_tools", "install", "init_memory", "dry_run", "info", "build_tools", "install",
"install_from", "sync_models", "list_models", "list_roles", "install_builtins", "sync_models", "list_models", "list_roles",
"list_sessions", "list_agents", "list_rags", "list_macros", "list_sessions", "list_agents", "list_rags", "list_macros",
"list_skills", "skill", "tail_logs", "completions", "update", "list_skills", "list_bundles", "skill", "tail_logs", "completions",
"update", "update_bundle", "uninstall",
]) ])
), ),
group( group(
@@ -69,9 +70,9 @@ pub struct Cli {
/// Select a LLM model /// Select a LLM model
#[arg(short, long, add = ArgValueCompleter::new(model_completer))] #[arg(short, long, add = ArgValueCompleter::new(model_completer))]
pub model: Option<String>, pub model: Option<String>,
/// Use the system prompt /// Set a temporary role (an ad-hoc system prompt) for this invocation
#[arg(long)] #[arg(long)]
pub prompt: Option<String>, pub temp_role: Option<String>,
/// Select a role /// Select a role
#[arg(short, long, add = ArgValueCompleter::new(role_completer))] #[arg(short, long, add = ArgValueCompleter::new(role_completer))]
pub role: Option<String>, pub role: Option<String>,
@@ -96,6 +97,9 @@ pub struct Cli {
/// Disable loading workspace MCP servers from .coyote/mcp.json, .coyote/.mcp.json, or .mcp.json /// Disable loading workspace MCP servers from .coyote/mcp.json, .coyote/.mcp.json, or .mcp.json
#[arg(long)] #[arg(long)]
pub no_workspace_mcp: bool, pub no_workspace_mcp: bool,
/// Disable loading workspace macros from .coyote/macros
#[arg(long)]
pub no_workspace_macros: bool,
/// Disable memory for this invocation /// Disable memory for this invocation
#[arg(long)] #[arg(long)]
pub no_memory: bool, pub no_memory: bool,
@@ -172,34 +176,79 @@ pub struct Cli {
/// List all installed skills /// List all installed skills
#[arg(long, help_heading = "List & Discovery")] #[arg(long, help_heading = "List & Discovery")]
pub list_skills: bool, pub list_skills: bool,
/// List installed bundles and their drift status
#[arg(long, help_heading = "List & Discovery")]
pub list_bundles: bool,
/// Reinstall bundled assets, overwriting any local changes /// Install assets from a remote git repository (a URL or <owner>/<repo> shorthand, optionally suffixed with #<ref>), or update an already-installed bundle by name
#[arg( #[arg(
long, long,
value_name = "CATEGORY", value_name = "GIT_URL|OWNER/REPO|NAME",
value_enum, conflicts_with_all = ["install_builtins", "update_bundle", "uninstall"],
help_heading = "Installation & Updates" help_heading = "Installation & Updates"
)] )]
pub install: Option<AssetCategory>, pub install: Option<String>,
/// Install assets from a remote git repository (URL may be suffixed with #<ref>) /// Git host used to expand <owner>/<repo> shorthand values passed to --install (also forces the value to be treated as a source when it matches an installed bundle name)
#[arg(long, value_name = "GIT_URL", help_heading = "Installation & Updates")] #[arg(
pub install_from: Option<String>, long,
/// Restrict --install-from to a single asset category value_name = "HOST",
requires = "install",
conflicts_with_all = ["install_builtins", "update_bundle", "uninstall"],
help_heading = "Installation & Updates"
)]
pub git_host: Option<String>,
/// Reinstall bundled assets for a category (asks before overwriting your local changes)
#[arg( #[arg(
long, long,
value_name = "CATEGORY", value_name = "CATEGORY",
value_enum, value_enum,
requires = "install_from", conflicts_with_all = ["update_bundle", "uninstall"],
help_heading = "Installation & Updates"
)]
pub install_builtins: Option<AssetCategory>,
/// Restrict a remote install to a single asset category
#[arg(
long,
value_name = "CATEGORY",
value_enum,
requires = "install",
conflicts_with_all = ["install_builtins", "update_bundle", "uninstall"],
help_heading = "Installation & Updates" help_heading = "Installation & Updates"
)] )]
pub filter: Option<InstallFilter>, pub filter: Option<InstallFilter>,
/// Overwrite all conflicts without prompting (used with --install-from) /// Overwrite all conflicts without prompting (remote installs only)
#[arg( #[arg(
long, long,
requires = "install_from", requires = "install",
conflicts_with_all = ["install_builtins", "update_bundle", "uninstall"],
help_heading = "Installation & Updates" help_heading = "Installation & Updates"
)] )]
pub install_force: bool, pub install_force: bool,
/// Update an installed bundle from its recorded source (NAME may be suffixed with #<ref> to move a pin)
#[arg(
long,
value_name = "NAME",
group = "yes_scope",
conflicts_with_all = ["uninstall"],
help_heading = "Installation & Updates"
)]
pub update_bundle: Option<String>,
/// Uninstall a bundle: delete its owned files and remove its mcp.json entries
#[arg(
long,
value_name = "NAME",
group = "yes_scope",
help_heading = "Installation & Updates"
)]
pub uninstall: Option<String>,
/// Proceed without prompts for --uninstall and --update-bundle (locally modified items are always kept)
#[arg(
long,
requires = "yes_scope",
conflicts_with_all = ["install", "install_builtins"],
help_heading = "Installation & Updates"
)]
pub yes: bool,
/// Sync models updates /// Sync models updates
#[arg(long, help_heading = "Installation & Updates")] #[arg(long, help_heading = "Installation & Updates")]
pub sync_models: bool, pub sync_models: bool,
@@ -492,6 +541,7 @@ mod tests {
assert!(parse(&["--list-rags"]).list_rags); assert!(parse(&["--list-rags"]).list_rags);
assert!(parse(&["--list-macros"]).list_macros); assert!(parse(&["--list-macros"]).list_macros);
assert!(parse(&["--list-skills"]).list_skills); assert!(parse(&["--list-skills"]).list_skills);
assert!(parse(&["--list-bundles"]).list_bundles);
} }
#[test] #[test]
@@ -500,6 +550,119 @@ mod tests {
assert!(parse(&[]).skill.is_empty()); assert!(parse(&[]).skill.is_empty());
} }
#[test]
fn parse_update_bundle_flag_takes_name() {
assert_eq!(
parse(&["--update-bundle", "foo"]).update_bundle.as_deref(),
Some("foo")
);
}
#[test]
fn parse_uninstall_flag_takes_name() {
assert_eq!(
parse(&["--uninstall", "foo"]).uninstall.as_deref(),
Some("foo")
);
assert!(!parse(&["--uninstall", "foo"]).yes);
}
#[test]
fn parse_yes_flag_requires_uninstall_or_update_bundle() {
assert!(parse(&["--uninstall", "foo", "--yes"]).yes);
assert!(parse(&["--update-bundle", "foo", "--yes"]).yes);
assert!(Cli::try_parse_from(["coyote", "--yes"]).is_err());
}
#[test]
fn parse_install_flag_takes_url_or_name() {
assert_eq!(
parse(&["--install", "https://github.com/x/y"])
.install
.as_deref(),
Some("https://github.com/x/y")
);
}
#[test]
fn parse_install_builtins_flag_takes_category() {
assert_eq!(
parse(&["--install-builtins", "agents"]).install_builtins,
Some(AssetCategory::Agents)
);
assert_eq!(
parse(&["--install-builtins", "mcp_config"]).install_builtins,
Some(AssetCategory::McpConfig)
);
}
#[test]
fn parse_install_builtins_conflicts_with_install() {
assert!(
Cli::try_parse_from(["coyote", "--install-builtins", "agents", "--install", "x"])
.is_err()
);
}
#[test]
fn parse_lifecycle_flags_are_mutually_exclusive() {
assert!(Cli::try_parse_from(["coyote", "--install", "x", "--uninstall", "y"]).is_err());
assert!(
Cli::try_parse_from(["coyote", "--update-bundle", "x", "--uninstall", "y"]).is_err()
);
assert!(
Cli::try_parse_from(["coyote", "--install-builtins", "agents", "--uninstall", "y"])
.is_err()
);
assert!(Cli::try_parse_from(["coyote", "--install", "x", "--update-bundle", "y"]).is_err());
}
#[test]
fn parse_companion_flags_conflict_with_other_lifecycle_actions() {
assert!(
Cli::try_parse_from(["coyote", "--update-bundle", "x", "--filter", "agents"]).is_err()
);
assert!(Cli::try_parse_from(["coyote", "--uninstall", "x", "--install-force"]).is_err());
assert!(Cli::try_parse_from(["coyote", "--install", "x", "--yes"]).is_err());
}
#[test]
fn parse_filter_requires_install() {
assert!(Cli::try_parse_from(["coyote", "--filter", "agents"]).is_err());
assert_eq!(
parse(&["--install", "https://github.com/x/y", "--filter", "agents"]).filter,
Some(InstallFilter::Agents)
);
}
#[test]
fn parse_install_force_requires_install() {
assert!(Cli::try_parse_from(["coyote", "--install-force"]).is_err());
assert!(parse(&["--install", "https://github.com/x/y", "--install-force"]).install_force);
}
#[test]
fn parse_git_host_requires_install() {
assert!(Cli::try_parse_from(["coyote", "--git-host", "git.x.com"]).is_err());
assert!(
Cli::try_parse_from(["coyote", "--git-host", "gitlab.com", "--update-bundle", "x"])
.is_err()
);
assert_eq!(
parse(&["--install", "someuser/omc", "--git-host", "git.x.com"])
.git_host
.as_deref(),
Some("git.x.com")
);
}
#[test]
fn help_shows_install_builtins() {
use clap::CommandFactory;
let help = Cli::command().render_long_help().to_string();
assert!(help.contains("--install-builtins"));
}
#[test] #[test]
fn parse_multiple_skill_flags_preserves_order() { fn parse_multiple_skill_flags_preserves_order() {
assert_eq!( assert_eq!(
@@ -542,9 +705,9 @@ mod tests {
} }
#[test] #[test]
fn parse_prompt_flag() { fn parse_temp_role_flag() {
let cli = parse(&["--prompt", "be a pirate"]); let cli = parse(&["--temp-role", "be a pirate"]);
assert_eq!(cli.prompt, Some("be a pirate".to_string())); assert_eq!(cli.temp_role, Some("be a pirate".to_string()));
} }
#[test] #[test]
@@ -765,10 +928,7 @@ mod tests {
assert_eq!(cli.mcp_add, Some("notion".to_string())); assert_eq!(cli.mcp_add, Some("notion".to_string()));
assert!(matches!(cli.transport, Some(McpTransportArg::Http))); assert!(matches!(cli.transport, Some(McpTransportArg::Http)));
assert_eq!(cli.url, Some("https://mcp.notion.com/mcp".to_string())); assert_eq!(cli.url, Some("https://mcp.notion.com/mcp".to_string()));
assert_eq!( assert_eq!(cli.header, vec!["Authorization: Bearer {{NOTION_TOKEN}}"]);
cli.header,
vec!["Authorization: Bearer {{NOTION_TOKEN}}"]
);
assert!(cli.mcp_command.is_empty()); assert!(cli.mcp_command.is_empty());
} }
+116 -1
View File
@@ -2,6 +2,7 @@ use anyhow::{Result, anyhow};
use chrono::Utc; use chrono::Utc;
use indexmap::IndexMap; use indexmap::IndexMap;
use parking_lot::RwLock; use parking_lot::RwLock;
use std::collections::HashMap;
use std::sync::LazyLock; use std::sync::LazyLock;
type AccessTokenEntry = (String, i64, Option<String>); type AccessTokenEntry = (String, i64, Option<String>);
@@ -9,6 +10,12 @@ type AccessTokenEntry = (String, i64, Option<String>);
static ACCESS_TOKENS: LazyLock<RwLock<IndexMap<String, AccessTokenEntry>>> = static ACCESS_TOKENS: LazyLock<RwLock<IndexMap<String, AccessTokenEntry>>> =
LazyLock::new(|| RwLock::new(IndexMap::new())); LazyLock::new(|| RwLock::new(IndexMap::new()));
/// Tokens a provider rejected (401) despite being locally unexpired.
/// Maps client name → the exact rejected token so a concurrently-refreshed
/// different token is never distrusted by mistake.
static REJECTED_TOKENS: LazyLock<RwLock<HashMap<String, String>>> =
LazyLock::new(|| RwLock::new(HashMap::new()));
pub fn get_access_token(client_name: &str) -> Result<String> { pub fn get_access_token(client_name: &str) -> Result<String> {
ACCESS_TOKENS ACCESS_TOKENS
.read() .read()
@@ -30,7 +37,7 @@ pub fn is_valid_access_token(client_name: &str) -> bool {
Some(v) => v, Some(v) => v,
None => return false, None => return false,
}; };
!token.is_empty() && Utc::now().timestamp() < *expires_at !token.is_empty() && Utc::now().timestamp() < *expires_at && !is_rejected(client_name, token)
} }
pub fn set_access_token( pub fn set_access_token(
@@ -45,3 +52,111 @@ pub fn set_access_token(
entry.1 = expires_at; entry.1 = expires_at;
entry.2 = account_id; entry.2 = account_id;
} }
/// Compare-and-invalidate a provider-rejected token.
///
/// Only if the currently-cached token EQUALS `rejected` is the cache entry
/// removed and the rejection marker recorded; a concurrently-refreshed
/// different token is left untouched and no marker is set.
///
/// Returns true if a cache entry existed for this client at all (whether or
/// not it matched `rejected`) — i.e. the client is token-authed and a retry
/// after refresh is worthwhile. Returns false when there is no entry
/// (API-key clients).
pub fn distrust_access_token(client_name: &str, rejected: &str) -> bool {
let mut access_tokens = ACCESS_TOKENS.write();
let (token, _, _) = match access_tokens.get(client_name) {
Some(v) => v,
None => return false,
};
if token == rejected {
access_tokens.shift_remove(client_name);
REJECTED_TOKENS
.write()
.insert(client_name.to_string(), rejected.to_string());
}
true
}
pub fn is_rejected(client_name: &str, token: &str) -> bool {
REJECTED_TOKENS
.read()
.get(client_name)
.is_some_and(|rejected| rejected == token)
}
pub fn clear_rejected(client_name: &str) {
REJECTED_TOKENS.write().remove(client_name);
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn distrust_removes_matching_token_and_sets_marker() {
let client = "distrust-match-test";
set_access_token(client, "at-1".into(), Utc::now().timestamp() + 3600, None);
assert!(distrust_access_token(client, "at-1"));
assert!(get_access_token(client).is_err(), "cache entry not removed");
assert!(is_rejected(client, "at-1"), "marker not set");
}
#[test]
fn distrust_keeps_differing_token_and_skips_marker() {
let client = "distrust-differ-test";
set_access_token(client, "at-new".into(), Utc::now().timestamp() + 3600, None);
assert!(distrust_access_token(client, "at-old"));
assert_eq!(get_access_token(client).unwrap(), "at-new");
assert!(!is_rejected(client, "at-old"), "marker set for stale token");
}
#[test]
fn distrust_returns_false_without_cache_entry() {
let client = "distrust-missing-test";
assert!(!distrust_access_token(client, "at-1"));
assert!(!is_rejected(client, "at-1"));
}
#[test]
fn is_valid_access_token_false_for_rejected_token() {
let client = "rejected-valid-test";
set_access_token(client, "at-1".into(), Utc::now().timestamp() + 3600, None);
assert!(is_valid_access_token(client));
distrust_access_token(client, "at-1");
// A concurrent in-flight prepare re-caches the rejected file token
// between mark and refresh; it must still be treated as invalid.
set_access_token(client, "at-1".into(), Utc::now().timestamp() + 3600, None);
assert!(!is_valid_access_token(client));
}
#[test]
fn clear_rejected_clears_marker_and_clients_are_isolated() {
let client_a = "rejected-isolation-a";
let client_b = "rejected-isolation-b";
set_access_token(client_a, "at-1".into(), Utc::now().timestamp() + 3600, None);
distrust_access_token(client_a, "at-1");
assert!(is_rejected(client_a, "at-1"));
assert!(
!is_rejected(client_b, "at-1"),
"marker leaked across clients"
);
clear_rejected(client_b);
assert!(
is_rejected(client_a, "at-1"),
"wrong client's marker cleared"
);
clear_rejected(client_a);
assert!(!is_rejected(client_a, "at-1"));
}
}
+289 -15
View File
@@ -1,5 +1,6 @@
use super::*; use super::*;
use super::access_token::{distrust_access_token, get_access_token};
use crate::config::{RenderMode, paths}; use crate::config::{RenderMode, paths};
use crate::{ use crate::{
config::{AppConfig, Input, RequestContext}, config::{AppConfig, Input, RequestContext},
@@ -56,12 +57,16 @@ pub trait Client: Sync + Send {
let mut builder = ReqwestClient::builder(); let mut builder = ReqwestClient::builder();
let extra = self.extra_config(); let extra = self.extra_config();
let timeout = extra.and_then(|v| v.connect_timeout).unwrap_or(10); let timeout = extra.and_then(|v| v.connect_timeout).unwrap_or(10);
let read_timeout = extra.and_then(|v| v.read_timeout).unwrap_or(300);
if let Some(proxy) = extra.and_then(|v| v.proxy.as_deref()) { if let Some(proxy) = extra.and_then(|v| v.proxy.as_deref()) {
builder = set_proxy(builder, proxy)?; builder = set_proxy(builder, proxy)?;
} }
if let Some(user_agent) = self.app_config().user_agent.as_ref() { if let Some(user_agent) = self.app_config().user_agent.as_ref() {
builder = builder.user_agent(user_agent); builder = builder.user_agent(user_agent);
} }
if read_timeout > 0 {
builder = builder.read_timeout(Duration::from_secs(read_timeout));
}
let client = builder let client = builder
.connect_timeout(Duration::from_secs(timeout)) .connect_timeout(Duration::from_secs(timeout))
.build() .build()
@@ -69,6 +74,11 @@ pub trait Client: Sync + Send {
Ok(client) Ok(client)
} }
/// On a 401 the cached access token is distrusted and the call retried
/// exactly once; the retry re-runs the per-client prepare step, which
/// sees the rejection marker, force-refreshes the token, and rebuilds
/// the whole request. A second 401 propagates the original error; any
/// other retry failure propagates as-is.
async fn chat_completions(&self, input: Input) -> Result<ChatCompletionsOutput> { async fn chat_completions(&self, input: Input) -> Result<ChatCompletionsOutput> {
if self.app_config().dry_run { if self.app_config().dry_run {
let content = input.echo_messages(); let content = input.echo_messages();
@@ -76,11 +86,30 @@ pub trait Client: Sync + Send {
} }
let client = self.build_client()?; let client = self.build_client()?;
let data = input.prepare_completion_data(self.model(), false)?; let data = input.prepare_completion_data(self.model(), false)?;
self.chat_completions_inner(&client, data) let err = match self.chat_completions_inner(&client, data).await {
.await Ok(output) => return Ok(output),
.with_context(|| "Failed to call chat-completions api") Err(err) => err,
};
let ret = if should_retry_auth(&err, self.name()) {
debug!(
"provider '{}' rejected access token (401); refreshing and retrying once",
self.name()
);
let data = input.prepare_completion_data(self.model(), false)?;
match self.chat_completions_inner(&client, data).await {
Err(retry_err) if is_auth_error(&retry_err) => Err(err),
ret => ret,
}
} else {
Err(err)
};
ret.with_context(|| "Failed to call chat-completions api")
} }
/// Same retry-once-on-401 semantics as [`Self::chat_completions`], but
/// only while the handler has received nothing yet: retrying after
/// partial output has streamed would render it to the user twice. The
/// retry lives inside the same `select!` arm so abort stays responsive.
async fn chat_completions_streaming( async fn chat_completions_streaming(
&self, &self,
input: &Input, input: &Input,
@@ -97,7 +126,22 @@ pub trait Client: Sync + Send {
} }
let client = self.build_client()?; let client = self.build_client()?;
let data = input.prepare_completion_data(self.model(), true)?; let data = input.prepare_completion_data(self.model(), true)?;
self.chat_completions_streaming_inner(&client, handler, data).await let err = match self.chat_completions_streaming_inner(&client, handler, data).await {
Ok(()) => return Ok(()),
Err(err) => err,
};
if handler.has_received_content() || !should_retry_auth(&err, self.name()) {
return Err(err);
}
debug!(
"provider '{}' rejected access token (401); refreshing and retrying once",
self.name()
);
let data = input.prepare_completion_data(self.model(), true)?;
match self.chat_completions_streaming_inner(&client, handler, data).await {
Err(retry_err) if is_auth_error(&retry_err) => Err(err),
ret => ret,
}
} => { } => {
handler.done(); handler.done();
ret.with_context(|| "Failed to call chat-completions api") ret.with_context(|| "Failed to call chat-completions api")
@@ -109,11 +153,27 @@ pub trait Client: Sync + Send {
} }
} }
/// Same retry-once-on-401 semantics as [`Self::chat_completions`]
/// (gemini OAuth embeddings route here).
async fn embeddings(&self, data: &EmbeddingsData) -> Result<Vec<Vec<f32>>> { async fn embeddings(&self, data: &EmbeddingsData) -> Result<Vec<Vec<f32>>> {
let client = self.build_client()?; let client = self.build_client()?;
self.embeddings_inner(&client, data) let err = match self.embeddings_inner(&client, data).await {
.await Ok(output) => return Ok(output),
.context("Failed to call embeddings api") Err(err) => err,
};
let ret = if should_retry_auth(&err, self.name()) {
debug!(
"provider '{}' rejected access token (401); refreshing and retrying once",
self.name()
);
match self.embeddings_inner(&client, data).await {
Err(retry_err) if is_auth_error(&retry_err) => Err(err),
ret => ret,
}
} else {
Err(err)
};
ret.context("Failed to call embeddings api")
} }
async fn rerank(&self, data: &RerankData) -> Result<RerankOutput> { async fn rerank(&self, data: &RerankData) -> Result<RerankOutput> {
@@ -205,6 +265,7 @@ impl Default for ClientConfig {
pub struct ExtraConfig { pub struct ExtraConfig {
pub proxy: Option<String>, pub proxy: Option<String>,
pub connect_timeout: Option<u64>, pub connect_timeout: Option<u64>,
pub read_timeout: Option<u64>,
} }
#[derive(Debug, Clone, Deserialize, Default)] #[derive(Debug, Clone, Deserialize, Default)]
@@ -557,46 +618,90 @@ pub async fn noop_rerank(_builder: RequestBuilder, _model: &Model) -> Result<Rer
bail!("The client doesn't support rerank api") bail!("The client doesn't support rerank api")
} }
#[derive(Debug)]
pub struct ApiStatusError {
pub status: u16,
pub message: String,
}
impl std::fmt::Display for ApiStatusError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.message)
}
}
impl std::error::Error for ApiStatusError {}
/// True when the error chain bottoms out in an [`ApiStatusError`] with
/// status 401 EXACTLY. 403 (entitlement) and 429 (rate limit) are never
/// auth failures, and message text is never inspected.
fn is_auth_error(err: &anyhow::Error) -> bool {
err.downcast_ref::<ApiStatusError>()
.is_some_and(|api_err| api_err.status == 401)
}
/// Decides whether a 401 from `client_name` warrants a single retry after a
/// forced token refresh: the error must be a 401 [`ApiStatusError`], and the
/// client must have a cached access token to distrust (API-key clients have
/// none and never retry). Distrusting marks the exact rejected token so the
/// retry's prepare step force-refreshes it. There is deliberately no backoff:
/// the blast radius is bounded at one extra request per user-visible call.
///
/// Note: vertexai shares the ACCESS_TOKENS cache, so a 401 there also
/// triggers distrust+retry — deliberate.
fn should_retry_auth(err: &anyhow::Error, client_name: &str) -> bool {
if !is_auth_error(err) {
return false;
}
let Ok(token) = get_access_token(client_name) else {
return false;
};
distrust_access_token(client_name, &token)
}
pub fn catch_error(data: &Value, status: u16) -> Result<()> { pub fn catch_error(data: &Value, status: u16) -> Result<()> {
if (200..300).contains(&status) { if (200..300).contains(&status) {
return Ok(()); return Ok(());
} }
debug!("Invalid response, status: {status}, data: {data}"); debug!("Invalid response, status: {status}, data: {data}");
let api_error = |message: String| anyhow::Error::new(ApiStatusError { status, message });
if let Some(error) = data["error"].as_object() { if let Some(error) = data["error"].as_object() {
if let (Some(typ), Some(message)) = ( if let (Some(typ), Some(message)) = (
json_str_from_map(error, "type"), json_str_from_map(error, "type"),
json_str_from_map(error, "message"), json_str_from_map(error, "message"),
) { ) {
bail!("{message} (type: {typ})"); return Err(api_error(format!("{message} (type: {typ})")));
} else if let (Some(typ), Some(message)) = ( } else if let (Some(typ), Some(message)) = (
json_str_from_map(error, "code"), json_str_from_map(error, "code"),
json_str_from_map(error, "message"), json_str_from_map(error, "message"),
) { ) {
bail!("{message} (code: {typ})"); return Err(api_error(format!("{message} (code: {typ})")));
} }
} else if let Some(error) = data["errors"][0].as_object() { } else if let Some(error) = data["errors"][0].as_object() {
if let (Some(code), Some(message)) = ( if let (Some(code), Some(message)) = (
error.get("code").and_then(|v| v.as_u64()), error.get("code").and_then(|v| v.as_u64()),
json_str_from_map(error, "message"), json_str_from_map(error, "message"),
) { ) {
bail!("{message} (status: {code})") return Err(api_error(format!("{message} (status: {code})")));
} }
} else if let Some(error) = data[0]["error"].as_object() { } else if let Some(error) = data[0]["error"].as_object() {
if let (Some(status), Some(message)) = ( if let (Some(status), Some(message)) = (
json_str_from_map(error, "status"), json_str_from_map(error, "status"),
json_str_from_map(error, "message"), json_str_from_map(error, "message"),
) { ) {
bail!("{message} (status: {status})") return Err(api_error(format!("{message} (status: {status})")));
} }
} else if let (Some(detail), Some(status)) = (data["detail"].as_str(), data["status"].as_i64()) } else if let (Some(detail), Some(status)) = (data["detail"].as_str(), data["status"].as_i64())
{ {
bail!("{detail} (status: {status})"); return Err(api_error(format!("{detail} (status: {status})")));
} else if let Some(error) = data["error"].as_str() { } else if let Some(error) = data["error"].as_str() {
bail!("{error}"); return Err(api_error(error.to_string()));
} else if let Some(message) = data["message"].as_str() { } else if let Some(message) = data["message"].as_str() {
bail!("{message}"); return Err(api_error(message.to_string()));
} }
bail!("Invalid response data: {data} (status: {status})"); Err(api_error(format!(
"Invalid response data: {data} (status: {status})"
)))
} }
pub fn json_str_from_map<'a>( pub fn json_str_from_map<'a>(
@@ -737,3 +842,172 @@ fn prompt_input_string(desc: &str, required: bool, help_message: Option<&str>) -
let text = text.prompt()?; let text = text.prompt()?;
Ok(text) Ok(text)
} }
#[cfg(test)]
mod tests {
use super::*;
use super::super::access_token::{is_rejected, set_access_token};
fn catch_error_message(data: &Value, status: u16) -> String {
catch_error(data, status).unwrap_err().to_string()
}
#[test]
fn test_catch_error_display_json_with_type() {
let data = json!({"error": {"type": "invalid_request_error", "message": "Bad request"}});
assert_eq!(
catch_error_message(&data, 400),
"Bad request (type: invalid_request_error)"
);
}
#[test]
fn test_catch_error_display_json_with_code() {
let data = json!({"error": {"code": "rate_limited", "message": "Too many requests"}});
assert_eq!(
catch_error_message(&data, 429),
"Too many requests (code: rate_limited)"
);
}
#[test]
fn test_catch_error_display_errors_array() {
let data = json!({"errors": [{"code": 7000, "message": "No route"}]});
assert_eq!(catch_error_message(&data, 404), "No route (status: 7000)");
}
#[test]
fn test_catch_error_display_array_error_status() {
let data = json!([{"error": {"status": "PERMISSION_DENIED", "message": "Denied"}}]);
assert_eq!(
catch_error_message(&data, 403),
"Denied (status: PERMISSION_DENIED)"
);
}
#[test]
fn test_catch_error_display_detail_status() {
let data = json!({"detail": "Not found", "status": 404});
assert_eq!(catch_error_message(&data, 404), "Not found (status: 404)");
}
#[test]
fn test_catch_error_display_error_string() {
let data = json!({"error": "Something went wrong"});
assert_eq!(catch_error_message(&data, 500), "Something went wrong");
}
#[test]
fn test_catch_error_display_message_string() {
let data = json!({"message": "Unauthorized"});
assert_eq!(catch_error_message(&data, 401), "Unauthorized");
}
#[test]
fn test_catch_error_display_fallback() {
let data = json!({"unexpected": true});
assert_eq!(
catch_error_message(&data, 500),
format!("Invalid response data: {data} (status: 500)")
);
}
#[test]
fn test_catch_error_ok_on_success_status() {
let data = json!({"error": {"type": "x", "message": "y"}});
assert!(catch_error(&data, 200).is_ok());
assert!(catch_error(&data, 299).is_ok());
}
#[test]
fn test_catch_error_downcast_through_context_chain() {
let data = json!({"error": {"type": "authentication_error", "message": "Invalid key"}});
let err = catch_error(&data, 401)
.context("Failed to call chat-completions api")
.unwrap_err();
let api_err = err
.downcast_ref::<ApiStatusError>()
.expect("should downcast through context chain");
assert_eq!(api_err.status, 401);
assert_eq!(api_err.message, "Invalid key (type: authentication_error)");
}
#[test]
fn test_catch_error_preserves_status() {
let data = json!({"message": "Unauthorized"});
let err = catch_error(&data, 401).unwrap_err();
assert_eq!(err.downcast_ref::<ApiStatusError>().unwrap().status, 401);
let data = json!({"detail": "Rate limited", "status": 429});
let err = catch_error(&data, 429).unwrap_err();
assert_eq!(err.downcast_ref::<ApiStatusError>().unwrap().status, 429);
// The struct carries the outer HTTP status even when the body embeds another code
let data = json!({"errors": [{"code": 7000, "message": "No route"}]});
let err = catch_error(&data, 429).unwrap_err();
assert_eq!(err.downcast_ref::<ApiStatusError>().unwrap().status, 429);
}
/// Wrapped in `.context(...)` so every test below proves the downcast
/// works through an anyhow context chain, as in the trait methods.
fn api_status_error(status: u16) -> anyhow::Error {
anyhow::Error::new(ApiStatusError {
status,
message: format!("error (status: {status})"),
})
.context("Failed to call chat-completions api")
}
fn cache_token(client: &str, token: &str) {
set_access_token(
client,
token.into(),
chrono::Utc::now().timestamp() + 3600,
None,
);
}
#[test]
fn test_should_retry_auth_401_with_cached_token() {
let client = "should-retry-auth-401";
cache_token(client, "at-1");
assert!(should_retry_auth(&api_status_error(401), client));
assert!(is_rejected(client, "at-1"), "rejected marker not set");
}
#[test]
fn test_should_retry_auth_non_401_statuses() {
let client = "should-retry-auth-non-401";
cache_token(client, "at-1");
for status in [403, 429, 500] {
assert!(
!should_retry_auth(&api_status_error(status), client),
"retried on {status}"
);
}
assert_eq!(get_access_token(client).unwrap(), "at-1");
assert!(!is_rejected(client, "at-1"), "marker set without a 401");
}
#[test]
fn test_should_retry_auth_non_api_status_error() {
let client = "should-retry-auth-non-api";
cache_token(client, "at-1");
let err = anyhow::anyhow!("connection reset").context("Failed to call embeddings api");
assert!(!should_retry_auth(&err, client));
assert_eq!(get_access_token(client).unwrap(), "at-1");
assert!(!is_rejected(client, "at-1"));
}
#[test]
fn test_should_retry_auth_401_without_cached_token() {
let client = "should-retry-auth-no-token";
assert!(!should_retry_auth(&api_status_error(401), client));
assert!(!is_rejected(client, "at-1"));
}
}
+404 -37
View File
@@ -1,14 +1,14 @@
use super::access_token::{is_valid_access_token, set_access_token}; use super::access_token::{clear_rejected, is_rejected, is_valid_access_token, set_access_token};
use super::openai_compatible_oauth::OpenAICompatibleOAuthProvider; use super::openai_compatible_oauth::OpenAICompatibleOAuthProvider;
use super::{ClientConfig, ProviderModels}; use super::{ClientConfig, ProviderModels};
use crate::config::paths; use crate::config::paths;
use anyhow::{Context, Result, anyhow, bail}; use anyhow::{Context, Error, Result, anyhow, bail};
use base64::Engine; use base64::Engine;
use base64::engine::general_purpose::URL_SAFE_NO_PAD; use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use chrono::Utc; use chrono::Utc;
use indexmap::IndexMap; use indexmap::IndexMap;
use inquire::Text; use inquire::Text;
use reqwest::{Client as ReqwestClient, RequestBuilder}; use reqwest::{Client as ReqwestClient, RequestBuilder, StatusCode};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use serde_json::Value; use serde_json::Value;
use sha2::{Digest, Sha256}; use sha2::{Digest, Sha256};
@@ -16,6 +16,10 @@ use std::collections::HashMap;
use std::fs; use std::fs;
use std::io::{BufRead, BufReader, Write}; use std::io::{BufRead, BufReader, Write};
use std::net::TcpListener; use std::net::TcpListener;
use std::path::PathBuf;
use std::sync::{Arc, OnceLock};
use std::time::Duration;
use tokio::sync;
use url::Url; use url::Url;
use uuid::Uuid; use uuid::Uuid;
@@ -197,10 +201,20 @@ pub struct OAuthTokens {
pub account_id: Option<String>, pub account_id: Option<String>,
} }
const TOKEN_ENDPOINT_TIMEOUT: Duration = Duration::from_secs(30);
pub async fn run_oauth_flow(provider: &dyn OAuthProvider, client_name: &str) -> Result<()> { pub async fn run_oauth_flow(provider: &dyn OAuthProvider, client_name: &str) -> Result<()> {
match provider.flow() { match provider.flow() {
OAuthFlow::Pkce => run_pkce_flow(provider, client_name).await, OAuthFlow::Pkce => run_pkce_flow(provider, client_name).await,
OAuthFlow::ClientCredentials => run_client_credentials_flow(provider, client_name).await, OAuthFlow::ClientCredentials => {
run_client_credentials_flow(provider, client_name).await?;
println!(
"Successfully authenticated client '{}' with {} via OAuth (client_credentials). Tokens saved.",
client_name,
provider.provider_name()
);
Ok(())
}
OAuthFlow::DeviceCode => run_device_code_flow(provider, client_name).await, OAuthFlow::DeviceCode => run_device_code_flow(provider, client_name).await,
} }
} }
@@ -301,12 +315,20 @@ async fn run_pkce_flow(provider: &dyn OAuthProvider, client_name: &str) -> Resul
let access_token = response["access_token"] let access_token = response["access_token"]
.as_str() .as_str()
.ok_or_else(|| anyhow!("Missing access_token in response: {response}"))? .ok_or_else(|| {
anyhow!(
"Missing access_token in response (keys: {})",
token_response_keys(&response)
)
})?
.to_string(); .to_string();
let refresh_token = response["refresh_token"].as_str().map(|s| s.to_string()); let refresh_token = response["refresh_token"].as_str().map(|s| s.to_string());
let expires_in = response["expires_in"] let expires_in = response["expires_in"].as_i64().ok_or_else(|| {
.as_i64() anyhow!(
.ok_or_else(|| anyhow!("Missing expires_in in response: {response}"))?; "Missing expires_in in response (keys: {})",
token_response_keys(&response)
)
})?;
let expires_at = Utc::now().timestamp() + expires_in; let expires_at = Utc::now().timestamp() + expires_in;
@@ -334,7 +356,9 @@ async fn run_client_credentials_flow(
provider: &dyn OAuthProvider, provider: &dyn OAuthProvider,
client_name: &str, client_name: &str,
) -> Result<()> { ) -> Result<()> {
let client = ReqwestClient::new(); let client = ReqwestClient::builder()
.timeout(TOKEN_ENDPOINT_TIMEOUT)
.build()?;
let scopes = provider.scopes(); let scopes = provider.scopes();
let mut params: Vec<(&str, &str)> = vec![ let mut params: Vec<(&str, &str)> = vec![
("grant_type", "client_credentials"), ("grant_type", "client_credentials"),
@@ -349,11 +373,19 @@ async fn run_client_credentials_flow(
let access_token = response["access_token"] let access_token = response["access_token"]
.as_str() .as_str()
.ok_or_else(|| anyhow!("Missing access_token in client_credentials response: {response}"))? .ok_or_else(|| {
anyhow!(
"Missing access_token in client_credentials response (keys: {})",
token_response_keys(&response)
)
})?
.to_string(); .to_string();
let expires_in = response["expires_in"] let expires_in = response["expires_in"].as_i64().ok_or_else(|| {
.as_i64() anyhow!(
.ok_or_else(|| anyhow!("Missing expires_in in client_credentials response: {response}"))?; "Missing expires_in in client_credentials response (keys: {})",
token_response_keys(&response)
)
})?;
let expires_at = Utc::now().timestamp() + expires_in; let expires_at = Utc::now().timestamp() + expires_in;
let tokens = OAuthTokens { let tokens = OAuthTokens {
@@ -363,11 +395,6 @@ async fn run_client_credentials_flow(
account_id: provider.extract_account_id(&response), account_id: provider.extract_account_id(&response),
}; };
save_oauth_tokens(client_name, &tokens)?; save_oauth_tokens(client_name, &tokens)?;
println!(
"Successfully authenticated client '{}' with {} via OAuth (client_credentials). Tokens saved.",
client_name,
provider.provider_name()
);
Ok(()) Ok(())
} }
@@ -417,19 +444,28 @@ async fn run_device_code_flow(provider: &dyn OAuthProvider, client_name: &str) -
let device_code = device_response["device_code"] let device_code = device_response["device_code"]
.as_str() .as_str()
.ok_or_else(|| { .ok_or_else(|| {
anyhow!("Missing device_code in device authorization response: {device_response}") anyhow!(
"Missing device_code in device authorization response (keys: {})",
token_response_keys(&device_response)
)
})? })?
.to_string(); .to_string();
let user_code = device_response["user_code"] let user_code = device_response["user_code"]
.as_str() .as_str()
.ok_or_else(|| { .ok_or_else(|| {
anyhow!("Missing user_code in device authorization response: {device_response}") anyhow!(
"Missing user_code in device authorization response (keys: {})",
token_response_keys(&device_response)
)
})? })?
.to_string(); .to_string();
let verification_uri = device_response["verification_uri"] let verification_uri = device_response["verification_uri"]
.as_str() .as_str()
.ok_or_else(|| { .ok_or_else(|| {
anyhow!("Missing verification_uri in device authorization response: {device_response}") anyhow!(
"Missing verification_uri in device authorization response (keys: {})",
token_response_keys(&device_response)
)
})? })?
.to_string(); .to_string();
let verification_uri_complete = device_response["verification_uri_complete"] let verification_uri_complete = device_response["verification_uri_complete"]
@@ -551,10 +587,76 @@ fn save_oauth_tokens(client_name: &str, tokens: &OAuthTokens) -> Result<()> {
fs::create_dir_all(parent)?; fs::create_dir_all(parent)?;
} }
let json = serde_json::to_string_pretty(tokens)?; let json = serde_json::to_string_pretty(tokens)?;
fs::write(path, json)?; // Write-then-rename so a crash mid-write never truncates the live token file.
let mut tmp = path.clone().into_os_string();
tmp.push(".tmp");
let tmp = PathBuf::from(tmp);
// Tokens are live credentials: create the file owner-only, not umask-default.
let mut options = fs::OpenOptions::new();
options.write(true).create(true).truncate(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
options.mode(0o600);
}
options.open(&tmp)?.write_all(json.as_bytes())?;
fs::rename(&tmp, &path)?;
Ok(()) Ok(())
} }
pub(crate) fn token_response_keys(response: &Value) -> String {
match response.as_object() {
Some(map) => {
let keys: Vec<&str> = map.keys().map(String::as_str).collect();
format!("[{}]", keys.join(", "))
}
None => "<non-object response>".to_string(),
}
}
fn parse_refresh_response(
status: StatusCode,
response: &Value,
previous_refresh_token: Option<&str>,
) -> Result<(String, Option<String>, i64)> {
if let Some(error) = response["error"].as_str() {
let description = response["error_description"]
.as_str()
.unwrap_or("no description");
if matches!(error, "invalid_grant" | "invalid_token") {
bail!(
"OAuth refresh token was rejected ({error}: {description}). Please re-authenticate."
);
}
bail!("Token refresh failed ({error}: {description})");
}
if !status.is_success() {
bail!("Token refresh failed with HTTP status {status}");
}
let access_token = response["access_token"]
.as_str()
.ok_or_else(|| {
anyhow!(
"Missing access_token in refresh response (keys: {})",
token_response_keys(response)
)
})?
.to_string();
let refresh_token = response["refresh_token"]
.as_str()
.or(previous_refresh_token)
.map(str::to_string);
let expires_in = response["expires_in"].as_i64().ok_or_else(|| {
anyhow!(
"Missing expires_in in refresh response (keys: {})",
token_response_keys(response)
)
})?;
Ok((access_token, refresh_token, expires_in))
}
pub async fn refresh_oauth_token( pub async fn refresh_oauth_token(
client: &ReqwestClient, client: &ReqwestClient,
provider: &dyn OAuthProvider, provider: &dyn OAuthProvider,
@@ -577,19 +679,23 @@ pub async fn refresh_oauth_token(
], ],
); );
let response: Value = request.send().await?.json().await?; let (status, response) = tokio::time::timeout(TOKEN_ENDPOINT_TIMEOUT, async {
let response = request.send().await?;
let status = response.status();
let body: Value = response.json().await?;
Ok::<_, Error>((status, body))
})
.await
.map_err(|_| {
anyhow!(
"Token refresh for '{}' timed out after {}s",
client_name,
TOKEN_ENDPOINT_TIMEOUT.as_secs()
)
})??;
let access_token = response["access_token"] let (access_token, refresh_token, expires_in) =
.as_str() parse_refresh_response(status, &response, tokens.refresh_token.as_deref())?;
.ok_or_else(|| anyhow!("Missing access_token in refresh response: {response}"))?
.to_string();
let refresh_token = response["refresh_token"]
.as_str()
.map(|s| s.to_string())
.or_else(|| tokens.refresh_token.clone());
let expires_in = response["expires_in"]
.as_i64()
.ok_or_else(|| anyhow!("Missing expires_in in refresh response: {response}"))?;
let expires_at = Utc::now().timestamp() + expires_in; let expires_at = Utc::now().timestamp() + expires_in;
@@ -609,6 +715,20 @@ pub async fn refresh_oauth_token(
Ok(new_tokens) Ok(new_tokens)
} }
/// Per-client lock so concurrent requests perform a single refresh.
/// Returns a clone of the Arc so the parking_lot guard is dropped before the
/// caller awaits on the tokio mutex.
fn refresh_guard(client_name: &str) -> Arc<sync::Mutex<()>> {
static GUARDS: OnceLock<parking_lot::Mutex<HashMap<String, Arc<sync::Mutex<()>>>>> =
OnceLock::new();
GUARDS
.get_or_init(Default::default)
.lock()
.entry(client_name.to_string())
.or_default()
.clone()
}
pub async fn prepare_oauth_access_token( pub async fn prepare_oauth_access_token(
client: &ReqwestClient, client: &ReqwestClient,
provider: &dyn OAuthProvider, provider: &dyn OAuthProvider,
@@ -623,19 +743,43 @@ pub async fn prepare_oauth_access_token(
None => return Ok(false), None => return Ok(false),
}; };
let tokens = if Utc::now().timestamp() >= tokens.expires_at { let tokens = if Utc::now().timestamp() >= tokens.expires_at
|| is_rejected(client_name, &tokens.access_token)
{
let guard = refresh_guard(client_name);
let _guard = guard.lock().await;
// A concurrent caller may have refreshed while we waited for the
// lock; a valid in-memory token means the winner already populated
// the cache.
if is_valid_access_token(client_name) {
return Ok(true);
}
let tokens = match load_oauth_tokens(client_name) {
Some(t) => t,
None => return Ok(false),
};
if Utc::now().timestamp() >= tokens.expires_at
|| is_rejected(client_name, &tokens.access_token)
{
match provider.flow() { match provider.flow() {
OAuthFlow::Pkce | OAuthFlow::DeviceCode => { OAuthFlow::Pkce | OAuthFlow::DeviceCode => {
refresh_oauth_token(client, provider, client_name, &tokens).await? refresh_oauth_token(client, provider, client_name, &tokens).await?
} }
OAuthFlow::ClientCredentials => { OAuthFlow::ClientCredentials => {
run_client_credentials_flow(provider, client_name).await?; run_client_credentials_flow(provider, client_name).await?;
load_oauth_tokens(client_name) load_oauth_tokens(client_name).ok_or_else(|| {
.ok_or_else(|| anyhow!("Token file missing after client_credentials refresh"))? anyhow!("Token file missing after client_credentials refresh")
})?
} }
} }
} else { } else {
tokens tokens
}
} else {
tokens
}; };
set_access_token( set_access_token(
@@ -644,6 +788,9 @@ pub async fn prepare_oauth_access_token(
tokens.expires_at, tokens.expires_at,
tokens.account_id, tokens.account_id,
); );
// Clear even when the refresh returned the same token (some IdPs reuse
// JWTs within validity); otherwise every request re-hits the token endpoint.
clear_rejected(client_name);
Ok(true) Ok(true)
} }
@@ -886,11 +1033,55 @@ pub(crate) fn client_config_info(
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::ffi::OsString;
use std::path::PathBuf;
use std::str; use std::str;
use std::time::UNIX_EPOCH;
use super::*; use super::*;
use crate::client::access_token::{distrust_access_token, get_access_token};
use crate::client::openai_compatible::OpenAICompatibleConfig; use crate::client::openai_compatible::OpenAICompatibleConfig;
use crate::client::{ModelData, ProviderModels}; use crate::client::{ModelData, ProviderModels};
use crate::utils::get_env_name;
use serial_test::serial;
use std::{env, time::SystemTime};
fn with_temp_cache<F: FnOnce()>(f: F) {
struct Restore {
key: String,
prev: Option<OsString>,
root: PathBuf,
}
impl Drop for Restore {
fn drop(&mut self) {
unsafe {
match self.prev.take() {
Some(v) => env::set_var(&self.key, v),
None => env::remove_var(&self.key),
}
}
let _ = fs::remove_dir_all(&self.root);
}
}
let unique = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos();
let root = env::temp_dir().join(format!("coyote-client-oauth-test-{unique}"));
fs::create_dir_all(&root).unwrap();
let env_key = get_env_name("cache_dir");
let prev = env::var_os(&env_key);
unsafe {
env::set_var(&env_key, &root);
}
let _restore = Restore {
key: env_key,
prev,
root,
};
f();
}
fn base_config() -> OAuthConfig { fn base_config() -> OAuthConfig {
OAuthConfig { OAuthConfig {
@@ -1468,4 +1659,180 @@ scopes:
"body missing grant_type param: {body}" "body missing grant_type param: {body}"
); );
} }
#[test]
#[serial]
fn save_oauth_tokens_roundtrips_and_leaves_no_tmp_file() {
with_temp_cache(|| {
let tokens = OAuthTokens {
access_token: "at-123".into(),
refresh_token: Some("rt-456".into()),
expires_at: 1234567890,
account_id: Some("acct-789".into()),
};
save_oauth_tokens("atomic-test", &tokens).unwrap();
let loaded = load_oauth_tokens("atomic-test").unwrap();
assert_eq!(loaded.access_token, "at-123");
assert_eq!(loaded.refresh_token.as_deref(), Some("rt-456"));
assert_eq!(loaded.expires_at, 1234567890);
assert_eq!(loaded.account_id.as_deref(), Some("acct-789"));
let dir = paths::oauth_tokens_dir();
let leftover_tmp = fs::read_dir(&dir)
.unwrap()
.any(|e| e.unwrap().file_name().to_string_lossy().ends_with(".tmp"));
assert!(!leftover_tmp, "temp file left behind in {dir:?}");
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mode = fs::metadata(paths::token_file("atomic-test"))
.unwrap()
.permissions()
.mode();
assert_eq!(mode & 0o777, 0o600, "token file mode was {mode:o}");
}
});
}
#[test]
#[serial]
fn prepare_rejected_valid_file_token_attempts_refresh_branch() {
with_temp_cache(|| {
let client_name = "prepare-rejected-branch-test";
let expires_at = Utc::now().timestamp() + 3600;
save_oauth_tokens(
client_name,
&OAuthTokens {
access_token: "rejected-at".into(),
refresh_token: None,
expires_at,
account_id: None,
},
)
.unwrap();
set_access_token(client_name, "rejected-at".into(), expires_at, None);
assert!(distrust_access_token(client_name, "rejected-at"));
let err = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
.block_on(prepare_oauth_access_token(
&ReqwestClient::new(),
&ResourceStubProvider,
client_name,
))
.unwrap_err()
.to_string();
// The timestamp-valid but rejected file token must not be trusted;
// the refresh branch is taken and bails on the missing refresh token.
assert!(err.contains("No refresh token"), "unexpected error: {err}");
});
}
#[test]
#[serial]
fn prepare_trusts_differing_unmarked_valid_file_token() {
with_temp_cache(|| {
let client_name = "prepare-differing-token-test";
let expires_at = Utc::now().timestamp() + 3600;
set_access_token(client_name, "rejected-at".into(), expires_at, None);
assert!(distrust_access_token(client_name, "rejected-at"));
save_oauth_tokens(
client_name,
&OAuthTokens {
access_token: "fresh-at".into(),
refresh_token: None,
expires_at,
account_id: None,
},
)
.unwrap();
let ready = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
.block_on(prepare_oauth_access_token(
&ReqwestClient::new(),
&ResourceStubProvider,
client_name,
))
.unwrap();
assert!(ready);
assert_eq!(get_access_token(client_name).unwrap(), "fresh-at");
assert!(
!is_rejected(client_name, "rejected-at"),
"marker not cleared after successful prepare"
);
});
}
#[test]
fn parse_refresh_response_invalid_grant_redacts_and_prompts_reauth() {
let response = serde_json::json!({
"error": "invalid_grant",
"error_description": "refresh token revoked",
"refresh_token": "planted-secret-token",
});
let err = parse_refresh_response(StatusCode::BAD_REQUEST, &response, Some("old-rt"))
.unwrap_err()
.to_string();
assert!(err.contains("re-authenticate"), "unexpected error: {err}");
assert!(
!err.contains("planted-secret-token"),
"error leaked token material: {err}"
);
}
#[test]
fn token_response_keys_lists_keys_without_values() {
let response = serde_json::json!({
"access_token": "secret-at",
"token_type": "SecretBearer",
});
let keys = token_response_keys(&response);
assert!(keys.contains("access_token"), "missing key name: {keys}");
assert!(keys.contains("token_type"), "missing key name: {keys}");
assert!(!keys.contains("secret-at"), "leaked value: {keys}");
assert!(!keys.contains("SecretBearer"), "leaked value: {keys}");
}
#[test]
fn parse_refresh_response_rotates_refresh_token_when_present() {
let response = serde_json::json!({
"access_token": "new-at",
"refresh_token": "new-rt",
"expires_in": 3600,
});
let (access_token, refresh_token, expires_in) =
parse_refresh_response(StatusCode::OK, &response, Some("old-rt")).unwrap();
assert_eq!(access_token, "new-at");
assert_eq!(refresh_token.as_deref(), Some("new-rt"));
assert_eq!(expires_in, 3600);
}
#[test]
fn parse_refresh_response_keeps_old_refresh_token_when_absent() {
let response = serde_json::json!({
"access_token": "new-at",
"expires_in": 3600,
});
let (_, refresh_token, _) =
parse_refresh_response(StatusCode::OK, &response, Some("old-rt")).unwrap();
assert_eq!(refresh_token.as_deref(), Some("old-rt"));
}
} }
+52 -3
View File
@@ -1,4 +1,4 @@
use super::{ThinkingBlock, ToolCall, catch_error}; use super::{ApiStatusError, ThinkingBlock, ToolCall, catch_error};
use crate::utils::AbortSignal; use crate::utils::AbortSignal;
use anyhow::{Context, Result, anyhow, bail}; use anyhow::{Context, Result, anyhow, bail};
@@ -176,6 +176,14 @@ impl SseHandler {
self.thinking.push(block); self.thinking.push(block);
} }
/// Whether any output (text, tool calls, or thinking blocks) has been
/// accumulated. `Client::chat_completions_streaming` gates its 401 retry
/// on this: content already streamed to the user would be rendered a
/// second time by a retry, so partial responses are never retried.
pub fn has_received_content(&self) -> bool {
!self.buffer.is_empty() || !self.tool_calls.is_empty() || !self.thinking.is_empty()
}
pub fn abort(&self) -> AbortSignal { pub fn abort(&self) -> AbortSignal {
self.abort_signal.clone() self.abort_signal.clone()
} }
@@ -224,10 +232,14 @@ where
let data: Value = match text.parse() { let data: Value = match text.parse() {
Ok(data) => data, Ok(data) => data,
Err(_) => { Err(_) => {
bail!( return Err(ApiStatusError {
status: status.as_u16(),
message: format!(
"Invalid response data: {text} (status: {})", "Invalid response data: {text} (status: {})",
status.as_u16() status.as_u16()
); ),
}
.into());
} }
}; };
catch_error(&data, status.as_u16())?; catch_error(&data, status.as_u16())?;
@@ -418,6 +430,43 @@ mod tests {
assert!(error_message.contains("test_function_loop")); assert!(error_message.contains("test_function_loop"));
} }
fn new_handler() -> (SseHandler, tokio::sync::mpsc::UnboundedReceiver<SseEvent>) {
let (sender, receiver) = tokio::sync::mpsc::unbounded_channel();
let abort_signal = crate::utils::create_abort_signal();
(SseHandler::new(sender, abort_signal), receiver)
}
#[test]
fn test_has_received_content_text() {
let (mut handler, _rx) = new_handler();
assert!(!handler.has_received_content());
handler.text("hello").unwrap();
assert!(handler.has_received_content());
}
#[test]
fn test_has_received_content_tool_call() {
let (mut handler, _rx) = new_handler();
assert!(!handler.has_received_content());
let call = ToolCall::new("test_function".to_string(), json!({"param": 1}), None);
handler.tool_call(call).unwrap();
assert!(handler.has_received_content());
}
#[test]
fn test_has_received_content_thinking() {
let (mut handler, _rx) = new_handler();
assert!(!handler.has_received_content());
handler.thinking_block(ThinkingBlock::Thinking {
thinking: "hmm".to_string(),
signature: "sig".to_string(),
});
assert!(handler.has_received_content());
}
fn split_chunks(text: &str) -> Vec<Vec<u8>> { fn split_chunks(text: &str) -> Vec<Vec<u8>> {
let len = text.len(); let len = text.len();
let cut1 = random_range(1..len - 1); let cut1 = random_range(1..len - 1);
+88 -10
View File
@@ -15,6 +15,7 @@ use crate::config::prompts::{
}; };
use crate::graph::types::RagNode; use crate::graph::types::RagNode;
use crate::graph::{Graph, GraphParser, NodeType}; use crate::graph::{Graph, GraphParser, NodeType};
use crate::mcp::McpServerFeatures;
use crate::rag::RagInitConfig; use crate::rag::RagInitConfig;
use crate::vault::SECRET_RE; use crate::vault::SECRET_RE;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
@@ -380,7 +381,7 @@ impl Agent {
self.graph_rags.get(node_id).cloned() self.graph_rags.get(node_id).cloned()
} }
pub fn append_mcp_meta_functions(&mut self, mcp_servers: Vec<String>) { pub fn append_mcp_meta_functions(&mut self, mcp_servers: Vec<McpServerFeatures>) {
self.functions.append_mcp_meta_functions(mcp_servers); self.functions.append_mcp_meta_functions(mcp_servers);
} }
@@ -400,6 +401,10 @@ impl Agent {
self.config.enabled_skills.as_deref() self.config.enabled_skills.as_deref()
} }
pub fn enabled_macros(&self) -> Option<&[String]> {
self.config.enabled_macros.as_deref()
}
pub fn memory(&self) -> Option<bool> { pub fn memory(&self) -> Option<bool> {
self.config.memory self.config.memory
} }
@@ -744,6 +749,8 @@ pub struct AgentConfig {
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub enabled_skills: Option<Vec<String>>, pub enabled_skills: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub enabled_macros: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub continuation_prompt: Option<String>, pub continuation_prompt: Option<String>,
#[serde(default)] #[serde(default)]
pub instructions: String, pub instructions: String,
@@ -1125,6 +1132,12 @@ struct AgentMetadataStub {
description: String, description: String,
} }
#[derive(Deserialize)]
struct AgentVariablesStub {
#[serde(default)]
variables: Vec<AgentVariable>,
}
fn load_agent_description(name: &str) -> String { fn load_agent_description(name: &str) -> String {
if let Ok(config) = AgentConfig::load(&paths::agent_config_file(name)) { if let Ok(config) = AgentConfig::load(&paths::agent_config_file(name)) {
return config.description; return config.description;
@@ -1139,16 +1152,22 @@ fn load_agent_description(name: &str) -> String {
String::new() String::new()
} }
pub fn complete_agent_variables(agent_name: &str) -> Vec<(String, Option<String>)> { fn load_agent_variables(name: &str) -> Vec<AgentVariable> {
let config_path = paths::agent_config_file(agent_name); if let Ok(config) = AgentConfig::load(&paths::agent_config_file(name)) {
if !config_path.exists() { return config.variables;
return vec![];
} }
let Ok(config) = AgentConfig::load(&config_path) else {
return vec![]; if let Ok(contents) = read_to_string(paths::agent_graph_file(name))
}; && let Ok(stub) = serde_yaml::from_str::<AgentVariablesStub>(&contents)
config {
.variables return stub.variables;
}
Vec::new()
}
pub fn complete_agent_variables(agent_name: &str) -> Vec<(String, Option<String>)> {
load_agent_variables(agent_name)
.iter() .iter()
.map(|v| { .map(|v| {
let description = match &v.default { let description = match &v.default {
@@ -1225,6 +1244,30 @@ variables:
assert!(config.top_p.is_none()); assert!(config.top_p.is_none());
} }
#[test]
fn agent_config_enabled_macros_absent_is_none() {
let yaml = "name: minimal\ninstructions: hi\n";
let config: AgentConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.enabled_macros, None);
}
#[test]
fn agent_config_enabled_macros_empty_list_is_some_empty() {
let yaml = "name: minimal\ninstructions: hi\nenabled_macros: []\n";
let config: AgentConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.enabled_macros, Some(vec![]));
}
#[test]
fn agent_config_enabled_macros_list() {
let yaml = "name: minimal\ninstructions: hi\nenabled_macros:\n - a\n";
let config: AgentConfig = serde_yaml::from_str(yaml).unwrap();
assert_eq!(config.enabled_macros, Some(vec!["a".to_string()]));
}
#[test] #[test]
fn agent_config_with_model() { fn agent_config_with_model() {
let yaml = let yaml =
@@ -1369,6 +1412,41 @@ version: "1.0"
assert_eq!(meta.description, ""); assert_eq!(meta.description, "");
} }
#[test]
fn agent_variables_stub_extracts_variables_from_graph_yaml() {
let yaml = r#"
name: coder
description: Implementation agent.
version: "1.0"
variables:
- name: task
description: The task to implement
- name: scope
description: Directory scope
default: src/
start: plan
nodes: {}
"#;
let stub: AgentVariablesStub = serde_yaml::from_str(yaml).unwrap();
assert_eq!(stub.variables.len(), 2);
assert_eq!(stub.variables[0].name, "task");
assert_eq!(stub.variables[0].description, "The task to implement");
assert_eq!(stub.variables[0].default, None);
assert_eq!(stub.variables[1].name, "scope");
assert_eq!(stub.variables[1].default.as_deref(), Some("src/"));
}
#[test]
fn agent_variables_stub_defaults_when_variables_missing() {
let yaml = "name: coder\nversion: \"1.0\"\nstart: plan\nnodes: {}\n";
let stub: AgentVariablesStub = serde_yaml::from_str(yaml).unwrap();
assert!(stub.variables.is_empty());
}
#[test] #[test]
fn rag_init_config_forwards_an_explicit_driver() { fn rag_init_config_forwards_an_explicit_driver() {
let node: RagNode = let node: RagNode =
+83 -4
View File
@@ -1,6 +1,6 @@
use crate::client::{ClientConfig, Model, ModelType, list_models}; use crate::client::{ClientConfig, Model, ModelType, list_models};
use crate::render::{MarkdownRender, RenderOptions}; use crate::render::{MarkdownRender, RenderOptions};
use crate::utils::{IS_STDOUT_TERMINAL, NO_COLOR, decode_bin, get_env_name}; use crate::utils::{IS_STDOUT_TERMINAL, NO_COLOR, decode_bin, drain_stale_tty_input, get_env_name};
use super::paths; use super::paths;
use anyhow::{Context, Result, anyhow, bail}; use anyhow::{Context, Result, anyhow, bail};
@@ -43,6 +43,8 @@ pub struct AppConfig {
#[serde(default, deserialize_with = "super::deserialize_csv_or_vec")] #[serde(default, deserialize_with = "super::deserialize_csv_or_vec")]
pub enabled_skills: Option<Vec<String>>, pub enabled_skills: Option<Vec<String>>,
pub visible_skills: Option<Vec<String>>, pub visible_skills: Option<Vec<String>>,
#[serde(default, deserialize_with = "super::deserialize_csv_or_vec")]
pub enabled_macros: Option<Vec<String>>,
pub mcp_server_support: bool, pub mcp_server_support: bool,
pub mapping_mcp_servers: IndexMap<String, String>, pub mapping_mcp_servers: IndexMap<String, String>,
@@ -96,6 +98,7 @@ pub struct AppConfig {
pub user_agent: Option<String>, pub user_agent: Option<String>,
pub save_shell_history: bool, pub save_shell_history: bool,
pub no_workspace_mcp: bool, pub no_workspace_mcp: bool,
pub no_workspace_macros: bool,
pub sync_models_url: Option<String>, pub sync_models_url: Option<String>,
pub clients: Vec<ClientConfig>, pub clients: Vec<ClientConfig>,
@@ -127,6 +130,7 @@ impl Default for AppConfig {
skills_enabled: true, skills_enabled: true,
enabled_skills: None, enabled_skills: None,
visible_skills: None, visible_skills: None,
enabled_macros: None,
mcp_server_support: true, mcp_server_support: true,
mapping_mcp_servers: Default::default(), mapping_mcp_servers: Default::default(),
@@ -178,6 +182,7 @@ impl Default for AppConfig {
user_agent: None, user_agent: None,
save_shell_history: true, save_shell_history: true,
no_workspace_mcp: false, no_workspace_mcp: false,
no_workspace_macros: false,
sync_models_url: None, sync_models_url: None,
clients: vec![], clients: vec![],
@@ -211,6 +216,7 @@ impl AppConfig {
skills_enabled: config.skills_enabled, skills_enabled: config.skills_enabled,
enabled_skills: config.enabled_skills, enabled_skills: config.enabled_skills,
visible_skills: config.visible_skills, visible_skills: config.visible_skills,
enabled_macros: config.enabled_macros,
mcp_server_support: config.mcp_server_support, mcp_server_support: config.mcp_server_support,
mapping_mcp_servers: config.mapping_mcp_servers, mapping_mcp_servers: config.mapping_mcp_servers,
@@ -262,6 +268,7 @@ impl AppConfig {
user_agent: config.user_agent, user_agent: config.user_agent,
save_shell_history: config.save_shell_history, save_shell_history: config.save_shell_history,
no_workspace_mcp: false, no_workspace_mcp: false,
no_workspace_macros: false,
sync_models_url: config.sync_models_url, sync_models_url: config.sync_models_url,
clients: config.clients, clients: config.clients,
@@ -533,6 +540,10 @@ impl AppConfig {
self.enabled_skills = v.map(|raw| super::csv_to_vec(&raw)); self.enabled_skills = v.map(|raw| super::csv_to_vec(&raw));
} }
if let Some(v) = super::read_env_value::<String>(&get_env_name("enabled_macros")) {
self.enabled_macros = v.map(|raw| super::csv_to_vec(&raw));
}
if let Some(Some(v)) = super::read_env_bool(&get_env_name("mcp_server_support")) { if let Some(Some(v)) = super::read_env_bool(&get_env_name("mcp_server_support")) {
self.mcp_server_support = v; self.mcp_server_support = v;
} }
@@ -616,15 +627,19 @@ impl AppConfig {
if self.highlight && self.theme.is_none() { if self.highlight && self.theme.is_none() {
if let Some(v) = super::read_env_value::<String>(&get_env_name("theme")) { if let Some(v) = super::read_env_value::<String>(&get_env_name("theme")) {
self.theme = v; self.theme = v;
} else if *IS_STDOUT_TERMINAL } else if *IS_STDOUT_TERMINAL {
&& let Ok(color_scheme) = color_scheme(QueryOptions::default()) if let Ok(color_scheme) = color_scheme(QueryOptions::default()) {
{
let theme = match color_scheme { let theme = match color_scheme {
ColorScheme::Dark => "dark", ColorScheme::Dark => "dark",
ColorScheme::Light => "light", ColorScheme::Light => "light",
}; };
self.theme = Some(theme.into()); self.theme = Some(theme.into());
} }
// The OSC/DA1 reply can arrive after colorsaurus stops reading
// (observed under zellij-in-kitty). Drain any late reply bytes so
// they are neither echoed nor read as line-editor input.
drain_stale_tty_input();
}
} }
if let Some(v) = super::read_env_value::<String>(&get_env_name("left_prompt")) { if let Some(v) = super::read_env_value::<String>(&get_env_name("left_prompt")) {
self.left_prompt = v; self.left_prompt = v;
@@ -765,6 +780,70 @@ mod tests {
); );
} }
#[test]
#[serial_test::serial]
fn from_config_copies_enabled_macros() {
let cfg = Config {
model_id: "provider:test".to_string(),
enabled_macros: Some(vec!["a".to_string()]),
..Config::default()
};
let app = AppConfig::from_config(cfg).unwrap();
assert_eq!(app.enabled_macros, Some(vec!["a".to_string()]));
}
#[test]
#[serial_test::serial]
fn from_config_preserves_explicit_empty_enabled_macros() {
let cfg = Config {
model_id: "provider:test".to_string(),
enabled_macros: Some(vec![]),
..Config::default()
};
let app = AppConfig::from_config(cfg).unwrap();
assert_eq!(app.enabled_macros, Some(vec![]));
}
#[test]
#[serial_test::serial]
fn load_envs_overrides_enabled_macros() {
let env_name = get_env_name("enabled_macros");
let prev = std::env::var_os(&env_name);
let mut app = AppConfig::default();
unsafe { std::env::set_var(&env_name, "a,b") };
app.load_envs();
assert_eq!(
app.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
unsafe { std::env::set_var(&env_name, "") };
app.load_envs();
assert_eq!(app.enabled_macros, Some(vec![]));
unsafe { std::env::set_var(&env_name, "null") };
app.load_envs();
assert_eq!(app.enabled_macros, None);
unsafe { std::env::remove_var(&env_name) };
app.enabled_macros = Some(vec!["keep".to_string()]);
app.load_envs();
assert_eq!(app.enabled_macros, Some(vec!["keep".to_string()]));
unsafe {
match prev {
Some(v) => std::env::set_var(&env_name, v),
None => std::env::remove_var(&env_name),
}
}
}
#[test] #[test]
fn editor_returns_configured_value() { fn editor_returns_configured_value() {
let configured = cached_editor() let configured = cached_editor()
+1 -1
View File
@@ -70,7 +70,7 @@ impl AppState {
let mut functions = Functions::init(config.visible_tools.as_ref().unwrap_or(&Vec::new()))?; let mut functions = Functions::init(config.visible_tools.as_ref().unwrap_or(&Vec::new()))?;
if !mcp_registry.is_empty() && config.mcp_server_support { if !mcp_registry.is_empty() && config.mcp_server_support {
functions.append_mcp_meta_functions(mcp_registry.list_started_servers()); functions.append_mcp_meta_functions(mcp_registry.server_features());
} }
let mcp_registry = if mcp_registry.is_empty() { let mcp_registry = if mcp_registry.is_empty() {
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+827 -27
View File
File diff suppressed because it is too large Load Diff
+8 -14
View File
@@ -1,6 +1,6 @@
use crate::mcp::{ use crate::mcp::{
ConnectedServer, JsonField, McpServer, McpTransportType, is_auth_required_error, oauth, ConnectedServer, JsonField, McpAuthRequired, McpServer, McpTransportType,
spawn_mcp_server, is_auth_required_error, resolve_http_auth, spawn_mcp_server,
}; };
use anyhow::Result; use anyhow::Result;
@@ -102,19 +102,13 @@ impl McpFactory {
return Ok(existing); return Ok(existing);
} }
let bearer_token = if spec.is_remote() { let (auth, auth_reason) = resolve_http_auth(name, spec).await;
oauth::load_valid_mcp_token(name) let handle = spawn_mcp_server(spec, log_path, auth).await.map_err(|e| {
} else {
None
};
let handle = spawn_mcp_server(spec, log_path, bearer_token)
.await
.map_err(|e| {
if is_auth_required_error(&e) { if is_auth_required_error(&e) {
e.context(format!( e.context(McpAuthRequired {
"MCP server '{name}' requires OAuth authentication. \ server: name.to_string(),
Run `coyote --auth-mcp {name}` or `.mcp auth {name}` in the REPL to authenticate." reason: auth_reason,
)) })
} else { } else {
e e
} }
+70 -1
View File
@@ -1,9 +1,11 @@
mod agent; mod agent;
mod app_config; mod app_config;
mod app_state; mod app_state;
mod bundles;
mod input; mod input;
mod install_remote; mod install_remote;
pub(crate) mod instructions; pub(crate) mod instructions;
mod macro_policy;
mod macros; mod macros;
mod mcp_factory; mod mcp_factory;
pub(crate) mod memory; pub(crate) mod memory;
@@ -20,6 +22,8 @@ pub(crate) mod todo;
mod tool_scope; mod tool_scope;
mod update; mod update;
#[cfg(test)]
pub(crate) use self::agent::AgentConfig;
pub use self::agent::{ pub use self::agent::{
Agent, AgentVariable, AgentVariables, complete_agent_variables, list_agents, Agent, AgentVariable, AgentVariables, complete_agent_variables, list_agents,
list_agents_with_descriptions, list_agents_with_descriptions,
@@ -28,8 +32,15 @@ pub use self::agent::{
pub use self::app_config::AppConfig; pub use self::app_config::AppConfig;
#[allow(unused_imports)] #[allow(unused_imports)]
pub use self::app_state::AppState; pub use self::app_state::AppState;
pub use self::bundles::list_installed_bundles;
pub use self::input::Input; pub use self::input::Input;
pub use self::install_remote::{install_remote, install_remote_from_repl_args}; pub use self::install_remote::{
DEFAULT_GIT_HOST, install_or_update, install_or_update_from_repl_args, uninstall_bundle,
update_bundle,
};
pub use self::macro_policy::{
MacroAllowlistLevel, MacroPolicy, MacroSource, MacroState, RESERVED_MACRO_NAMES, ResolvedMacro,
};
#[allow(unused_imports)] #[allow(unused_imports)]
pub use self::request_context::{RenderMode, RequestContext, should_inject_skill_instructions}; pub use self::request_context::{RenderMode, RequestContext, should_inject_skill_instructions};
pub use self::role::{ pub use self::role::{
@@ -42,6 +53,11 @@ pub use self::skill::Skill;
pub use self::skill_policy::SkillPolicy; pub use self::skill_policy::SkillPolicy;
#[allow(unused_imports)] #[allow(unused_imports)]
pub use self::skill_registry::SkillRegistry; pub use self::skill_registry::SkillRegistry;
#[cfg(test)]
pub(crate) use self::tool_scope::test_fixtures;
pub use self::tool_scope::{
McpPromptCompletion, flatten_prompt_messages, resolve_prompt_args, sanitize_display_text,
};
pub use self::update::run_self_update; pub use self::update::run_self_update;
use crate::client::{ use crate::client::{
self, ClientConfig, MessageContentToolCalls, Model, ModelType, OPENAI_COMPATIBLE_PROVIDERS, self, ClientConfig, MessageContentToolCalls, Model, ModelType, OPENAI_COMPATIBLE_PROVIDERS,
@@ -151,6 +167,8 @@ const SBX_KIT_DIR_NAME: &str = "sbx-kit";
const SBX_KIT_HASH_FILE: &str = "kit.sha256"; const SBX_KIT_HASH_FILE: &str = "kit.sha256";
const SBX_MIXIN_FILE_NAME: &str = "sbx-mixin.yaml"; const SBX_MIXIN_FILE_NAME: &str = "sbx-mixin.yaml";
pub(crate) const VAULT_DATA_FILE_NAME: &str = "vault.yml"; pub(crate) const VAULT_DATA_FILE_NAME: &str = "vault.yml";
const INSTALLED_BUNDLES_FILE_NAME: &str = "installed-bundles.yaml";
const BUNDLE_MANIFEST_FILE: &str = "coyote-bundle.yaml";
const SBX_MIXIN_KITS_DIR_NAME: &str = "sbx-mixin-kits"; const SBX_MIXIN_KITS_DIR_NAME: &str = "sbx-mixin-kits";
const GIT_DIR_NAME: &str = ".git"; const GIT_DIR_NAME: &str = ".git";
const GITIGNORE_FILE_NAME: &str = ".gitignore"; const GITIGNORE_FILE_NAME: &str = ".gitignore";
@@ -221,6 +239,8 @@ pub struct Config {
#[serde(default, deserialize_with = "deserialize_csv_or_vec")] #[serde(default, deserialize_with = "deserialize_csv_or_vec")]
pub enabled_skills: Option<Vec<String>>, pub enabled_skills: Option<Vec<String>>,
pub visible_skills: Option<Vec<String>>, pub visible_skills: Option<Vec<String>>,
#[serde(default, deserialize_with = "deserialize_csv_or_vec")]
pub enabled_macros: Option<Vec<String>>,
pub mcp_server_support: bool, pub mcp_server_support: bool,
pub mapping_mcp_servers: IndexMap<String, String>, pub mapping_mcp_servers: IndexMap<String, String>,
@@ -303,6 +323,7 @@ impl Default for Config {
skills_enabled: true, skills_enabled: true,
enabled_skills: None, enabled_skills: None,
visible_skills: None, visible_skills: None,
enabled_macros: None,
mcp_server_support: true, mcp_server_support: true,
mapping_mcp_servers: Default::default(), mapping_mcp_servers: Default::default(),
@@ -1124,9 +1145,50 @@ clients:
assert!(cfg.enabled_mcp_servers.is_none()); assert!(cfg.enabled_mcp_servers.is_none());
} }
#[test]
fn config_enabled_macros_absent_is_none() {
let cfg: Config = serde_yaml::from_str("model: provider:test").unwrap();
assert_eq!(cfg.enabled_macros, None);
}
#[test]
fn config_enabled_macros_empty_string_is_some_empty() {
let cfg: Config = serde_yaml::from_str("enabled_macros: \"\"").unwrap();
assert_eq!(cfg.enabled_macros, Some(vec![]));
}
#[test]
fn config_enabled_macros_csv_string() {
let cfg: Config = serde_yaml::from_str("enabled_macros: \"a, b\"").unwrap();
assert_eq!(
cfg.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
}
#[test]
fn config_enabled_macros_list() {
let cfg: Config = serde_yaml::from_str("enabled_macros:\n - a\n - b").unwrap();
assert_eq!(
cfg.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
}
#[test]
fn config_enabled_macros_null_is_none() {
let cfg: Config = serde_yaml::from_str("enabled_macros: null").unwrap();
assert_eq!(cfg.enabled_macros, None);
}
#[test] #[test]
fn assert_state_pass_always_true() { fn assert_state_pass_always_true() {
let pass = AssertState::pass(); let pass = AssertState::pass();
assert!(pass.assert(StateFlags::empty())); assert!(pass.assert(StateFlags::empty()));
assert!(pass.assert(StateFlags::ROLE)); assert!(pass.assert(StateFlags::ROLE));
assert!(pass.assert(StateFlags::SESSION | StateFlags::AGENT)); assert!(pass.assert(StateFlags::SESSION | StateFlags::AGENT));
@@ -1136,6 +1198,7 @@ clients:
#[test] #[test]
fn assert_state_bare_only_empty() { fn assert_state_bare_only_empty() {
let bare = AssertState::bare(); let bare = AssertState::bare();
assert!(bare.assert(StateFlags::empty())); assert!(bare.assert(StateFlags::empty()));
assert!(!bare.assert(StateFlags::ROLE)); assert!(!bare.assert(StateFlags::ROLE));
assert!(!bare.assert(StateFlags::SESSION)); assert!(!bare.assert(StateFlags::SESSION));
@@ -1144,6 +1207,7 @@ clients:
#[test] #[test]
fn assert_state_true_requires_flag_present() { fn assert_state_true_requires_flag_present() {
let state = AssertState::True(StateFlags::ROLE); let state = AssertState::True(StateFlags::ROLE);
assert!(state.assert(StateFlags::ROLE)); assert!(state.assert(StateFlags::ROLE));
assert!(state.assert(StateFlags::ROLE | StateFlags::SESSION)); assert!(state.assert(StateFlags::ROLE | StateFlags::SESSION));
assert!(!state.assert(StateFlags::empty())); assert!(!state.assert(StateFlags::empty()));
@@ -1153,6 +1217,7 @@ clients:
#[test] #[test]
fn assert_state_true_with_multiple_flags_any_match() { fn assert_state_true_with_multiple_flags_any_match() {
let state = AssertState::True(StateFlags::SESSION_EMPTY | StateFlags::SESSION); let state = AssertState::True(StateFlags::SESSION_EMPTY | StateFlags::SESSION);
assert!(state.assert(StateFlags::SESSION_EMPTY)); assert!(state.assert(StateFlags::SESSION_EMPTY));
assert!(state.assert(StateFlags::SESSION)); assert!(state.assert(StateFlags::SESSION));
assert!(state.assert(StateFlags::SESSION | StateFlags::ROLE)); assert!(state.assert(StateFlags::SESSION | StateFlags::ROLE));
@@ -1163,6 +1228,7 @@ clients:
#[test] #[test]
fn assert_state_false_requires_flag_absent() { fn assert_state_false_requires_flag_absent() {
let state = AssertState::False(StateFlags::AGENT); let state = AssertState::False(StateFlags::AGENT);
assert!(state.assert(StateFlags::empty())); assert!(state.assert(StateFlags::empty()));
assert!(state.assert(StateFlags::ROLE)); assert!(state.assert(StateFlags::ROLE));
assert!(!state.assert(StateFlags::AGENT)); assert!(!state.assert(StateFlags::AGENT));
@@ -1172,6 +1238,7 @@ clients:
#[test] #[test]
fn assert_state_false_with_multiple_flags() { fn assert_state_false_with_multiple_flags() {
let state = AssertState::False(StateFlags::SESSION | StateFlags::AGENT); let state = AssertState::False(StateFlags::SESSION | StateFlags::AGENT);
assert!(state.assert(StateFlags::empty())); assert!(state.assert(StateFlags::empty()));
assert!(state.assert(StateFlags::ROLE)); assert!(state.assert(StateFlags::ROLE));
assert!(!state.assert(StateFlags::SESSION)); assert!(!state.assert(StateFlags::SESSION));
@@ -1182,6 +1249,7 @@ clients:
#[test] #[test]
fn assert_state_truefalse_requires_true_present_and_false_absent() { fn assert_state_truefalse_requires_true_present_and_false_absent() {
let state = AssertState::TrueFalse(StateFlags::ROLE, StateFlags::SESSION); let state = AssertState::TrueFalse(StateFlags::ROLE, StateFlags::SESSION);
assert!(state.assert(StateFlags::ROLE)); assert!(state.assert(StateFlags::ROLE));
assert!(state.assert(StateFlags::ROLE | StateFlags::RAG)); assert!(state.assert(StateFlags::ROLE | StateFlags::RAG));
assert!(!state.assert(StateFlags::empty())); assert!(!state.assert(StateFlags::empty()));
@@ -1192,6 +1260,7 @@ clients:
#[test] #[test]
fn assert_state_equal_exact_match() { fn assert_state_equal_exact_match() {
let state = AssertState::Equal(StateFlags::ROLE | StateFlags::SESSION); let state = AssertState::Equal(StateFlags::ROLE | StateFlags::SESSION);
assert!(state.assert(StateFlags::ROLE | StateFlags::SESSION)); assert!(state.assert(StateFlags::ROLE | StateFlags::SESSION));
assert!(!state.assert(StateFlags::ROLE)); assert!(!state.assert(StateFlags::ROLE));
assert!(!state.assert(StateFlags::SESSION)); assert!(!state.assert(StateFlags::SESSION));
+22 -9
View File
@@ -2,10 +2,10 @@ use super::role::Role;
use super::{ use super::{
AGENT_GRAPH_FILE_NAME, AGENTS_DIR_NAME, BASH_PROMPT_UTILS_FILE_NAME, CONFIG_FILE_NAME, AGENT_GRAPH_FILE_NAME, AGENTS_DIR_NAME, BASH_PROMPT_UTILS_FILE_NAME, CONFIG_FILE_NAME,
ENV_FILE_NAME, FUNCTIONS_BIN_DIR_NAME, FUNCTIONS_DIR_NAME, GLOBAL_TOOLS_DIR_NAME, ENV_FILE_NAME, FUNCTIONS_BIN_DIR_NAME, FUNCTIONS_DIR_NAME, GLOBAL_TOOLS_DIR_NAME,
GLOBAL_TOOLS_UTILS_DIR_NAME, HIDDEN_MCP_FILE_NAME, MACROS_DIR_NAME, MCP_FILE_NAME, GLOBAL_TOOLS_UTILS_DIR_NAME, HIDDEN_MCP_FILE_NAME, INSTALLED_BUNDLES_FILE_NAME,
MEMORY_DIR_NAME, MEMORY_INDEX_FILE_NAME, ModelsOverride, RAGS_DIR_NAME, ROLES_DIR_NAME, MACROS_DIR_NAME, MCP_FILE_NAME, MEMORY_DIR_NAME, MEMORY_INDEX_FILE_NAME, ModelsOverride,
SBX_KIT_DIR_NAME, SBX_KIT_HASH_FILE, SBX_MIXIN_FILE_NAME, SBX_MIXIN_KITS_DIR_NAME, RAGS_DIR_NAME, ROLES_DIR_NAME, SBX_KIT_DIR_NAME, SBX_KIT_HASH_FILE, SBX_MIXIN_FILE_NAME,
SKILLS_DIR_NAME, WORKSPACE_COYOTE_DIR_NAME, SBX_MIXIN_KITS_DIR_NAME, SKILLS_DIR_NAME, WORKSPACE_COYOTE_DIR_NAME,
}; };
use crate::client::ProviderModels; use crate::client::ProviderModels;
use crate::config::REPL_HISTORY_DIR_NAME; use crate::config::REPL_HISTORY_DIR_NAME;
@@ -148,6 +148,16 @@ pub fn sbx_kit_hash_file() -> PathBuf {
sbx_kit_dir().join(SBX_KIT_HASH_FILE) sbx_kit_dir().join(SBX_KIT_HASH_FILE)
} }
pub fn sandbox_mixin_hashes_dir() -> PathBuf {
cache_dir().join("sandbox-mixin-hashes")
}
pub fn sandbox_mixin_hash_file(sandbox_name: &str) -> PathBuf {
// Sandbox names are sanitized by the caller, but never trust a path
// component: a stray separator must not escape the hash directory.
sandbox_mixin_hashes_dir().join(format!("{}.hash", sandbox_name.replace('/', "_")))
}
pub fn sbx_mixin_kits_dir() -> PathBuf { pub fn sbx_mixin_kits_dir() -> PathBuf {
cache_dir().join(SBX_MIXIN_KITS_DIR_NAME) cache_dir().join(SBX_MIXIN_KITS_DIR_NAME)
} }
@@ -159,6 +169,10 @@ pub fn config_file() -> PathBuf {
} }
} }
pub fn installed_bundles_file() -> PathBuf {
local_dir(INSTALLED_BUNDLES_FILE_NAME)
}
pub fn roles_dir() -> PathBuf { pub fn roles_dir() -> PathBuf {
match env::var(get_env_name("roles_dir")) { match env::var(get_env_name("roles_dir")) {
Ok(value) => PathBuf::from(value), Ok(value) => PathBuf::from(value),
@@ -204,6 +218,10 @@ pub fn workspace_skill_file(name: &str) -> PathBuf {
workspace_skills_dir().join(name).join("SKILL.md") workspace_skills_dir().join(name).join("SKILL.md")
} }
pub fn workspace_macros_dir() -> PathBuf {
workspace_config_dir().join(MACROS_DIR_NAME)
}
pub fn workspace_mcp_config_file() -> Option<PathBuf> { pub fn workspace_mcp_config_file() -> Option<PathBuf> {
workspace_mcp_config_file_in(&env::current_dir().unwrap_or_default()) workspace_mcp_config_file_in(&env::current_dir().unwrap_or_default())
} }
@@ -460,11 +478,6 @@ pub fn list_macros() -> Vec<String> {
list_file_names(macros_dir(), ".yaml") list_file_names(macros_dir(), ".yaml")
} }
pub fn has_macro(name: &str) -> bool {
let names = list_macros();
names.contains(&name.to_string())
}
pub fn list_skills() -> Vec<String> { pub fn list_skills() -> Vec<String> {
let mut names = Vec::new(); let mut names = Vec::new();
let mut seen = HashSet::new(); let mut seen = HashSet::new();
File diff suppressed because it is too large Load Diff
+93
View File
@@ -75,6 +75,12 @@ pub struct Role {
deserialize_with = "super::deserialize_csv_or_vec" deserialize_with = "super::deserialize_csv_or_vec"
)] )]
enabled_skills: Option<Vec<String>>, enabled_skills: Option<Vec<String>>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
deserialize_with = "super::deserialize_csv_or_vec"
)]
enabled_macros: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
auto_continue: Option<bool>, auto_continue: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
@@ -129,6 +135,7 @@ impl Role {
} }
"skills_enabled" => role.skills_enabled = value.as_bool(), "skills_enabled" => role.skills_enabled = value.as_bool(),
"enabled_skills" => role.enabled_skills = parse_string_or_array(value), "enabled_skills" => role.enabled_skills = parse_string_or_array(value),
"enabled_macros" => role.enabled_macros = parse_string_or_array(value),
"auto_continue" => role.auto_continue = value.as_bool(), "auto_continue" => role.auto_continue = value.as_bool(),
"max_auto_continues" => { "max_auto_continues" => {
role.max_auto_continues = value.as_u64().map(|v| v as usize) role.max_auto_continues = value.as_u64().map(|v| v as usize)
@@ -196,6 +203,10 @@ impl Role {
let inline = serde_json::to_string(enabled_skills).unwrap_or_else(|_| "[]".to_string()); let inline = serde_json::to_string(enabled_skills).unwrap_or_else(|_| "[]".to_string());
metadata.push(format!("enabled_skills: {inline}")); metadata.push(format!("enabled_skills: {inline}"));
} }
if let Some(enabled_macros) = &self.enabled_macros {
let inline = serde_json::to_string(enabled_macros).unwrap_or_else(|_| "[]".to_string());
metadata.push(format!("enabled_macros: {inline}"));
}
if let Some(auto_continue) = self.auto_continue { if let Some(auto_continue) = self.auto_continue {
metadata.push(format!("auto_continue: {auto_continue}")); metadata.push(format!("auto_continue: {auto_continue}"));
} }
@@ -357,6 +368,10 @@ impl Role {
self.enabled_skills.as_deref() self.enabled_skills.as_deref()
} }
pub fn enabled_macros(&self) -> Option<&[String]> {
self.enabled_macros.as_deref()
}
pub fn append_to_prompt(&mut self, text: &str) { pub fn append_to_prompt(&mut self, text: &str) {
self.prompt.push_str(text); self.prompt.push_str(text);
} }
@@ -543,6 +558,7 @@ mod tests {
#[test] #[test]
fn role_new_parses_prompt() { fn role_new_parses_prompt() {
let role = Role::new("test", "You are a helpful assistant"); let role = Role::new("test", "You are a helpful assistant");
assert_eq!(role.name(), "test"); assert_eq!(role.name(), "test");
assert_eq!(role.prompt(), "You are a helpful assistant"); assert_eq!(role.prompt(), "You are a helpful assistant");
} }
@@ -551,7 +567,9 @@ mod tests {
fn role_new_parses_metadata() { fn role_new_parses_metadata() {
let content = let content =
"---\nmodel: openai:gpt-4\ntemperature: 0.7\ntop_p: 0.9\n---\nYou are helpful"; "---\nmodel: openai:gpt-4\ntemperature: 0.7\ntop_p: 0.9\n---\nYou are helpful";
let role = Role::new("test", content); let role = Role::new("test", content);
assert_eq!(role.model_id(), Some("openai:gpt-4")); assert_eq!(role.model_id(), Some("openai:gpt-4"));
assert_eq!(role.temperature(), Some(0.7)); assert_eq!(role.temperature(), Some(0.7));
assert_eq!(role.top_p(), Some(0.9)); assert_eq!(role.top_p(), Some(0.9));
@@ -561,7 +579,9 @@ mod tests {
#[test] #[test]
fn role_new_parses_enabled_tools() { fn role_new_parses_enabled_tools() {
let content = "---\nenabled_tools: tool1,tool2\n---\nPrompt"; let content = "---\nenabled_tools: tool1,tool2\n---\nPrompt";
let role = Role::new("test", content); let role = Role::new("test", content);
assert_eq!( assert_eq!(
role.enabled_tools(), role.enabled_tools(),
Some(vec!["tool1".to_string(), "tool2".to_string()]) Some(vec!["tool1".to_string(), "tool2".to_string()])
@@ -571,7 +591,9 @@ mod tests {
#[test] #[test]
fn role_new_parses_enabled_mcp_servers() { fn role_new_parses_enabled_mcp_servers() {
let content = "---\nenabled_mcp_servers: github,jira\n---\nPrompt"; let content = "---\nenabled_mcp_servers: github,jira\n---\nPrompt";
let role = Role::new("test", content); let role = Role::new("test", content);
assert_eq!( assert_eq!(
role.enabled_mcp_servers(), role.enabled_mcp_servers(),
Some(vec!["github".to_string(), "jira".to_string()]) Some(vec!["github".to_string(), "jira".to_string()])
@@ -581,6 +603,7 @@ mod tests {
#[test] #[test]
fn role_new_no_metadata_has_none_fields() { fn role_new_no_metadata_has_none_fields() {
let role = Role::new("test", "Just a prompt"); let role = Role::new("test", "Just a prompt");
assert_eq!(role.model_id(), None); assert_eq!(role.model_id(), None);
assert_eq!(role.temperature(), None); assert_eq!(role.temperature(), None);
assert_eq!(role.top_p(), None); assert_eq!(role.top_p(), None);
@@ -588,9 +611,67 @@ mod tests {
assert_eq!(role.enabled_mcp_servers(), None); assert_eq!(role.enabled_mcp_servers(), None);
} }
#[test]
fn role_new_enabled_macros_absent_is_none() {
let role = Role::new("test", "---\ntemperature: 0.5\n---\nPrompt");
assert_eq!(role.enabled_macros, None);
}
#[test]
fn role_new_enabled_macros_empty_string_is_some_empty() {
let role = Role::new("test", "---\nenabled_macros: \"\"\n---\nPrompt");
assert_eq!(role.enabled_macros, Some(vec![]));
}
#[test]
fn role_new_enabled_macros_csv_string() {
let role = Role::new("test", "---\nenabled_macros: a, b\n---\nPrompt");
assert_eq!(
role.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
}
#[test]
fn role_new_enabled_macros_list() {
let role = Role::new("test", "---\nenabled_macros: [a, b]\n---\nPrompt");
assert_eq!(
role.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
}
#[test]
fn role_new_enabled_macros_null_is_none() {
let role = Role::new("test", "---\nenabled_macros: null\n---\nPrompt");
assert_eq!(role.enabled_macros, None);
}
#[test]
fn role_export_includes_enabled_macros() {
let role = Role::new("test", "---\nenabled_macros: [a]\n---\nPrompt");
let exported = role.export();
assert!(exported.contains("enabled_macros: [\"a\"]"));
}
#[test]
fn role_export_omits_enabled_macros_when_none() {
let role = Role::new("test", "Just a prompt");
assert!(!role.export().contains("enabled_macros"));
}
#[test] #[test]
fn role_builtin_shell_loads() { fn role_builtin_shell_loads() {
let role = Role::builtin("shell").unwrap(); let role = Role::builtin("shell").unwrap();
assert_eq!(role.name(), "shell"); assert_eq!(role.name(), "shell");
assert!(!role.prompt().is_empty()); assert!(!role.prompt().is_empty());
} }
@@ -598,6 +679,7 @@ mod tests {
#[test] #[test]
fn role_builtin_code_loads() { fn role_builtin_code_loads() {
let role = Role::builtin("code").unwrap(); let role = Role::builtin("code").unwrap();
assert_eq!(role.name(), "code"); assert_eq!(role.name(), "code");
assert!(!role.prompt().is_empty()); assert!(!role.prompt().is_empty());
} }
@@ -605,12 +687,14 @@ mod tests {
#[test] #[test]
fn role_builtin_nonexistent_errors() { fn role_builtin_nonexistent_errors() {
let result = Role::builtin("nonexistent_role_xyz"); let result = Role::builtin("nonexistent_role_xyz");
assert!(result.is_err()); assert!(result.is_err());
} }
#[test] #[test]
fn role_default_has_empty_fields() { fn role_default_has_empty_fields() {
let role = Role::default(); let role = Role::default();
assert_eq!(role.name(), ""); assert_eq!(role.name(), "");
assert_eq!(role.prompt(), ""); assert_eq!(role.prompt(), "");
assert_eq!(role.model_id(), None); assert_eq!(role.model_id(), None);
@@ -620,14 +704,18 @@ mod tests {
fn role_set_model_updates_model() { fn role_set_model_updates_model() {
let mut role = Role::new("test", "prompt"); let mut role = Role::new("test", "prompt");
let model = Model::default(); let model = Model::default();
role.set_model(model.clone()); role.set_model(model.clone());
assert_eq!(role.model().id(), model.id()); assert_eq!(role.model().id(), model.id());
} }
#[test] #[test]
fn role_set_temperature_works() { fn role_set_temperature_works() {
let mut role = Role::new("test", "prompt"); let mut role = Role::new("test", "prompt");
role.set_temperature(Some(0.5)); role.set_temperature(Some(0.5));
assert_eq!(role.temperature(), Some(0.5)); assert_eq!(role.temperature(), Some(0.5));
} }
@@ -635,7 +723,9 @@ mod tests {
fn role_export_includes_metadata() { fn role_export_includes_metadata() {
let content = "---\ntemperature: 0.8\n---\nMy prompt"; let content = "---\ntemperature: 0.8\n---\nMy prompt";
let role = Role::new("test", content); let role = Role::new("test", content);
let exported = role.export(); let exported = role.export();
assert!(exported.contains("temperature")); assert!(exported.contains("temperature"));
assert!(exported.contains("My prompt")); assert!(exported.contains("My prompt"));
} }
@@ -649,6 +739,7 @@ Input 1
### OUTPUT: ### OUTPUT:
Output 1 Output 1
"#; "#;
assert_eq!( assert_eq!(
parse_structure_prompt(prompt), parse_structure_prompt(prompt),
("System message", vec![("Input 1", "Output 1")]) ("System message", vec![("Input 1", "Output 1")])
@@ -663,6 +754,7 @@ Input 1
### OUTPUT: ### OUTPUT:
Output 1 Output 1
"#; "#;
assert_eq!( assert_eq!(
parse_structure_prompt(prompt), parse_structure_prompt(prompt),
("", vec![("Input 1", "Output 1")]) ("", vec![("Input 1", "Output 1")])
@@ -676,6 +768,7 @@ System message
### INPUT: ### INPUT:
Input 1 Input 1
"#; "#;
assert_eq!(parse_structure_prompt(prompt), (prompt, vec![])); assert_eq!(parse_structure_prompt(prompt), (prompt, vec![]));
} }
} }
+71
View File
@@ -46,6 +46,12 @@ pub struct Session {
deserialize_with = "super::deserialize_csv_or_vec" deserialize_with = "super::deserialize_csv_or_vec"
)] )]
enabled_skills: Option<Vec<String>>, enabled_skills: Option<Vec<String>>,
#[serde(
default,
skip_serializing_if = "Option::is_none",
deserialize_with = "super::deserialize_csv_or_vec"
)]
enabled_macros: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
save_session: Option<bool>, save_session: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
@@ -107,6 +113,10 @@ impl Session {
self.enabled_skills.as_deref() self.enabled_skills.as_deref()
} }
pub fn enabled_macros(&self) -> Option<&[String]> {
self.enabled_macros.as_deref()
}
pub fn set_skills_enabled(&mut self, value: Option<bool>) { pub fn set_skills_enabled(&mut self, value: Option<bool>) {
if self.skills_enabled != value { if self.skills_enabled != value {
self.skills_enabled = value; self.skills_enabled = value;
@@ -236,6 +246,9 @@ impl Session {
if let Some(enabled_skills) = self.enabled_skills() { if let Some(enabled_skills) = self.enabled_skills() {
data["enabled_skills"] = json!(enabled_skills); data["enabled_skills"] = json!(enabled_skills);
} }
if let Some(enabled_macros) = self.enabled_macros() {
data["enabled_macros"] = json!(enabled_macros);
}
if let Some(save_session) = self.save_session() { if let Some(save_session) = self.save_session() {
data["save_session"] = save_session.into(); data["save_session"] = save_session.into();
} }
@@ -315,6 +328,10 @@ impl Session {
items.push(("enabled_skills", enabled_skills.join(","))); items.push(("enabled_skills", enabled_skills.join(",")));
} }
if let Some(enabled_macros) = self.enabled_macros() {
items.push(("enabled_macros", enabled_macros.join(",")));
}
if let Some(save_session) = self.save_session() { if let Some(save_session) = self.save_session() {
items.push(("save_session", save_session.to_string())); items.push(("save_session", save_session.to_string()));
} }
@@ -925,12 +942,57 @@ mod tests {
#[test] #[test]
fn session_default_is_empty() { fn session_default_is_empty() {
let session = Session::default(); let session = Session::default();
assert!(session.is_empty()); assert!(session.is_empty());
assert_eq!(session.name(), ""); assert_eq!(session.name(), "");
assert_eq!(session.role_name(), None); assert_eq!(session.role_name(), None);
assert!(!session.dirty()); assert!(!session.dirty());
} }
#[test]
fn session_enabled_macros_absent_is_none() {
let session: Session = serde_yaml::from_str("model: provider:test\nmessages: []").unwrap();
assert_eq!(session.enabled_macros, None);
}
#[test]
fn session_enabled_macros_empty_list_is_some_empty() {
let session: Session =
serde_yaml::from_str("model: provider:test\nenabled_macros: []\nmessages: []").unwrap();
assert_eq!(session.enabled_macros, Some(vec![]));
}
#[test]
fn session_enabled_macros_empty_string_is_some_empty() {
let session: Session =
serde_yaml::from_str("model: provider:test\nenabled_macros: \"\"\nmessages: []")
.unwrap();
assert_eq!(session.enabled_macros, Some(vec![]));
}
#[test]
fn session_enabled_macros_csv_string() {
let session: Session =
serde_yaml::from_str("model: provider:test\nenabled_macros: \"a,b\"\nmessages: []")
.unwrap();
assert_eq!(
session.enabled_macros,
Some(vec!["a".to_string(), "b".to_string()])
);
}
#[test]
fn session_serialize_omits_enabled_macros_when_none() {
let session = Session::default();
let yaml = serde_yaml::to_string(&session).unwrap();
assert!(!yaml.contains("enabled_macros"));
}
#[test] #[test]
fn session_new_from_ctx_captures_save_session() { fn session_new_from_ctx_captures_save_session() {
let app_config = Arc::new(AppConfig::default()); let app_config = Arc::new(AppConfig::default());
@@ -945,6 +1007,7 @@ mod tests {
functions: Functions::default(), functions: Functions::default(),
}); });
let ctx = RequestContext::new(app_state, WorkingMode::Cmd); let ctx = RequestContext::new(app_state, WorkingMode::Cmd);
let session = Session::new_from_ctx(&ctx, &app_config, "test-session").unwrap(); let session = Session::new_from_ctx(&ctx, &app_config, "test-session").unwrap();
assert_eq!(session.name(), "test-session"); assert_eq!(session.name(), "test-session");
@@ -984,25 +1047,30 @@ mod tests {
#[test] #[test]
fn session_guard_empty_passes_when_empty() { fn session_guard_empty_passes_when_empty() {
let session = Session::default(); let session = Session::default();
assert!(session.guard_empty().is_ok()); assert!(session.guard_empty().is_ok());
} }
#[test] #[test]
fn session_needs_compression_threshold() { fn session_needs_compression_threshold() {
let session = Session::default(); let session = Session::default();
assert!(!session.needs_compression(4000)); assert!(!session.needs_compression(4000));
} }
#[test] #[test]
fn session_needs_compression_returns_false_when_compressing() { fn session_needs_compression_returns_false_when_compressing() {
let mut session = Session::default(); let mut session = Session::default();
session.set_compressing(true); session.set_compressing(true);
assert!(!session.needs_compression(0)); assert!(!session.needs_compression(0));
} }
#[test] #[test]
fn session_needs_compression_returns_false_when_threshold_zero() { fn session_needs_compression_returns_false_when_threshold_zero() {
let session = Session::default(); let session = Session::default();
assert!(!session.needs_compression(0)); assert!(!session.needs_compression(0));
} }
@@ -1074,13 +1142,16 @@ mod tests {
#[test] #[test]
fn session_need_autoname_default_false() { fn session_need_autoname_default_false() {
let session = Session::default(); let session = Session::default();
assert!(!session.need_autoname()); assert!(!session.need_autoname());
} }
#[test] #[test]
fn session_set_autonaming_doesnt_panic_without_autoname() { fn session_set_autonaming_doesnt_panic_without_autoname() {
let mut session = Session::default(); let mut session = Session::default();
session.set_autonaming(true); session.set_autonaming(true);
assert!(!session.need_autoname()); assert!(!session.need_autoname());
} }
+1474 -20
View File
File diff suppressed because it is too large Load Diff
+1986 -133
View File
File diff suppressed because it is too large Load Diff
+136 -5
View File
@@ -85,7 +85,17 @@ pub fn check_pending_agents_guardrail(ctx: &mut RequestContext) -> GuardrailActi
} }
ctx.pending_agents_guardrail_count += 1; ctx.pending_agents_guardrail_count += 1;
GuardrailAction::Inject(build_pending_agents_guardrail_prompt(&pending)) let mut prompt = build_pending_agents_guardrail_prompt(&pending);
if let Some(queue) = ctx.root_escalation_queue()
&& queue.has_pending()
{
let summary = serde_json::to_string(&queue.pending_summary()).unwrap_or_default();
prompt.push_str(&format!(
"\n\nAdditionally, child agents have pending escalations blocking them. Reply to each \
via `agent__reply_escalation` first:\n{summary}"
));
}
GuardrailAction::Inject(prompt)
} }
pub fn escalation_function_declarations() -> Vec<FunctionDeclaration> { pub fn escalation_function_declarations() -> Vec<FunctionDeclaration> {
@@ -620,14 +630,14 @@ async fn populate_agent_mcp_runtime(ctx: &mut RequestContext, server_ids: &[Stri
} }
fn sync_agent_functions_to_ctx(ctx: &mut RequestContext) -> Result<()> { fn sync_agent_functions_to_ctx(ctx: &mut RequestContext) -> Result<()> {
let server_names = ctx.tool_scope.mcp_runtime.server_names(); let server_features = ctx.tool_scope.mcp_runtime.server_features();
let functions = { let functions = {
let agent = ctx let agent = ctx
.agent .agent
.as_mut() .as_mut()
.with_context(|| "Agent should be initialized")?; .with_context(|| "Agent should be initialized")?;
if !server_names.is_empty() { if !server_features.is_empty() {
agent.append_mcp_meta_functions(server_names); agent.append_mcp_meta_functions(server_features);
} }
agent.functions().clone() agent.functions().clone()
}; };
@@ -1443,7 +1453,8 @@ async fn summarize_output(ctx: &RequestContext, agent_name: &str, output: &str)
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::config::{AppState, WorkingMode}; use crate::config::test_fixtures::{FixtureServer, fixture_runtime};
use crate::config::{AgentConfig, AppState, WorkingMode};
use crate::supervisor::escalation::{EscalationQueue, EscalationRequest}; use crate::supervisor::escalation::{EscalationQueue, EscalationRequest};
use serde_json::json; use serde_json::json;
use serial_test::serial; use serial_test::serial;
@@ -1500,6 +1511,28 @@ mod tests {
.block_on(f) .block_on(f)
} }
#[tokio::test]
async fn sync_agent_functions_gates_meta_functions_on_live_capabilities() {
let mut ctx = RequestContext::new(default_app_state(), WorkingMode::Cmd);
ctx.agent = Some(Agent::test_new(AgentConfig::default()));
let (runtime, _server) = fixture_runtime(FixtureServer {
tools_capability: false,
resources_capability: true,
..FixtureServer::default()
})
.await;
ctx.tool_scope.mcp_runtime = runtime;
sync_agent_functions_to_ctx(&mut ctx).unwrap();
let functions = &ctx.tool_scope.functions;
assert_eq!(functions.declarations().len(), 3);
assert!(functions.contains("mcp_search_fixture"));
assert!(functions.contains("mcp_describe_fixture"));
assert!(functions.contains("mcp_read_fixture"));
assert!(!functions.contains("mcp_invoke_fixture"));
}
#[test] #[test]
fn handle_list_running_empty_supervisor() { fn handle_list_running_empty_supervisor() {
let mut ctx = ctx_with_supervisor(4, 3); let mut ctx = ctx_with_supervisor(4, 3);
@@ -1996,4 +2029,102 @@ mod tests {
let q2 = ctx.ensure_root_escalation_queue(); let q2 = ctx.ensure_root_escalation_queue();
assert!(Arc::ptr_eq(&q1, &q2)); assert!(Arc::ptr_eq(&q1, &q2));
} }
#[test]
fn guardrail_prompt_mentions_pending_escalations() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
rt.block_on(async {
let mut ctx = ctx_with_supervisor(4, 3);
let join_handle = tokio::spawn(async {
time::sleep(Duration::from_secs(60)).await;
Ok(AgentResult {
id: "slow".into(),
agent_name: "test".into(),
output: String::new(),
exit_status: AgentExitStatus::Completed,
})
});
let handle = AgentHandle {
id: "slow".into(),
agent_name: "test".into(),
depth: 1,
inbox: Arc::new(Inbox::new()),
abort_signal: create_abort_signal(),
join_handle,
child_supervisor: None,
};
ctx.supervisor
.as_ref()
.unwrap()
.write()
.register(handle)
.unwrap();
let queue = ctx.ensure_root_escalation_queue();
let (tx, _rx) = tokio::sync::oneshot::channel();
queue.submit(EscalationRequest {
id: "esc_9".into(),
from_agent_id: "a1".into(),
from_agent_name: "explore".into(),
question: "Which option?".into(),
options: None,
reply_tx: tx,
});
match check_pending_agents_guardrail(&mut ctx) {
GuardrailAction::Inject(prompt) => {
assert!(prompt.contains("agent__reply_escalation"));
assert!(prompt.contains("esc_9"));
}
_ => panic!("expected Inject action"),
}
});
}
#[test]
fn guardrail_prompt_omits_escalations_when_none_pending() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
rt.block_on(async {
let mut ctx = ctx_with_supervisor(4, 3);
let join_handle = tokio::spawn(async {
time::sleep(Duration::from_secs(60)).await;
Ok(AgentResult {
id: "slow".into(),
agent_name: "test".into(),
output: String::new(),
exit_status: AgentExitStatus::Completed,
})
});
let handle = AgentHandle {
id: "slow".into(),
agent_name: "test".into(),
depth: 1,
inbox: Arc::new(Inbox::new()),
abort_signal: create_abort_signal(),
join_handle,
child_supervisor: None,
};
ctx.supervisor
.as_ref()
.unwrap()
.write()
.register(handle)
.unwrap();
match check_pending_agents_guardrail(&mut ctx) {
GuardrailAction::Inject(prompt) => {
assert!(!prompt.contains("agent__reply_escalation"));
}
_ => panic!("expected Inject action"),
}
});
}
} }
+64 -1
View File
@@ -250,7 +250,30 @@ impl GraphExecutor {
branch_tasks.push(task); branch_tasks.push(task);
} }
let joined = join_all(branch_tasks).await; let joined = match graph_timeout {
Some(t) => {
let remaining = t.saturating_sub(start.elapsed());
let abort_handles: Vec<_> = branch_tasks
.iter()
.map(|task| task.abort_handle())
.collect();
match tokio::time::timeout(remaining, join_all(branch_tasks)).await {
Ok(joined) => joined,
Err(_) => {
for handle in abort_handles {
handle.abort();
}
bail!(
"Graph '{}' timed out after {}s during super-step with frontier {:?}",
graph.name,
t.as_secs(),
sorted_frontier(&frontier)
);
}
}
}
None => join_all(branch_tasks).await,
};
let mut branch_writes: Vec<BranchWrites> = Vec::new(); let mut branch_writes: Vec<BranchWrites> = Vec::new();
let mut next_frontier: HashSet<String> = HashSet::new(); let mut next_frontier: HashSet<String> = HashSet::new();
@@ -793,4 +816,44 @@ nodes:
"error should list both End nodes: {err}" "error should list both End nodes: {err}"
); );
} }
#[tokio::test]
async fn graph_timeout_interrupts_in_flight_super_step() {
if !cmd_available("bash") {
eprintln!("skipping: bash not available");
return;
}
let ws = TestWorkspace::new();
ws.write_script("sleeper.sh", "#!/bin/bash\nsleep 10\necho '{}'\n");
let yaml = r#"
name: inflight_timeout_test
start: sleeper
settings:
timeout: 1
nodes:
sleeper:
type: script
script: sleeper.sh
state_updates: {}
next: done
done:
type: end
output: "done"
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
let mut ctx = make_ctx();
let abort = create_abort_signal();
let result = GraphExecutor::new(graph, &ws.dir)
.execute(&mut ctx, abort)
.await;
assert!(result.is_err(), "expected in-flight timeout to error");
let err = format!("{:#}", result.unwrap_err());
assert!(
err.contains("timed out after 1s during super-step"),
"error should report during-super-step timeout: {err}"
);
assert!(err.contains("sleeper"), "error should name frontier: {err}");
}
} }
+1
View File
@@ -54,6 +54,7 @@ impl ScriptExecutor {
let mut cmd = build_command(language, &script_path)?; let mut cmd = build_command(language, &script_path)?;
cmd.stdout(Stdio::piped()); cmd.stdout(Stdio::piped());
cmd.stderr(Stdio::piped()); cmd.stderr(Stdio::piped());
cmd.kill_on_drop(true);
cmd.envs(&self.extra_envs); cmd.envs(&self.extra_envs);
cmd.env("AUTO_CONFIRM", "true"); cmd.env("AUTO_CONFIRM", "true");
match &state_repr { match &state_repr {
+10
View File
@@ -588,6 +588,16 @@ nodes:
)); ));
} }
#[test]
fn graph_silently_ignores_enabled_macros_key() {
let yaml = "name: g\nenabled_macros: [\"x\"]\nstart: x\nnodes:\n x:\n id: x\n type: end\n output: ok\n";
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
assert_eq!(graph.name, "g");
assert_eq!(graph.start, "x");
}
#[test] #[test]
fn graph_settings_have_sensible_defaults() { fn graph_settings_have_sensible_defaults() {
let yaml = "name: g\nstart: x\nnodes:\n x:\n id: x\n type: end\n output: ok\n"; let yaml = "name: g\nstart: x\nnodes:\n x:\n id: x\n type: end\n output: ok\n";
+1 -1
View File
@@ -23,7 +23,7 @@ impl ApprovalNodeExecutor {
let response = handle_user_tool( let response = handle_user_tool(
ctx, ctx,
&format!("{USER_FUNCTION_PREFIX}ask"), &format!("{USER_FUNCTION_PREFIX}select"),
&json!({ "question": question, "options": node.options }), &json!({ "question": question, "options": node.options }),
) )
.await .await
+25 -5
View File
@@ -127,14 +127,31 @@ async fn main() -> Result<()> {
return sandbox::launch(name.clone(), cli.fresh); return sandbox::launch(name.clone(), cli.fresh);
} }
if cli.list_bundles {
return config::list_installed_bundles();
}
install_builtins()?; install_builtins()?;
if let Some(category) = cli.install { if let Some(category) = cli.install_builtins {
return config::install_assets(category); return config::install_assets(category);
} }
if let Some(url) = cli.install_from.as_deref() { if let Some(value) = cli.install.as_deref() {
return config::install_remote(url, cli.filter, cli.install_force); return config::install_or_update(
value,
cli.git_host.as_deref(),
cli.filter,
cli.install_force,
);
}
if let Some(spec) = cli.update_bundle.as_deref() {
return config::update_bundle(spec, cli.yes);
}
if let Some(name) = cli.uninstall.as_deref() {
return config::uninstall_bundle(name, cli.yes);
} }
if let Some(client_arg) = &cli.authenticate { if let Some(client_arg) = &cli.authenticate {
@@ -222,6 +239,9 @@ async fn main() -> Result<()> {
if cli.no_workspace_mcp { if cli.no_workspace_mcp {
app_config.no_workspace_mcp = true; app_config.no_workspace_mcp = true;
} }
if cli.no_workspace_macros {
app_config.no_workspace_macros = true;
}
let app_config: Arc<AppConfig> = Arc::new(app_config); let app_config: Arc<AppConfig> = Arc::new(app_config);
let app_state: Arc<AppState> = Arc::new( let app_state: Arc<AppState> = Arc::new(
AppState::init( AppState::init(
@@ -360,8 +380,8 @@ async fn run(
.await?; .await?;
} else { } else {
let app: Arc<AppConfig> = Arc::clone(&ctx.app.config); let app: Arc<AppConfig> = Arc::clone(&ctx.app.config);
if let Some(prompt) = &cli.prompt { if let Some(prompt) = &cli.temp_role {
ctx.use_prompt(app.as_ref(), prompt)?; ctx.use_temp_role(app.as_ref(), prompt)?;
} else if let Some(name) = &cli.role { } else if let Some(name) = &cli.role {
ctx.use_role(app.as_ref(), name, abort_signal.clone()) ctx.use_role(app.as_ref(), name, abort_signal.clone())
.await?; .await?;
+651
View File
@@ -0,0 +1,651 @@
use crate::mcp::oauth::{force_refresh_mcp_token, load_or_refresh_mcp_token};
use http::{HeaderName, HeaderValue};
use log::debug;
use rmcp::model::ClientJsonRpcMessage;
use rmcp::transport::common::client_side_sse::BoxedSseResponse;
use rmcp::transport::streamable_http_client::{
AuthRequiredError, StreamableHttpClient, StreamableHttpError, StreamableHttpPostResponse,
};
use std::collections::HashMap;
use std::future::Future;
use std::sync::Arc;
/// [`StreamableHttpClient`] wrapper that injects the OAuth bearer token for an
/// MCP server on every request instead of pinning it at spawn time, so tokens
/// refreshed mid-session take effect without reconnecting.
///
/// A caller-supplied `auth_header` always passes through untouched; only a
/// `None` header is filled from the stored token. When the wrapper injected
/// the token and a POST comes back 401, it forces a token refresh and retries
/// exactly once (see [`Self::post_with_retry`]).
#[derive(Clone)]
pub struct McpOAuthClient<C = reqwest::Client> {
inner: C,
server: Arc<str>,
}
impl<C> McpOAuthClient<C> {
pub fn new(inner: C, server: &str) -> Self {
Self {
inner,
server: Arc::from(server),
}
}
}
impl<C: StreamableHttpClient + Sync> McpOAuthClient<C> {
/// Resolves the effective auth header. Caller-supplied values pass through
/// untouched; `None` is filled from the stored token for this server.
/// Returns the header plus whether the wrapper injected it. Errors with
/// [`StreamableHttpError::AuthRequired`] (without contacting the server)
/// when no usable token exists.
async fn resolve_auth(
&self,
auth_header: Option<String>,
) -> Result<(Option<String>, bool), StreamableHttpError<C::Error>> {
if auth_header.is_some() {
return Ok((auth_header, false));
}
match load_or_refresh_mcp_token(&self.server).await.into_token() {
Some(token) => Ok((Some(token), true)),
None => Err(self.auth_required()),
}
}
fn auth_required(&self) -> StreamableHttpError<C::Error> {
StreamableHttpError::AuthRequired(AuthRequiredError::new(format!(
"no valid OAuth token for MCP server '{server}'; \
run `.mcp auth {server}` to re-authenticate",
server = self.server
)))
}
async fn post_with_retry<F, Fut>(
&self,
auth_header: Option<String>,
mut post: F,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<C::Error>>
where
F: FnMut(Option<String>) -> Fut,
Fut: Future<Output = Result<StreamableHttpPostResponse, StreamableHttpError<C::Error>>>,
{
let (auth, injected) = self.resolve_auth(auth_header).await?;
let (original, rejected) = match (post(auth.clone()).await, auth) {
(Err(err @ StreamableHttpError::AuthRequired(_)), Some(rejected)) if injected => {
(err, rejected)
}
(result, _) => return result,
};
debug!(
"MCP server '{}' rejected the injected token; forcing a refresh and retrying once",
self.server
);
let Some(token) = force_refresh_mcp_token(&self.server, &rejected).await else {
return Err(original);
};
match post(Some(token)).await {
Err(StreamableHttpError::AuthRequired(_)) => {
debug!(
"Retry after forced token refresh was rejected again by MCP server '{}'",
self.server
);
Err(original)
}
result => result,
}
}
}
impl<C: StreamableHttpClient + Sync> StreamableHttpClient for McpOAuthClient<C> {
type Error = C::Error;
async fn post_message(
&self,
uri: Arc<str>,
message: ClientJsonRpcMessage,
session_id: Option<Arc<str>>,
auth_header: Option<String>,
custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<Self::Error>> {
self.post_with_retry(auth_header, |auth| {
self.inner.post_message(
uri.clone(),
message.clone(),
session_id.clone(),
auth,
custom_headers.clone(),
)
})
.await
}
/// Overridden rather than left to the trait default: the default impl
/// delegates to [`Self::post_message`], silently dropping the
/// transport-wide SSE event size limit. Delegating to the inner client's
/// size-enforcing variant keeps the limit applied at the raw byte layer.
async fn post_message_with_max_sse_event_size(
&self,
uri: Arc<str>,
message: ClientJsonRpcMessage,
session_id: Option<Arc<str>>,
auth_header: Option<String>,
custom_headers: HashMap<HeaderName, HeaderValue>,
max_sse_event_size: usize,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<Self::Error>> {
self.post_with_retry(auth_header, |auth| {
self.inner.post_message_with_max_sse_event_size(
uri.clone(),
message.clone(),
session_id.clone(),
auth,
custom_headers.clone(),
max_sse_event_size,
)
})
.await
}
async fn delete_session(
&self,
uri: Arc<str>,
session_id: Arc<str>,
auth_header: Option<String>,
custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<(), StreamableHttpError<Self::Error>> {
let (auth, _) = self.resolve_auth(auth_header).await?;
self.inner
.delete_session(uri, session_id, auth, custom_headers)
.await
}
async fn get_stream(
&self,
uri: Arc<str>,
session_id: Option<Arc<str>>,
last_event_id: Option<String>,
auth_header: Option<String>,
custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<BoxedSseResponse, StreamableHttpError<Self::Error>> {
let (auth, _) = self.resolve_auth(auth_header).await?;
self.inner
.get_stream(uri, session_id, last_event_id, auth, custom_headers)
.await
}
/// Overridden for the same reason as
/// [`Self::post_message_with_max_sse_event_size`]: the trait default
/// bypasses SSE event size enforcement.
async fn get_stream_with_max_sse_event_size(
&self,
uri: Arc<str>,
session_id: Option<Arc<str>>,
last_event_id: Option<String>,
auth_header: Option<String>,
custom_headers: HashMap<HeaderName, HeaderValue>,
max_sse_event_size: usize,
) -> Result<BoxedSseResponse, StreamableHttpError<Self::Error>> {
let (auth, _) = self.resolve_auth(auth_header).await?;
self.inner
.get_stream_with_max_sse_event_size(
uri,
session_id,
last_event_id,
auth,
custom_headers,
max_sse_event_size,
)
.await
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::config::paths;
use crate::mcp::oauth::test_support::with_temp_cache;
use futures_util::StreamExt;
use parking_lot::Mutex;
use serial_test::serial;
use std::convert::Infallible;
use std::fs;
use std::sync::atomic::{AtomicUsize, Ordering};
const FRESH: i64 = 9999999999;
type Calls = Arc<Mutex<Vec<(&'static str, Option<String>)>>>;
type AfterFirstCallAction = Option<Box<dyn FnOnce() + Send + 'static>>;
/// Inner client that records `(method, auth_header)` per call, rejects the
/// first `reject_times` POSTs/streams with `AuthRequired` (then the next
/// `transport_error_times` with a non-auth error), and runs an optional
/// side effect after the first call (to mutate token files between the
/// initial attempt and the retry).
#[derive(Clone, Default)]
struct FakeInner {
calls: Calls,
reject_times: Arc<AtomicUsize>,
transport_error_times: Arc<AtomicUsize>,
after_first_call: Arc<Mutex<AfterFirstCallAction>>,
}
impl FakeInner {
fn record(
&self,
method: &'static str,
auth: Option<String>,
) -> Result<(), StreamableHttpError<Infallible>> {
self.calls.lock().push((method, auth));
if let Some(f) = self.after_first_call.lock().take() {
f();
}
if self.reject_times.load(Ordering::SeqCst) > 0 {
self.reject_times.fetch_sub(1, Ordering::SeqCst);
return Err(Self::rejection());
}
if self.transport_error_times.load(Ordering::SeqCst) > 0 {
self.transport_error_times.fetch_sub(1, Ordering::SeqCst);
return Err(StreamableHttpError::UnexpectedServerResponse(
"connection reset".into(),
));
}
Ok(())
}
fn rejection() -> StreamableHttpError<Infallible> {
StreamableHttpError::AuthRequired(AuthRequiredError::new(
"Bearer error=\"invalid_token\"".to_string(),
))
}
}
impl StreamableHttpClient for FakeInner {
type Error = Infallible;
async fn post_message(
&self,
_uri: Arc<str>,
_message: ClientJsonRpcMessage,
_session_id: Option<Arc<str>>,
auth_header: Option<String>,
_custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<Self::Error>> {
self.record("post_message", auth_header)?;
Ok(StreamableHttpPostResponse::Accepted)
}
async fn post_message_with_max_sse_event_size(
&self,
_uri: Arc<str>,
_message: ClientJsonRpcMessage,
_session_id: Option<Arc<str>>,
auth_header: Option<String>,
_custom_headers: HashMap<HeaderName, HeaderValue>,
_max_sse_event_size: usize,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<Self::Error>> {
self.record("post_message_with_max_sse_event_size", auth_header)?;
Ok(StreamableHttpPostResponse::Accepted)
}
async fn delete_session(
&self,
_uri: Arc<str>,
_session_id: Arc<str>,
auth_header: Option<String>,
_custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<(), StreamableHttpError<Self::Error>> {
self.record("delete_session", auth_header)?;
Ok(())
}
async fn get_stream(
&self,
_uri: Arc<str>,
_session_id: Option<Arc<str>>,
_last_event_id: Option<String>,
auth_header: Option<String>,
_custom_headers: HashMap<HeaderName, HeaderValue>,
) -> Result<BoxedSseResponse, StreamableHttpError<Self::Error>> {
self.record("get_stream", auth_header)?;
Ok(futures_util::stream::empty().boxed())
}
async fn get_stream_with_max_sse_event_size(
&self,
_uri: Arc<str>,
_session_id: Option<Arc<str>>,
_last_event_id: Option<String>,
auth_header: Option<String>,
_custom_headers: HashMap<HeaderName, HeaderValue>,
_max_sse_event_size: usize,
) -> Result<BoxedSseResponse, StreamableHttpError<Self::Error>> {
self.record("get_stream_with_max_sse_event_size", auth_header)?;
Ok(futures_util::stream::empty().boxed())
}
}
fn write_token_file(server: &str, access_token: &str, expires_at: i64) {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file(&format!("mcp_{server}")),
format!(
r#"{{"access_token":"{access_token}","refresh_token":"r","expires_at":{expires_at}}}"#
),
)
.unwrap();
}
fn ping() -> ClientJsonRpcMessage {
serde_json::from_value(serde_json::json!({
"jsonrpc": "2.0",
"id": 1,
"method": "ping"
}))
.unwrap()
}
fn rt() -> tokio::runtime::Runtime {
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
}
fn post(
client: &McpOAuthClient<FakeInner>,
auth_header: Option<String>,
) -> Result<StreamableHttpPostResponse, StreamableHttpError<Infallible>> {
rt().block_on(client.post_message(
Arc::from("http://mcp.test/mcp"),
ping(),
None,
auth_header,
HashMap::new(),
))
}
#[test]
#[serial]
fn injects_token_from_disk_when_auth_header_none() {
with_temp_cache(|| {
write_token_file("wrapper-inject", "tok-live", FRESH);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-inject");
let result = post(&client, None);
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![("post_message", Some("tok-live".to_string()))]
);
});
}
#[test]
#[serial]
fn caller_supplied_auth_header_passes_through() {
with_temp_cache(|| {
write_token_file("wrapper-passthrough", "tok-disk", FRESH);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-passthrough");
let result = post(&client, Some("caller-tok".to_string()));
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![("post_message", Some("caller-tok".to_string()))]
);
});
}
#[test]
#[serial]
fn caller_supplied_header_rejection_propagates_without_refresh() {
with_temp_cache(|| {
// A fresh, different token sits on disk: if the injected guard
// were dropped, the wrapper would refresh and retry with it.
write_token_file("wrapper-caller-401", "tok-disk", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(1, Ordering::SeqCst);
let client = McpOAuthClient::new(inner.clone(), "wrapper-caller-401");
let result = post(&client, Some("caller-tok".to_string()));
assert!(matches!(result, Err(StreamableHttpError::AuthRequired(_))));
assert_eq!(
*inner.calls.lock(),
vec![("post_message", Some("caller-tok".to_string()))]
);
});
}
#[test]
#[serial]
fn missing_token_returns_auth_required_without_calling_inner() {
with_temp_cache(|| {
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-no-token");
let result = post(&client, None);
assert!(matches!(result, Err(StreamableHttpError::AuthRequired(_))));
assert!(inner.calls.lock().is_empty());
});
}
#[test]
#[serial]
fn rejected_token_forces_refresh_and_retries_once() {
with_temp_cache(|| {
write_token_file("wrapper-retry", "tok-a", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(1, Ordering::SeqCst);
// Simulate a concurrent refresh landing between the rejection and
// the forced refresh: the retry must carry the new token.
*inner.after_first_call.lock() = Some(Box::new(|| {
write_token_file("wrapper-retry", "tok-b", FRESH);
}));
let client = McpOAuthClient::new(inner.clone(), "wrapper-retry");
let result = post(&client, None);
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![
("post_message", Some("tok-a".to_string())),
("post_message", Some("tok-b".to_string())),
]
);
});
}
#[test]
#[serial]
fn failed_force_refresh_propagates_original_error_after_one_call() {
with_temp_cache(|| {
write_token_file("wrapper-refresh-fail", "tok-a", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(1, Ordering::SeqCst);
// Token file gone by refresh time: force refresh yields nothing.
*inner.after_first_call.lock() = Some(Box::new(|| {
fs::remove_file(paths::token_file("mcp_wrapper-refresh-fail")).unwrap();
}));
let client = McpOAuthClient::new(inner.clone(), "wrapper-refresh-fail");
let result = post(&client, None);
assert!(matches!(result, Err(StreamableHttpError::AuthRequired(_))));
assert_eq!(inner.calls.lock().len(), 1);
});
}
#[test]
#[serial]
fn second_rejection_after_retry_propagates_original_error() {
with_temp_cache(|| {
write_token_file("wrapper-double-401", "tok-a", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(2, Ordering::SeqCst);
// A changed token appears before the forced refresh, so the retry
// actually runs (an unchanged token would trigger a real refresh
// attempt, which fails without a cached registration).
*inner.after_first_call.lock() = Some(Box::new(|| {
write_token_file("wrapper-double-401", "tok-b", FRESH);
}));
let client = McpOAuthClient::new(inner.clone(), "wrapper-double-401");
let result = post(&client, None);
assert!(matches!(result, Err(StreamableHttpError::AuthRequired(_))));
assert_eq!(inner.calls.lock().len(), 2);
});
}
#[test]
#[serial]
fn non_auth_retry_error_propagates_as_is() {
with_temp_cache(|| {
write_token_file("wrapper-retry-transport", "tok-a", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(1, Ordering::SeqCst);
inner.transport_error_times.store(1, Ordering::SeqCst);
*inner.after_first_call.lock() = Some(Box::new(|| {
write_token_file("wrapper-retry-transport", "tok-b", FRESH);
}));
let client = McpOAuthClient::new(inner.clone(), "wrapper-retry-transport");
let result = post(&client, None);
assert!(matches!(
result,
Err(StreamableHttpError::UnexpectedServerResponse(_))
));
assert_eq!(inner.calls.lock().len(), 2);
});
}
#[test]
#[serial]
fn sized_post_delegates_to_inner_sized_variant() {
with_temp_cache(|| {
write_token_file("wrapper-sized-post", "tok-live", FRESH);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-sized-post");
let result = rt().block_on(client.post_message_with_max_sse_event_size(
Arc::from("http://mcp.test/mcp"),
ping(),
None,
None,
HashMap::new(),
4096,
));
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![(
"post_message_with_max_sse_event_size",
Some("tok-live".to_string())
)]
);
});
}
#[test]
#[serial]
fn sized_get_stream_delegates_to_inner_sized_variant() {
with_temp_cache(|| {
write_token_file("wrapper-sized-get", "tok-live", FRESH);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-sized-get");
let result = rt().block_on(client.get_stream_with_max_sse_event_size(
Arc::from("http://mcp.test/mcp"),
None,
None,
None,
HashMap::new(),
4096,
));
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![(
"get_stream_with_max_sse_event_size",
Some("tok-live".to_string())
)]
);
});
}
#[test]
#[serial]
fn get_stream_does_not_retry_on_rejection() {
with_temp_cache(|| {
write_token_file("wrapper-get-401", "tok-live", FRESH);
let inner = FakeInner::default();
inner.reject_times.store(1, Ordering::SeqCst);
let client = McpOAuthClient::new(inner.clone(), "wrapper-get-401");
let result = rt().block_on(client.get_stream(
Arc::from("http://mcp.test/mcp"),
None,
None,
None,
HashMap::new(),
));
assert!(matches!(result, Err(StreamableHttpError::AuthRequired(_))));
assert_eq!(inner.calls.lock().len(), 1);
});
}
#[test]
#[serial]
fn delete_session_injects_token() {
with_temp_cache(|| {
write_token_file("wrapper-delete", "tok-live", FRESH);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-delete");
let result = rt().block_on(client.delete_session(
Arc::from("http://mcp.test/mcp"),
Arc::from("session-1"),
None,
HashMap::new(),
));
assert!(result.is_ok());
assert_eq!(
*inner.calls.lock(),
vec![("delete_session", Some("tok-live".to_string()))]
);
});
}
#[test]
#[serial]
fn auth_required_error_contains_no_token_material() {
with_temp_cache(|| {
write_token_file("wrapper-redact", "stale-secret-token", 0);
let inner = FakeInner::default();
let client = McpOAuthClient::new(inner.clone(), "wrapper-redact");
let err = post(&client, None).unwrap_err();
let display = format!("{err}");
let debug = format!("{err:?}");
assert!(!display.contains("stale-secret-token"));
assert!(!debug.contains("stale-secret-token"));
assert!(inner.calls.lock().is_empty());
});
}
}
+464 -59
View File
@@ -1,5 +1,7 @@
mod auth_client;
pub(crate) mod manage; pub(crate) mod manage;
pub(crate) mod oauth; pub(crate) mod oauth;
pub(crate) mod render;
mod sse_transport; mod sse_transport;
use crate::config::AppConfig; use crate::config::AppConfig;
@@ -9,10 +11,12 @@ use crate::vault::Vault;
use crate::vault::interpolate_secrets; use crate::vault::interpolate_secrets;
use anyhow::Error; use anyhow::Error;
use anyhow::{Context, Result, anyhow}; use anyhow::{Context, Result, anyhow};
use auth_client::McpOAuthClient;
use futures_util::{StreamExt, TryStreamExt, stream}; use futures_util::{StreamExt, TryStreamExt, stream};
use http::{HeaderName, HeaderValue}; use http::{HeaderName, HeaderValue};
use indexmap::IndexMap; use indexmap::IndexMap;
use indoc::formatdoc; use indoc::formatdoc;
use rmcp::model::{PromptArgument, ServerCapabilities};
use rmcp::service::RunningService; use rmcp::service::RunningService;
use rmcp::transport::StreamableHttpClientTransport; use rmcp::transport::StreamableHttpClientTransport;
use rmcp::transport::TokioChildProcess; use rmcp::transport::TokioChildProcess;
@@ -21,6 +25,8 @@ use rmcp::{RoleClient, ServiceExt};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use sse_transport::LegacySseTransport; use sse_transport::LegacySseTransport;
use std::collections::{HashMap, HashSet}; use std::collections::{HashMap, HashSet};
use std::fmt;
use std::fmt::Display;
use std::fs::OpenOptions; use std::fs::OpenOptions;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::Stdio; use std::process::Stdio;
@@ -30,27 +36,97 @@ use tokio::process::Command;
pub const MCP_INVOKE_META_FUNCTION_NAME_PREFIX: &str = "mcp_invoke"; pub const MCP_INVOKE_META_FUNCTION_NAME_PREFIX: &str = "mcp_invoke";
pub const MCP_SEARCH_META_FUNCTION_NAME_PREFIX: &str = "mcp_search"; pub const MCP_SEARCH_META_FUNCTION_NAME_PREFIX: &str = "mcp_search";
pub const MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX: &str = "mcp_describe"; pub const MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX: &str = "mcp_describe";
pub const MCP_READ_META_FUNCTION_NAME_PREFIX: &str = "mcp_read";
pub const MCP_PROMPT_META_FUNCTION_NAME_PREFIX: &str = "mcp_prompt";
pub const MCP_META_FUNCTION_PREFIXES: [&str; 5] = [
MCP_INVOKE_META_FUNCTION_NAME_PREFIX,
MCP_SEARCH_META_FUNCTION_NAME_PREFIX,
MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX,
MCP_READ_META_FUNCTION_NAME_PREFIX,
MCP_PROMPT_META_FUNCTION_NAME_PREFIX,
];
pub fn is_mcp_meta_function(name: &str) -> bool {
MCP_META_FUNCTION_PREFIXES
.iter()
.any(|prefix| name.starts_with(prefix))
}
pub fn mcp_meta_function_names(server: &str) -> Vec<String> {
MCP_META_FUNCTION_PREFIXES
.iter()
.map(|prefix| format!("{prefix}_{server}"))
.collect()
}
pub type ConnectedServer = RunningService<RoleClient, ()>; pub type ConnectedServer = RunningService<RoleClient, ()>;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct McpServerFeatures {
pub name: String,
pub tools: bool,
pub resources: bool,
pub prompts: bool,
}
impl McpServerFeatures {
pub fn from_capabilities(
name: impl Into<String>,
capabilities: Option<&ServerCapabilities>,
) -> Self {
Self {
name: name.into(),
tools: capabilities.is_none_or(|c| c.tools.is_some()),
resources: capabilities.is_some_and(|c| c.resources.is_some()),
prompts: capabilities.is_some_and(|c| c.prompts.is_some()),
}
}
}
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum CatalogItemKind {
#[default]
Tool,
Resource,
ResourceTemplate,
Prompt,
}
impl CatalogItemKind {
pub fn as_str(self) -> &'static str {
match self {
Self::Tool => "tool",
Self::Resource => "resource",
Self::ResourceTemplate => "resource_template",
Self::Prompt => "prompt",
}
}
}
impl Display for CatalogItemKind {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(self.as_str())
}
}
#[derive(Clone, Debug, Default, Serialize)] #[derive(Clone, Debug, Default, Serialize)]
pub struct CatalogItem { pub struct CatalogItem {
pub kind: CatalogItemKind,
pub name: String, pub name: String,
pub server: String, pub server: String,
pub description: String, pub description: String,
} #[serde(skip_serializing_if = "Option::is_none")]
pub uri: Option<String>,
#[derive(Debug)] #[serde(skip_serializing_if = "Option::is_none")]
struct ServerCatalog { pub mime_type: Option<String>,
items: HashMap<String, CatalogItem>, #[serde(skip_serializing_if = "Option::is_none")]
} pub size: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
impl Clone for ServerCatalog { pub arguments: Option<Vec<PromptArgument>>,
fn clone(&self) -> Self { #[serde(skip_serializing_if = "Option::is_none")]
Self { pub audience: Option<Vec<String>>,
items: self.items.clone(),
}
}
} }
#[derive(Debug, Clone, Deserialize, Serialize)] #[derive(Debug, Clone, Deserialize, Serialize)]
@@ -156,7 +232,6 @@ pub struct McpRegistry {
log_path: Option<PathBuf>, log_path: Option<PathBuf>,
config: Option<McpServersConfig>, config: Option<McpServersConfig>,
servers: HashMap<String, Arc<ConnectedServer>>, servers: HashMap<String, Arc<ConnectedServer>>,
catalogs: HashMap<String, ServerCatalog>,
} }
impl McpRegistry { impl McpRegistry {
@@ -299,7 +374,7 @@ impl McpRegistry {
debug!("Starting selected MCP servers: {:?}", ids_to_start); debug!("Starting selected MCP servers: {:?}", ids_to_start);
let results: Vec<Option<(String, Arc<ConnectedServer>, ServerCatalog)>> = stream::iter( let results: Vec<Option<(String, Arc<ConnectedServer>)>> = stream::iter(
ids_to_start ids_to_start
.into_iter() .into_iter()
.map(|id| async { self.start_server(id).await }), .map(|id| async { self.start_server(id).await }),
@@ -308,66 +383,40 @@ impl McpRegistry {
.try_collect() .try_collect()
.await?; .await?;
for (id, server, catalog) in results.into_iter().flatten() { for (id, server) in results.into_iter().flatten() {
self.servers.insert(id.clone(), server); self.servers.insert(id, server);
self.catalogs.insert(id, catalog);
} }
Ok(()) Ok(())
} }
async fn start_server( async fn start_server(&self, id: String) -> Result<Option<(String, Arc<ConnectedServer>)>> {
&self,
id: String,
) -> Result<Option<(String, Arc<ConnectedServer>, ServerCatalog)>> {
let spec = self let spec = self
.config .config
.as_ref() .as_ref()
.and_then(|c| c.mcp_servers.get(&id)) .and_then(|c| c.mcp_servers.get(&id))
.with_context(|| format!("MCP server not found in config: {id}"))?; .with_context(|| format!("MCP server not found in config: {id}"))?;
let bearer_token = if spec.is_remote() { let (auth, auth_reason) = resolve_http_auth(&id, spec).await;
oauth::load_valid_mcp_token(&id)
} else {
None
};
let service = match spawn_mcp_server(spec, self.log_path.as_deref(), bearer_token).await { let service = match spawn_mcp_server(spec, self.log_path.as_deref(), auth).await {
Ok(s) => s, Ok(s) => s,
Err(e) if is_auth_required_error(&e) => { Err(e) if is_auth_required_error(&e) => {
warn!( warn!(
"MCP server '{id}' requires OAuth authentication. \ "{}",
Run `coyote --auth-mcp {id}` or `.mcp auth {id}` in the REPL to authenticate." McpAuthRequired {
server: id,
reason: auth_reason,
}
); );
return Ok(None); return Ok(None);
} }
Err(e) => return Err(e), Err(e) => return Err(e),
}; };
let tools = service.list_tools(None).await?;
debug!("Available tools for MCP server {id}: {tools:?}");
let mut items_vec = Vec::new();
for t in tools.tools {
let name = t.name.to_string();
let description = t.description.unwrap_or_default().to_string();
items_vec.push(CatalogItem {
name,
server: id.clone(),
description,
});
}
let mut items_map = HashMap::new();
items_vec.into_iter().for_each(|it| {
items_map.insert(it.name.clone(), it);
});
let catalog = ServerCatalog { items: items_map };
info!("Started MCP server: {id}"); info!("Started MCP server: {id}");
Ok(Some((id.to_string(), service, catalog))) Ok(Some((id, service)))
} }
fn resolve_server_ids(&self, enabled_mcp_servers: Option<Vec<String>>) -> Vec<String> { fn resolve_server_ids(&self, enabled_mcp_servers: Option<Vec<String>>) -> Vec<String> {
@@ -395,8 +444,21 @@ impl McpRegistry {
&self.servers &self.servers
} }
pub fn list_started_servers(&self) -> Vec<String> { pub fn server_features(&self) -> Vec<McpServerFeatures> {
self.servers.keys().cloned().collect() let mut features: Vec<McpServerFeatures> = self
.servers
.iter()
.map(|(name, handle)| {
let info = handle.peer_info();
McpServerFeatures::from_capabilities(
name.as_str(),
info.as_ref().map(|info| &info.capabilities),
)
})
.collect();
features.sort_by(|a, b| a.name.cmp(&b.name));
features
} }
pub fn is_empty(&self) -> bool { pub fn is_empty(&self) -> bool {
@@ -412,19 +474,76 @@ impl McpRegistry {
} }
} }
/// How a remote MCP server authenticates outgoing requests.
pub(crate) enum HttpAuth {
/// Only the static headers from the server spec; no OAuth token.
StaticOnly,
/// OAuth-managed: HTTP transports inject a fresh bearer token per request
/// via [`McpOAuthClient`] (ignoring the carried token); SSE transports
/// send the carried token as a static header.
Managed { server: String, token: String },
}
impl fmt::Debug for HttpAuth {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::StaticOnly => f.write_str("StaticOnly"),
Self::Managed { server, token: _ } => f
.debug_struct("Managed")
.field("server", server)
.field("token", &"<redacted>")
.finish(),
}
}
}
impl HttpAuth {
pub(crate) fn from_token_status(status: &oauth::McpTokenStatus, server: &str) -> Self {
match status {
oauth::McpTokenStatus::Token(token) => Self::Managed {
server: server.to_string(),
token: token.clone(),
},
oauth::McpTokenStatus::NotAuthenticated | oauth::McpTokenStatus::RefreshFailed => {
Self::StaticOnly
}
}
}
}
pub(crate) async fn resolve_http_auth(name: &str, spec: &McpServer) -> (HttpAuth, McpAuthReason) {
let token_status = if spec.is_remote() {
oauth::load_or_refresh_mcp_token(name).await
} else {
oauth::McpTokenStatus::NotAuthenticated
};
(
HttpAuth::from_token_status(&token_status, name),
McpAuthReason::from_token_status(&token_status),
)
}
pub(crate) async fn spawn_mcp_server( pub(crate) async fn spawn_mcp_server(
spec: &McpServer, spec: &McpServer,
log_path: Option<&Path>, log_path: Option<&Path>,
bearer_token: Option<String>, auth: HttpAuth,
) -> Result<Arc<ConnectedServer>> { ) -> Result<Arc<ConnectedServer>> {
match spec.transport_type { match spec.transport_type {
McpTransportType::Http => { McpTransportType::Http => {
let url = spec.url.as_deref().expect("validated: http spec has url"); let url = spec.url.as_deref().expect("validated: http spec has url");
let headers = merge_bearer_token(spec.headers.as_ref(), bearer_token); match auth {
spawn_http_mcp_server(url, headers.as_ref()).await HttpAuth::Managed { server, token: _ } => {
spawn_oauth_http_mcp_server(url, &server, spec.headers.as_ref()).await
}
HttpAuth::StaticOnly => spawn_http_mcp_server(url, spec.headers.as_ref()).await,
}
} }
McpTransportType::Sse => { McpTransportType::Sse => {
let url = spec.url.as_deref().expect("validated: sse spec has url"); let url = spec.url.as_deref().expect("validated: sse spec has url");
let bearer_token = match auth {
HttpAuth::Managed { server: _, token } => Some(token),
HttpAuth::StaticOnly => None,
};
let headers = merge_bearer_token(spec.headers.as_ref(), bearer_token); let headers = merge_bearer_token(spec.headers.as_ref(), bearer_token);
spawn_sse_mcp_server(url, headers.as_ref()).await spawn_sse_mcp_server(url, headers.as_ref()).await
} }
@@ -452,14 +571,65 @@ fn merge_bearer_token(
} }
(Some(h), Some(token)) => { (Some(h), Some(token)) => {
let mut m = h.clone(); let mut m = h.clone();
m.retain(|k, _| !k.eq_ignore_ascii_case("authorization"));
m.insert("Authorization".to_string(), format!("Bearer {token}")); m.insert("Authorization".to_string(), format!("Bearer {token}"));
Some(m) Some(m)
} }
} }
} }
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum McpAuthReason {
NotAuthenticated,
RefreshFailed,
TokenRejected,
}
impl McpAuthReason {
pub(crate) fn from_token_status(status: &oauth::McpTokenStatus) -> Self {
match status {
oauth::McpTokenStatus::Token(_) => Self::TokenRejected,
oauth::McpTokenStatus::NotAuthenticated => Self::NotAuthenticated,
oauth::McpTokenStatus::RefreshFailed => Self::RefreshFailed,
}
}
}
#[derive(Debug)]
pub(crate) struct McpAuthRequired {
pub server: String,
pub reason: McpAuthReason,
}
impl Display for McpAuthRequired {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let server = &self.server;
match self.reason {
McpAuthReason::NotAuthenticated => write!(
f,
"MCP server '{server}' requires OAuth authentication and was not started \
(no stored credentials). Run `.mcp auth {server}` (or `coyote --auth-mcp \
{server}`) to authenticate and attach it."
),
McpAuthReason::RefreshFailed => write!(
f,
"MCP server '{server}' was not started: stored OAuth token has expired and \
automatic refresh failed. Run `.mcp auth {server}` (or `coyote --auth-mcp \
{server}`) to re-authenticate and attach it."
),
McpAuthReason::TokenRejected => write!(
f,
"MCP server '{server}' was not started: the server rejected the stored OAuth \
token. Run `.mcp auth {server}` (or `coyote --auth-mcp {server}`) to \
re-authenticate and attach it."
),
}
}
}
pub(crate) fn is_auth_required_error(e: &Error) -> bool { pub(crate) fn is_auth_required_error(e: &Error) -> bool {
e.chain() e.downcast_ref::<McpAuthRequired>().is_some()
|| e.chain()
.any(|cause| cause.to_string().contains("Auth required")) .any(|cause| cause.to_string().contains("Auth required"))
} }
@@ -493,6 +663,66 @@ async fn spawn_http_mcp_server(
Ok(service) Ok(service)
} }
/// Builds the custom-header map for an OAuth-managed HTTP transport, dropping
/// any static `Authorization` entry case-insensitively: [`McpOAuthClient`]
/// owns that header, and a stale configured value must not collide with the
/// per-request token.
fn oauth_custom_headers(
headers: Option<&IndexMap<String, String>>,
) -> Result<HashMap<HeaderName, HeaderValue>> {
let mut custom = HashMap::new();
let Some(hdrs) = headers else {
return Ok(custom);
};
for (k, v) in hdrs {
if k.eq_ignore_ascii_case("authorization") {
continue;
}
let name = k
.parse::<HeaderName>()
.with_context(|| format!("Invalid header name: {k}"))?;
let value = v
.parse::<HeaderValue>()
.with_context(|| format!("Invalid header value for {k}"))?;
custom.insert(name, value);
}
Ok(custom)
}
async fn spawn_oauth_http_mcp_server(
url: &str,
server: &str,
headers: Option<&IndexMap<String, String>>,
) -> Result<Arc<ConnectedServer>> {
// Mirror rmcp's default_http_client, which `with_client` bypasses:
// idle pooling off avoids a documented TCP delayed-ACK stall, and
// redirects off keeps custom headers from being replayed to a redirect
// target.
let inner = reqwest::Client::builder()
.pool_max_idle_per_host(0)
.redirect(reqwest::redirect::Policy::none())
.build()
.context("Failed to build HTTP client for OAuth-managed MCP transport")?;
let client = McpOAuthClient::new(inner, server);
// `auth_header` stays None so the wrapper injects a fresh token per
// request; `reinit_on_expired_session` defaults to true in rmcp 3.1.2
// but is pinned explicitly because transparent session re-init is
// load-bearing for long-lived sessions.
let config = StreamableHttpClientTransportConfig::with_uri(url)
.custom_headers(oauth_custom_headers(headers)?)
.reinit_on_expired_session(true);
let transport = StreamableHttpClientTransport::with_client(client, config);
let service = Arc::new(
().serve(transport)
.await
.with_context(|| format!("Failed to connect to HTTP MCP server: {url}"))?,
);
Ok(service)
}
async fn spawn_sse_mcp_server( async fn spawn_sse_mcp_server(
url: &str, url: &str,
headers: Option<&IndexMap<String, String>>, headers: Option<&IndexMap<String, String>>,
@@ -990,7 +1220,7 @@ mod tests {
let registry = McpRegistry::default(); let registry = McpRegistry::default();
assert!(registry.is_empty()); assert!(registry.is_empty());
assert!(registry.list_started_servers().is_empty()); assert!(registry.server_features().is_empty());
assert!(registry.mcp_config().is_none()); assert!(registry.mcp_config().is_none());
assert!(registry.log_path().is_none()); assert!(registry.log_path().is_none());
} }
@@ -1014,6 +1244,51 @@ mod tests {
assert_eq!(MCP_INVOKE_META_FUNCTION_NAME_PREFIX, "mcp_invoke"); assert_eq!(MCP_INVOKE_META_FUNCTION_NAME_PREFIX, "mcp_invoke");
assert_eq!(MCP_SEARCH_META_FUNCTION_NAME_PREFIX, "mcp_search"); assert_eq!(MCP_SEARCH_META_FUNCTION_NAME_PREFIX, "mcp_search");
assert_eq!(MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX, "mcp_describe"); assert_eq!(MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX, "mcp_describe");
assert_eq!(MCP_READ_META_FUNCTION_NAME_PREFIX, "mcp_read");
assert_eq!(MCP_PROMPT_META_FUNCTION_NAME_PREFIX, "mcp_prompt");
}
#[test]
fn is_mcp_meta_function_classifies_names() {
assert!(is_mcp_meta_function("mcp_invoke_github"));
assert!(is_mcp_meta_function("mcp_search_github"));
assert!(is_mcp_meta_function("mcp_describe_github"));
assert!(is_mcp_meta_function("mcp_read_github"));
assert!(is_mcp_meta_function("mcp_prompt_github"));
assert!(!is_mcp_meta_function("mcp_gateway_tool"));
assert!(!is_mcp_meta_function("fs_read"));
assert!(!is_mcp_meta_function(""));
assert!(!is_mcp_meta_function("mcp_"));
}
#[test]
fn meta_function_prefixes_are_not_prefixes_of_each_other() {
for (i, a) in MCP_META_FUNCTION_PREFIXES.iter().enumerate() {
for (j, b) in MCP_META_FUNCTION_PREFIXES.iter().enumerate() {
if i != j {
assert!(!b.starts_with(a), "{a} is a prefix of {b}");
}
}
}
}
#[test]
fn is_mcp_meta_function_preserves_lax_prefix_matching() {
assert!(is_mcp_meta_function("mcp_invoker_x"));
}
#[test]
fn mcp_meta_function_names_returns_all_prefixes_in_order() {
assert_eq!(
mcp_meta_function_names("github"),
vec![
"mcp_invoke_github",
"mcp_search_github",
"mcp_describe_github",
"mcp_read_github",
"mcp_prompt_github",
]
);
} }
#[test] #[test]
@@ -1051,6 +1326,92 @@ mod tests {
assert_eq!(result["X-Custom"], "keep"); assert_eq!(result["X-Custom"], "keep");
} }
#[test]
fn merge_bearer_token_replaces_authorization_case_insensitively() {
let mut h = IndexMap::new();
h.insert("authorization".to_string(), "Bearer stale-1".to_string());
h.insert("AUTHORIZATION".to_string(), "Bearer stale-2".to_string());
h.insert("X-Custom".to_string(), "keep".to_string());
let result = merge_bearer_token(Some(&h), Some("newtoken".to_string())).unwrap();
assert_eq!(result.len(), 2);
assert_eq!(result["Authorization"], "Bearer newtoken");
assert_eq!(result["X-Custom"], "keep");
assert!(!result.contains_key("authorization"));
assert!(!result.contains_key("AUTHORIZATION"));
}
#[test]
fn http_auth_from_token_status_maps_token_to_managed() {
assert!(matches!(
HttpAuth::from_token_status(&oauth::McpTokenStatus::Token("tok".into()), "srv"),
HttpAuth::Managed { server, token } if server == "srv" && token == "tok"
));
assert!(matches!(
HttpAuth::from_token_status(&oauth::McpTokenStatus::NotAuthenticated, "srv"),
HttpAuth::StaticOnly
));
assert!(matches!(
HttpAuth::from_token_status(&oauth::McpTokenStatus::RefreshFailed, "srv"),
HttpAuth::StaticOnly
));
}
#[test]
fn http_auth_debug_redacts_token() {
let auth = HttpAuth::Managed {
server: "srv".into(),
token: "live-secret".into(),
};
let debug = format!("{auth:?}");
assert!(debug.contains("srv"));
assert!(debug.contains("<redacted>"));
assert!(!debug.contains("live-secret"));
}
#[test]
fn oauth_custom_headers_strips_authorization_case_insensitively() {
let mut h = IndexMap::new();
h.insert("Authorization".to_string(), "Bearer stale-1".to_string());
h.insert("authorization".to_string(), "Bearer stale-2".to_string());
h.insert("AUTHORIZATION".to_string(), "Bearer stale-3".to_string());
h.insert("X-Custom".to_string(), "keep".to_string());
let custom = oauth_custom_headers(Some(&h)).unwrap();
assert_eq!(custom.len(), 1);
assert_eq!(custom[&HeaderName::from_static("x-custom")], "keep");
}
#[test]
fn oauth_custom_headers_none_is_empty() {
assert!(oauth_custom_headers(None).unwrap().is_empty());
}
#[test]
fn oauth_custom_headers_rejects_invalid_header_name() {
let mut h = IndexMap::new();
h.insert("bad header".to_string(), "v".to_string());
assert!(oauth_custom_headers(Some(&h)).is_err());
}
#[test]
fn oauth_custom_headers_keeps_non_authorization_headers() {
let mut h = IndexMap::new();
h.insert("X-Api-Key".to_string(), "k".to_string());
h.insert("X-Trace".to_string(), "t".to_string());
let custom = oauth_custom_headers(Some(&h)).unwrap();
assert_eq!(custom.len(), 2);
assert_eq!(custom[&HeaderName::from_static("x-api-key")], "k");
assert_eq!(custom[&HeaderName::from_static("x-trace")], "t");
}
#[test] #[test]
fn is_auth_required_error_matches_rmcp_message() { fn is_auth_required_error_matches_rmcp_message() {
let e = anyhow!("Auth required, when send initialize request"); let e = anyhow!("Auth required, when send initialize request");
@@ -1074,4 +1435,48 @@ mod tests {
assert!(is_auth_required_error(&e)); assert!(is_auth_required_error(&e));
} }
#[test]
fn auth_reason_maps_token_status() {
assert_eq!(
McpAuthReason::from_token_status(&oauth::McpTokenStatus::Token("tok".into())),
McpAuthReason::TokenRejected
);
assert_eq!(
McpAuthReason::from_token_status(&oauth::McpTokenStatus::NotAuthenticated),
McpAuthReason::NotAuthenticated
);
assert_eq!(
McpAuthReason::from_token_status(&oauth::McpTokenStatus::RefreshFailed),
McpAuthReason::RefreshFailed
);
}
#[test]
fn mcp_auth_required_context_downcasts_with_reason() {
let e = anyhow!("Auth required, when send initialize request").context(McpAuthRequired {
server: "github".into(),
reason: McpAuthReason::RefreshFailed,
});
assert!(is_auth_required_error(&e));
let ctx = e.downcast_ref::<McpAuthRequired>().unwrap();
assert_eq!(ctx.server, "github");
assert_eq!(ctx.reason, McpAuthReason::RefreshFailed);
}
#[test]
fn mcp_auth_required_display_is_reason_specific() {
let msg = |reason| {
McpAuthRequired {
server: "github".into(),
reason,
}
.to_string()
};
assert!(msg(McpAuthReason::NotAuthenticated).contains("no stored credentials"));
assert!(msg(McpAuthReason::RefreshFailed).contains("expired and automatic refresh failed"));
assert!(msg(McpAuthReason::TokenRejected).contains("rejected the stored OAuth token"));
}
} }
+532 -33
View File
@@ -1,15 +1,26 @@
use crate::client::oauth::{OAuthProvider, TokenRequestFormat, load_oauth_tokens, run_oauth_flow}; use crate::client::oauth::{
OAuthProvider, OAuthTokens, TokenRequestFormat, load_oauth_tokens, refresh_oauth_token,
run_oauth_flow, token_response_keys,
};
use crate::config::paths; use crate::config::paths;
use anyhow::{Context, Result, anyhow}; use anyhow::{Context, Result, anyhow};
use chrono::Utc; use chrono::Utc;
use inquire::Text; use inquire::Text;
use log::warn; use log::{debug, warn};
use reqwest::Client; use reqwest::Client;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::fmt;
use std::fs; use std::fs;
use std::net::TcpListener; use std::net::TcpListener;
use std::sync::{Arc, OnceLock};
use std::time::{Duration, Instant};
use tokio::sync;
use url::Url; use url::Url;
const REFRESH_HTTP_TIMEOUT: Duration = Duration::from_secs(10);
const REFRESH_FAILURE_BACKOFF: Duration = Duration::from_secs(60);
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
struct ProtectedResourceMetadata { struct ProtectedResourceMetadata {
#[serde(default)] #[serde(default)]
@@ -34,6 +45,10 @@ struct McpRegistration {
client_id: String, client_id: String,
#[serde(default)] #[serde(default)]
redirect_uri: Option<String>, redirect_uri: Option<String>,
#[serde(default)]
token_url: Option<String>,
#[serde(default)]
resource: Option<String>,
} }
struct DiscoveredOAuth { struct DiscoveredOAuth {
@@ -124,8 +139,19 @@ pub async fn run_mcp_oauth_flow(
None None
}; };
let (client_id, redirect_uri) = if let Some(reused) = cached_reuse { let (client_id, redirect_uri) = if let Some((client_id, redirect_uri)) = cached_reuse {
reused // Re-save so registrations cached before token_url/resource were
// persisted gain them, enabling token refresh next time.
if let Err(e) = save_registration(
server_name,
&client_id,
&redirect_uri,
&metadata.token_endpoint,
&resource,
) {
debug!("Failed to update cached MCP registration for '{server_name}': {e}");
}
(client_id, redirect_uri)
} else { } else {
let bind_addr = format!("127.0.0.1:{}", callback_port.unwrap_or(0)); let bind_addr = format!("127.0.0.1:{}", callback_port.unwrap_or(0));
let listener = TcpListener::bind(&bind_addr)?; let listener = TcpListener::bind(&bind_addr)?;
@@ -137,10 +163,7 @@ pub async fn run_mcp_oauth_flow(
id.to_string() id.to_string()
} else if let Some(reg_endpoint) = &metadata.registration_endpoint { } else if let Some(reg_endpoint) = &metadata.registration_endpoint {
match register_client(reg_endpoint, &redirect_uri).await { match register_client(reg_endpoint, &redirect_uri).await {
Ok(id) => { Ok(id) => id,
let _ = save_registration(server_name, &id, &redirect_uri);
id
}
Err(e) => { Err(e) => {
warn!("Dynamic client registration failed: {e}. Falling back to manual entry."); warn!("Dynamic client registration failed: {e}. Falling back to manual entry.");
Text::new("Enter the OAuth client ID for this MCP server:") Text::new("Enter the OAuth client ID for this MCP server:")
@@ -153,6 +176,18 @@ pub async fn run_mcp_oauth_flow(
.prompt() .prompt()
.context("Failed to read client ID")? .context("Failed to read client ID")?
}; };
// Persist regardless of how the client_id was obtained (DCR, config,
// or manual entry) so refresh_mcp_token can run the refresh_token
// grant later without interactive re-auth.
if let Err(e) = save_registration(
server_name,
&client_id,
&redirect_uri,
&metadata.token_endpoint,
&resource,
) {
debug!("Failed to cache MCP registration for '{server_name}': {e}");
}
(client_id, redirect_uri) (client_id, redirect_uri)
}; };
@@ -168,12 +203,164 @@ pub async fn run_mcp_oauth_flow(
run_oauth_flow(&provider, &mcp_token_key(server_name)).await run_oauth_flow(&provider, &mcp_token_key(server_name)).await
} }
pub fn load_valid_mcp_token(server_name: &str) -> Option<String> { #[derive(PartialEq, Eq)]
let tokens = load_oauth_tokens(&mcp_token_key(server_name))?; pub enum McpTokenStatus {
if Utc::now().timestamp() < tokens.expires_at { Token(String),
Some(tokens.access_token) NotAuthenticated,
} else { RefreshFailed,
None }
impl fmt::Debug for McpTokenStatus {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Token(_) => f.write_str("Token(<redacted>)"),
Self::NotAuthenticated => f.write_str("NotAuthenticated"),
Self::RefreshFailed => f.write_str("RefreshFailed"),
}
}
}
impl McpTokenStatus {
pub fn into_token(self) -> Option<String> {
match self {
Self::Token(token) => Some(token),
Self::NotAuthenticated | Self::RefreshFailed => None,
}
}
}
pub async fn load_or_refresh_mcp_token(server_name: &str) -> McpTokenStatus {
load_or_refresh_inner(server_name, None).await
}
/// Re-acquires a token after the server rejected the current one mid-session
/// (HTTP 401). The rejection proves the stored token is bad regardless of its
/// expiry timestamp, so the unexpired fast-paths only short-circuit when the
/// stored token DIFFERS from `rejected_token` (a concurrent caller genuinely
/// refreshed while we waited); an unexpired copy of the rejected token is
/// refreshed anyway. The failure backoff and per-server single-flight lock
/// still apply.
pub async fn force_refresh_mcp_token(server_name: &str, rejected_token: &str) -> Option<String> {
load_or_refresh_inner(server_name, Some(rejected_token))
.await
.into_token()
}
async fn load_or_refresh_inner(server_name: &str, rejected_token: Option<&str>) -> McpTokenStatus {
let key = mcp_token_key(server_name);
let Some(tokens) = load_oauth_tokens(&key) else {
return McpTokenStatus::NotAuthenticated;
};
if rejected_token.is_none() && Utc::now().timestamp() < tokens.expires_at {
return McpTokenStatus::Token(tokens.access_token);
}
if in_refresh_failure_backoff(server_name) {
debug!("Skipping token refresh for MCP server '{server_name}': recent attempt failed");
return McpTokenStatus::RefreshFailed;
}
let lock = refresh_lock(server_name);
let _guard = lock.lock().await;
// A concurrent caller may have refreshed while we waited for the lock. An
// unexpired token is only trusted if it differs from the rejected one:
// the server already proved that exact token bad.
let Some(tokens) = load_oauth_tokens(&key) else {
return McpTokenStatus::NotAuthenticated;
};
if Utc::now().timestamp() < tokens.expires_at
&& rejected_token.is_none_or(|rejected| rejected != tokens.access_token)
{
return McpTokenStatus::Token(tokens.access_token);
}
if in_refresh_failure_backoff(server_name) {
debug!("Skipping token refresh for MCP server '{server_name}': recent attempt failed");
return McpTokenStatus::RefreshFailed;
}
match refresh_mcp_token(server_name, &key, &tokens).await {
Ok(access_token) => McpTokenStatus::Token(access_token),
Err(e) => {
note_refresh_failure(server_name);
warn!(
"Failed to refresh OAuth token for MCP server '{server_name}'. \
Run `.mcp auth {server_name}` to re-authenticate."
);
debug!(
"Token refresh error for MCP server '{server_name}': {}",
redact_refresh_error(&e)
);
McpTokenStatus::RefreshFailed
}
}
}
async fn refresh_mcp_token(server_name: &str, key: &str, tokens: &OAuthTokens) -> Result<String> {
if tokens.refresh_token.is_none() {
return Err(anyhow!("no refresh token stored"));
}
let reg =
load_registration(server_name).ok_or_else(|| anyhow!("no cached client registration"))?;
let token_url = reg.token_url.ok_or_else(|| {
anyhow!("cached registration has no token URL (saved by an older version)")
})?;
let resource = reg.resource.ok_or_else(|| {
anyhow!("cached registration has no resource (saved by an older version)")
})?;
let provider = McpOAuthProvider {
client_id: reg.client_id,
authorize_url: String::new(),
token_url,
scopes: String::new(),
fixed_redirect: String::new(),
resource,
};
let client = Client::builder().timeout(REFRESH_HTTP_TIMEOUT).build()?;
let refreshed = refresh_oauth_token(&client, &provider, key, tokens).await?;
Ok(refreshed.access_token)
}
fn refresh_lock(server_name: &str) -> Arc<sync::Mutex<()>> {
static LOCKS: OnceLock<parking_lot::Mutex<HashMap<String, Arc<sync::Mutex<()>>>>> =
OnceLock::new();
LOCKS
.get_or_init(Default::default)
.lock()
.entry(server_name.to_string())
.or_default()
.clone()
}
fn refresh_failures() -> &'static parking_lot::Mutex<HashMap<String, Instant>> {
static FAILURES: OnceLock<parking_lot::Mutex<HashMap<String, Instant>>> = OnceLock::new();
FAILURES.get_or_init(Default::default)
}
fn note_refresh_failure(server_name: &str) {
refresh_failures()
.lock()
.insert(server_name.to_string(), Instant::now());
}
fn in_refresh_failure_backoff(server_name: &str) -> bool {
refresh_failures()
.lock()
.get(server_name)
.is_some_and(|failed_at| failed_at.elapsed() < REFRESH_FAILURE_BACKOFF)
}
/// Refresh errors may embed the token endpoint's JSON response, which can
/// contain live tokens; strip everything from the first `{` before logging.
fn redact_refresh_error(e: &anyhow::Error) -> String {
let msg = e.to_string();
match msg.find('{') {
Some(idx) => format!("{}<response body redacted>", &msg[..idx]),
None => msg,
} }
} }
@@ -187,7 +374,13 @@ fn load_registration(server_name: &str) -> Option<McpRegistration> {
serde_json::from_str(&content).ok() serde_json::from_str(&content).ok()
} }
fn save_registration(server_name: &str, client_id: &str, redirect_uri: &str) -> Result<()> { fn save_registration(
server_name: &str,
client_id: &str,
redirect_uri: &str,
token_url: &str,
resource: &str,
) -> Result<()> {
let dir = paths::oauth_tokens_dir(); let dir = paths::oauth_tokens_dir();
fs::create_dir_all(&dir)?; fs::create_dir_all(&dir)?;
@@ -195,6 +388,8 @@ fn save_registration(server_name: &str, client_id: &str, redirect_uri: &str) ->
let reg = McpRegistration { let reg = McpRegistration {
client_id: client_id.to_string(), client_id: client_id.to_string(),
redirect_uri: Some(redirect_uri.to_string()), redirect_uri: Some(redirect_uri.to_string()),
token_url: Some(token_url.to_string()),
resource: Some(resource.to_string()),
}; };
fs::write(path, serde_json::to_string_pretty(&reg)?)?; fs::write(path, serde_json::to_string_pretty(&reg)?)?;
@@ -244,7 +439,12 @@ async fn register_client(endpoint: &str, redirect_uri: &str) -> Result<String> {
response["client_id"] response["client_id"]
.as_str() .as_str()
.ok_or_else(|| anyhow!("Missing client_id in registration response: {response}")) .ok_or_else(|| {
anyhow!(
"Missing client_id in registration response (keys: {})",
token_response_keys(&response)
)
})
.map(|s| s.to_string()) .map(|s| s.to_string())
} }
@@ -421,16 +621,34 @@ fn extract_base_url(url: &str) -> Result<String> {
} }
#[cfg(test)] #[cfg(test)]
mod tests { pub(crate) mod test_support {
use super::*;
use crate::utils::get_env_name; use crate::utils::get_env_name;
use serial_test::serial;
use std::{ use std::{
env, fs, env,
ffi::OsString,
fs,
path::PathBuf,
time::{self, SystemTime}, time::{self, SystemTime},
}; };
fn with_temp_cache<F: FnOnce()>(f: F) { pub(crate) fn with_temp_cache<F: FnOnce()>(f: F) {
struct Restore {
key: String,
prev: Option<OsString>,
root: PathBuf,
}
impl Drop for Restore {
fn drop(&mut self) {
unsafe {
match self.prev.take() {
Some(v) => env::set_var(&self.key, v),
None => env::remove_var(&self.key),
}
}
let _ = fs::remove_dir_all(&self.root);
}
}
let unique = SystemTime::now() let unique = SystemTime::now()
.duration_since(time::UNIX_EPOCH) .duration_since(time::UNIX_EPOCH)
.unwrap() .unwrap()
@@ -442,15 +660,21 @@ mod tests {
unsafe { unsafe {
env::set_var(&env_key, &root); env::set_var(&env_key, &root);
} }
let _restore = Restore {
key: env_key,
prev,
root,
};
f(); f();
unsafe {
match prev {
Some(v) => env::set_var(&env_key, v),
None => env::remove_var(&env_key),
}
}
let _ = fs::remove_dir_all(&root);
} }
}
#[cfg(test)]
mod tests {
use super::test_support::with_temp_cache;
use super::*;
use serial_test::serial;
use std::fs;
#[test] #[test]
fn extract_base_url_strips_path_and_query() { fn extract_base_url_strips_path_and_query() {
@@ -685,12 +909,19 @@ mod tests {
"notion", "notion",
"client-xyz-123", "client-xyz-123",
"http://127.0.0.1:49152/callback", "http://127.0.0.1:49152/callback",
"https://as.example/token",
"https://mcp.example/mcp",
) )
.unwrap(); .unwrap();
let loaded = load_registration("notion"); let loaded = load_registration("notion").unwrap();
assert_eq!(loaded.unwrap().client_id, "client-xyz-123"); assert_eq!(loaded.client_id, "client-xyz-123");
assert_eq!(
loaded.token_url.as_deref(),
Some("https://as.example/token")
);
assert_eq!(loaded.resource.as_deref(), Some("https://mcp.example/mcp"));
}); });
} }
@@ -708,8 +939,22 @@ mod tests {
#[serial] #[serial]
fn registration_second_save_overwrites_first() { fn registration_second_save_overwrites_first() {
with_temp_cache(|| { with_temp_cache(|| {
save_registration("github", "first-id", "http://127.0.0.1:49152/callback").unwrap(); save_registration(
save_registration("github", "second-id", "http://127.0.0.1:49153/callback").unwrap(); "github",
"first-id",
"http://127.0.0.1:49152/callback",
"https://as.example/token",
"https://mcp.example/mcp",
)
.unwrap();
save_registration(
"github",
"second-id",
"http://127.0.0.1:49153/callback",
"https://as.example/token",
"https://mcp.example/mcp",
)
.unwrap();
let loaded = load_registration("github").unwrap(); let loaded = load_registration("github").unwrap();
@@ -737,6 +982,8 @@ mod tests {
assert_eq!(loaded.client_id, "legacy-id"); assert_eq!(loaded.client_id, "legacy-id");
assert_eq!(loaded.redirect_uri, None); assert_eq!(loaded.redirect_uri, None);
assert_eq!(loaded.token_url, None);
assert_eq!(loaded.resource, None);
}); });
} }
@@ -744,7 +991,14 @@ mod tests {
#[serial] #[serial]
fn save_registration_persists_redirect_uri() { fn save_registration_persists_redirect_uri() {
with_temp_cache(|| { with_temp_cache(|| {
save_registration("aws", "client-abc", "http://127.0.0.1:49152/callback").unwrap(); save_registration(
"aws",
"client-abc",
"http://127.0.0.1:49152/callback",
"https://as.example/token",
"https://mcp.example/mcp",
)
.unwrap();
let loaded = load_registration("aws").unwrap(); let loaded = load_registration("aws").unwrap();
@@ -756,6 +1010,251 @@ mod tests {
}); });
} }
#[test]
fn mcp_registration_deserializes_without_new_fields_and_roundtrips() {
let old: McpRegistration = serde_json::from_str(r#"{"client_id":"legacy-id"}"#).unwrap();
assert_eq!(old.client_id, "legacy-id");
assert_eq!(old.token_url, None);
assert_eq!(old.resource, None);
let full = McpRegistration {
client_id: "client-abc".into(),
redirect_uri: Some("http://127.0.0.1:49152/callback".into()),
token_url: Some("https://as.example/token".into()),
resource: Some("https://mcp.example/mcp".into()),
};
let json = serde_json::to_string(&full).unwrap();
let back: McpRegistration = serde_json::from_str(&json).unwrap();
assert_eq!(back.token_url.as_deref(), Some("https://as.example/token"));
assert_eq!(back.resource.as_deref(), Some("https://mcp.example/mcp"));
}
#[test]
#[serial]
fn expired_token_with_old_format_registration_reports_refresh_failed() {
with_temp_cache(|| {
let dir = paths::oauth_tokens_dir();
fs::create_dir_all(&dir).unwrap();
fs::write(
paths::token_file("mcp_legacyref"),
r#"{"access_token":"stale","refresh_token":"refresh-abc","expires_at":0}"#,
)
.unwrap();
fs::write(
dir.join("mcp_legacyref_registration.json"),
r#"{"client_id":"legacy-id"}"#,
)
.unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = rt.block_on(load_or_refresh_mcp_token("legacyref"));
assert_eq!(status, McpTokenStatus::RefreshFailed);
});
}
#[test]
#[serial]
fn missing_token_file_reports_not_authenticated() {
with_temp_cache(|| {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let status = rt.block_on(load_or_refresh_mcp_token("never-authed"));
assert_eq!(status, McpTokenStatus::NotAuthenticated);
});
}
#[test]
#[serial]
fn force_refresh_returns_concurrently_refreshed_token() {
with_temp_cache(|| {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file("mcp_force-fresh"),
r#"{"access_token":"fresh-tok","refresh_token":"r","expires_at":9999999999}"#,
)
.unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let token = rt.block_on(force_refresh_mcp_token("force-fresh", "rejected-tok"));
assert_eq!(token.as_deref(), Some("fresh-tok"));
});
}
#[test]
#[serial]
fn force_refresh_unexpired_rejected_token_attempts_real_refresh() {
with_temp_cache(|| {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file("mcp_force-rejected"),
r#"{"access_token":"same-tok","refresh_token":"r","expires_at":9999999999}"#,
)
.unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let token = rt.block_on(force_refresh_mcp_token("force-rejected", "same-tok"));
assert_eq!(token, None);
});
}
#[test]
#[serial]
fn force_refresh_missing_token_file_returns_none() {
with_temp_cache(|| {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let token = rt.block_on(force_refresh_mcp_token(
"force-never-authed",
"rejected-tok",
));
assert_eq!(token, None);
});
}
#[test]
#[serial]
fn force_refresh_failed_refresh_returns_none() {
with_temp_cache(|| {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file("mcp_force-fail"),
r#"{"access_token":"stale","refresh_token":"r","expires_at":0}"#,
)
.unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let token = rt.block_on(force_refresh_mcp_token("force-fail", "stale"));
assert_eq!(token, None);
});
}
#[test]
#[serial]
fn force_refresh_concurrent_callers_complete_without_deadlock() {
with_temp_cache(|| {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file("mcp_force-concurrent"),
r#"{"access_token":"same-tok","refresh_token":"r","expires_at":9999999999}"#,
)
.unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let (a, b) = rt.block_on(async {
tokio::join!(
force_refresh_mcp_token("force-concurrent", "same-tok"),
force_refresh_mcp_token("force-concurrent", "same-tok"),
)
});
assert_eq!(a, None);
assert_eq!(b, None);
});
}
#[test]
#[serial]
fn force_refresh_respects_failure_backoff() {
with_temp_cache(|| {
fs::create_dir_all(paths::oauth_tokens_dir()).unwrap();
fs::write(
paths::token_file("mcp_force-backoff"),
r#"{"access_token":"same-tok","refresh_token":"r","expires_at":9999999999}"#,
)
.unwrap();
note_refresh_failure("force-backoff");
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let token = rt.block_on(force_refresh_mcp_token("force-backoff", "same-tok"));
assert_eq!(token, None);
});
}
#[test]
fn token_status_debug_redacts_token() {
assert_eq!(
format!("{:?}", McpTokenStatus::Token("live-secret".into())),
"Token(<redacted>)"
);
assert_eq!(
format!("{:?}", McpTokenStatus::NotAuthenticated),
"NotAuthenticated"
);
assert_eq!(
format!("{:?}", McpTokenStatus::RefreshFailed),
"RefreshFailed"
);
}
#[test]
fn token_status_into_token_extracts_only_token_variant() {
assert_eq!(
McpTokenStatus::Token("tok".into()).into_token(),
Some("tok".to_string())
);
assert_eq!(McpTokenStatus::NotAuthenticated.into_token(), None);
assert_eq!(McpTokenStatus::RefreshFailed.into_token(), None);
}
#[test]
fn refresh_failure_backoff_memoizes_per_server() {
assert!(!in_refresh_failure_backoff("backoff-test-server"));
note_refresh_failure("backoff-test-server");
assert!(in_refresh_failure_backoff("backoff-test-server"));
assert!(!in_refresh_failure_backoff("backoff-other-server"));
}
#[test]
fn redact_refresh_error_strips_response_body() {
let with_body = anyhow!(
"Missing access_token in refresh response: {}",
r#"{"access_token":"live-secret"}"#
);
let without_body = anyhow!("no refresh token stored");
assert_eq!(
redact_refresh_error(&with_body),
"Missing access_token in refresh response: <response body redacted>"
);
assert_eq!(
redact_refresh_error(&without_body),
"no refresh token stored"
);
}
#[test] #[test]
fn cached_redirect_port_matches() { fn cached_redirect_port_matches() {
let port = cached_redirect_port("http://127.0.0.1:49152/callback", "127.0.0.1", None); let port = cached_redirect_port("http://127.0.0.1:49152/callback", "127.0.0.1", None);
+896
View File
@@ -0,0 +1,896 @@
//! Content policy for MCP resource and tool content: UTF-8-boundary-safe text
//! paging, grep-style pattern filtering, and spill-to-disk for binary blobs.
use crate::config::paths;
use base64::engine::general_purpose::STANDARD;
use base64::read::DecoderReader;
use fancy_regex::Regex;
use serde::Serialize;
use sha2::{Digest, Sha256};
use std::error::Error;
use std::fs::{self, OpenOptions};
use std::io::{ErrorKind, Read, Write};
#[cfg(unix)]
use std::os::unix::fs::OpenOptionsExt;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::SystemTime;
use std::{fmt, io};
/// Default page size when the caller does not specify `max_bytes`.
pub const DEFAULT_TEXT_MAX_BYTES: usize = 51_200;
/// Hard upper bound on a single text slice regardless of requested `max_bytes`.
pub const TEXT_MAX_BYTES_CLAMP: usize = 204_800;
/// Maximum decoded size of a base64 blob before rendering is refused.
pub const BLOB_DECODE_CEILING_BYTES: usize = 50 * 1024 * 1024;
/// Total size bound for the spill tree; oldest files are evicted beyond it.
pub const SPILL_DIR_MAX_BYTES: u64 = 512 * 1024 * 1024;
/// Byte bound on server-supplied metadata strings (uri, mime type) copied into output.
pub const METADATA_MAX_BYTES: usize = 4096;
const PATTERN_CONTEXT_LINES: usize = 2;
const HUNK_SEPARATOR: &str = "--";
const MIME_EXTENSIONS: &[(&str, &str)] = &[
("application/gzip", "gz"),
("application/json", "json"),
("application/pdf", "pdf"),
("application/zip", "zip"),
("audio/mpeg", "mp3"),
("image/gif", "gif"),
("image/jpeg", "jpg"),
("image/png", "png"),
("image/webp", "webp"),
("text/csv", "csv"),
("video/mp4", "mp4"),
];
#[derive(Debug)]
pub enum RenderError {
InvalidPattern { pattern: String, error: String },
DecodedSizeExceeded,
InvalidBase64(String),
Io(io::Error),
}
impl fmt::Display for RenderError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::InvalidPattern { pattern, error } => write!(
f,
"Invalid filter pattern '{pattern}': {error}. Provide a valid regex; \
lines matching it are returned with {PATTERN_CONTEXT_LINES} lines of context."
),
Self::DecodedSizeExceeded => write!(
f,
"Decoded blob exceeds BLOB_DECODE_CEILING_BYTES ({} MiB); refusing to render it",
BLOB_DECODE_CEILING_BYTES / (1024 * 1024)
),
Self::InvalidBase64(error) => write!(f, "Invalid base64 in blob content: {error}"),
Self::Io(error) => write!(f, "Failed to spill blob to disk: {error}"),
}
}
}
impl Error for RenderError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
Self::Io(error) => Some(error),
_ => None,
}
}
}
impl From<io::Error> for RenderError {
fn from(error: io::Error) -> Self {
Self::Io(error)
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct RenderedText {
pub text: String,
pub truncated: bool,
pub total_bytes: usize,
pub next_offset: Option<usize>,
}
#[derive(Debug)]
pub enum RenderedBlob {
Text(String),
Spilled(SpillMetadata),
}
#[derive(Debug, Serialize)]
pub struct SpillMetadata {
pub spilled: bool,
pub path: PathBuf,
pub mime_type: Option<String>,
pub sniffed: bool,
pub size_bytes: u64,
pub sha256: String,
}
/// Pages `text` with UTF-8-boundary-safe slicing. When `pattern` is set, the
/// text is first reduced to matching lines plus context (grep-style, with
/// 1-based line-number prefixes), and all offset/size math operates on that
/// filtered stream.
pub fn render_text(
text: &str,
pattern: Option<&str>,
offset: usize,
max_bytes: Option<usize>,
) -> Result<RenderedText, RenderError> {
let filtered = match pattern {
Some(pattern) => Some(filter_lines(text, pattern)?),
None => None,
};
let stream = filtered.as_deref().unwrap_or(text);
let max_bytes = max_bytes
.unwrap_or(DEFAULT_TEXT_MAX_BYTES)
.min(TEXT_MAX_BYTES_CLAMP);
let total_bytes = stream.len();
let mut start = offset.min(total_bytes);
while !stream.is_char_boundary(start) {
start += 1;
}
let mut end = start.saturating_add(max_bytes).min(total_bytes);
while !stream.is_char_boundary(end) {
end -= 1;
}
// A max_bytes smaller than one codepoint would produce an empty page with
// next_offset == offset, stalling paging; always advance by at least one.
if end == start && start < total_bytes {
end += 1;
while !stream.is_char_boundary(end) {
end += 1;
}
}
let truncated = end < total_bytes;
Ok(RenderedText {
text: stream[start..end].to_string(),
truncated,
total_bytes,
next_offset: truncated.then_some(end),
})
}
/// Decodes a base64 blob, returning it as text when it is valid UTF-8 and
/// spilling it under `cache_dir()/mcp-resources/<server>/` otherwise.
pub fn render_blob(
b64: &str,
claimed_mime: Option<&str>,
server: &str,
) -> Result<RenderedBlob, RenderError> {
let spill_base = paths::cache_dir().join("mcp-resources");
render_blob_at(b64, claimed_mime, server, &spill_base)
}
pub fn render_blob_at(
b64: &str,
claimed_mime: Option<&str>,
server: &str,
spill_base: &Path,
) -> Result<RenderedBlob, RenderError> {
let decoded = decode_base64_bounded(b64)?;
let decoded = match String::from_utf8(decoded) {
Ok(text) => return Ok(RenderedBlob::Text(text)),
Err(error) => error.into_bytes(),
};
let sha256 = format!("{:x}", Sha256::digest(&decoded));
let dir = spill_base.join(sanitize_server(server));
fs::create_dir_all(&dir)?;
let path = dir.join(format!("{sha256}.{}", extension_for_mime(claimed_mime)));
// Writes land in a temp file and are renamed into place, so a visible
// file at the final path is always complete and the dedup check below is
// race-safe across processes (same sha means same content).
if !path.exists() {
static TEMP_COUNTER: AtomicUsize = AtomicUsize::new(0);
let temp = dir.join(format!(
"{sha256}.tmp-{}-{}",
std::process::id(),
TEMP_COUNTER.fetch_add(1, Ordering::Relaxed)
));
let mut options = OpenOptions::new();
options.write(true).create_new(true);
#[cfg(unix)]
options.mode(0o600);
let written = options
.open(&temp)
.and_then(|mut file| file.write_all(&decoded))
.and_then(|()| fs::rename(&temp, &path));
if let Err(error) = written {
let _ = fs::remove_file(&temp);
return Err(RenderError::Io(error));
}
}
enforce_spill_bound(spill_base, SPILL_DIR_MAX_BYTES, &path);
Ok(RenderedBlob::Spilled(SpillMetadata {
spilled: true,
path,
mime_type: claimed_mime.map(str::to_string),
sniffed: false,
size_bytes: decoded.len() as u64,
sha256,
}))
}
/// Truncates `text` to at most `max_bytes`, rounding the cut point back to a
/// UTF-8 character boundary.
pub fn truncate_utf8(text: &str, max_bytes: usize) -> &str {
if text.len() <= max_bytes {
return text;
}
let mut end = max_bytes;
while !text.is_char_boundary(end) {
end -= 1;
}
&text[..end]
}
/// Bounds a server-supplied metadata string to [`METADATA_MAX_BYTES`],
/// appending a marker citing the constant when the input is truncated.
pub fn clamp_metadata(text: &str) -> String {
if text.len() <= METADATA_MAX_BYTES {
return text.to_string();
}
let clamped = truncate_utf8(text, METADATA_MAX_BYTES);
format!("{clamped} [truncated: exceeds METADATA_MAX_BYTES ({METADATA_MAX_BYTES} bytes)]")
}
fn filter_lines(text: &str, pattern: &str) -> Result<String, RenderError> {
let regex = Regex::new(pattern).map_err(|error| RenderError::InvalidPattern {
pattern: pattern.to_string(),
error: error.to_string(),
})?;
let lines: Vec<&str> = text.lines().collect();
// fancy_regex can also fail at match time (backtracking limits); treat
// that as a non-match rather than failing the whole render.
let is_match: Vec<bool> = lines
.iter()
.map(|line| regex.is_match(line).unwrap_or(false))
.collect();
let mut keep = vec![false; lines.len()];
for (i, _) in is_match.iter().enumerate().filter(|&(_, matched)| *matched) {
let start = i.saturating_sub(PATTERN_CONTEXT_LINES);
let end = (i + PATTERN_CONTEXT_LINES).min(lines.len() - 1);
keep[start..=end].fill(true);
}
let mut out: Vec<String> = Vec::new();
let mut prev_kept: Option<usize> = None;
for (i, line) in lines.iter().enumerate() {
if !keep[i] {
continue;
}
if prev_kept.is_some_and(|prev| i > prev + 1) {
out.push(HUNK_SEPARATOR.to_string());
}
let marker = if is_match[i] { ':' } else { '-' };
out.push(format!("{}{marker}{line}", i + 1));
prev_kept = Some(i);
}
Ok(out.join("\n"))
}
fn decode_base64_bounded(b64: &str) -> Result<Vec<u8>, RenderError> {
// The encoded length puts a lower bound on the decoded size; reject
// inputs that bound already proves oversized before decoding anything.
let min_decoded = (b64.len() / 4).saturating_mul(3).saturating_sub(2);
if min_decoded > BLOB_DECODE_CEILING_BYTES {
return Err(RenderError::DecodedSizeExceeded);
}
let mut reader = DecoderReader::new(b64.as_bytes(), &STANDARD);
let mut decoded = Vec::new();
let mut chunk = [0u8; 8192];
loop {
match reader.read(&mut chunk) {
Ok(0) => return Ok(decoded),
Ok(n) => {
if decoded.len() + n > BLOB_DECODE_CEILING_BYTES {
return Err(RenderError::DecodedSizeExceeded);
}
decoded.extend_from_slice(&chunk[..n]);
}
Err(error) => return Err(RenderError::InvalidBase64(error.to_string())),
}
}
}
/// Maps a server-controlled mime type to a spill-file extension via an exact
/// allowlist lookup; anything unrecognized falls back to `bin`.
fn extension_for_mime(mime: Option<&str>) -> &'static str {
let Some(mime) = mime else {
return "bin";
};
let bare = mime
.split(';')
.next()
.unwrap_or("")
.trim()
.to_ascii_lowercase();
let ext = MIME_EXTENSIONS
.iter()
.find(|(known, _)| *known == bare)
.map(|(_, ext)| *ext)
.unwrap_or("bin");
let safe = !ext.is_empty()
&& ext.len() <= 8
&& ext
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit());
if safe { ext } else { "bin" }
}
fn sanitize_server(server: &str) -> String {
let mut sanitized: String = server
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-') {
c
} else {
'_'
}
})
.take(64)
.collect();
// Windows strips trailing dots at create time, which would make the
// constructed path disagree with the on-disk name.
while sanitized.ends_with('.') {
sanitized.pop();
}
if sanitized.is_empty() {
return "_".to_string();
}
// Windows reserves device names (bare or with any extension).
let stem = sanitized.split('.').next().unwrap_or("");
if is_windows_reserved(stem) {
sanitized.insert(0, '_');
}
sanitized
}
fn is_windows_reserved(stem: &str) -> bool {
let lower = stem.to_ascii_lowercase();
matches!(lower.as_str(), "con" | "prn" | "aux" | "nul")
|| (lower.len() == 4
&& (lower.starts_with("com") || lower.starts_with("lpt"))
&& matches!(lower.as_bytes()[3], b'1'..=b'9'))
}
struct SpillEntry {
path: PathBuf,
size: u64,
modified: SystemTime,
}
fn enforce_spill_bound(base: &Path, max_total: u64, protect: &Path) {
let mut entries = Vec::new();
collect_spill_files(base, &mut entries);
evict_oldest(entries, max_total, protect);
}
/// Best-effort eviction: the spill dir is shared across processes, so a file
/// vanishing underneath us (`NotFound`) is expected and never fails the spill.
fn evict_oldest(mut entries: Vec<SpillEntry>, max_total: u64, protect: &Path) {
let mut total: u64 = entries.iter().map(|entry| entry.size).sum();
if total <= max_total {
return;
}
entries.sort_by_key(|entry| entry.modified);
for entry in &entries {
if total <= max_total {
break;
}
// Filenames are content-hashed, so name equality is sufficient and
// survives filesystems that normalize directory names (case folding,
// trailing-dot stripping) where a full-path comparison would miss.
if entry.path.file_name() == protect.file_name() {
continue;
}
match fs::remove_file(&entry.path) {
Ok(()) => total -= entry.size,
Err(error) if error.kind() == ErrorKind::NotFound => total -= entry.size,
Err(_) => {}
}
}
}
fn collect_spill_files(dir: &Path, out: &mut Vec<SpillEntry>) {
let Ok(entries) = fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let path = entry.path();
let Ok(metadata) = entry.metadata() else {
continue;
};
if metadata.is_dir() {
collect_spill_files(&path, out);
} else if metadata.is_file() {
out.push(SpillEntry {
path,
size: metadata.len(),
modified: metadata.modified().unwrap_or(SystemTime::UNIX_EPOCH),
});
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use base64::Engine;
use std::env;
#[cfg(unix)]
use std::os::unix::fs::PermissionsExt;
use std::process;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
fn with_spill_base<F: FnOnce(&Path)>(f: F) {
static COUNTER: AtomicUsize = AtomicUsize::new(0);
let unique = format!(
"{}-{}",
process::id(),
COUNTER.fetch_add(1, Ordering::Relaxed)
);
let base = env::temp_dir().join(format!("coyote-render-test-{unique}"));
fs::create_dir_all(&base).unwrap();
f(&base);
let _ = fs::remove_dir_all(&base);
}
fn set_mtime(path: &Path, secs_after_epoch: u64) {
let file = OpenOptions::new().write(true).open(path).unwrap();
file.set_modified(SystemTime::UNIX_EPOCH + Duration::from_secs(secs_after_epoch))
.unwrap();
}
fn write_spill_file(dir: &Path, name: &str, len: usize, mtime_secs: u64) -> PathBuf {
let path = dir.join(name);
fs::write(&path, vec![0u8; len]).unwrap();
set_mtime(&path, mtime_secs);
path
}
const TEN_LINES: &str = "one\ntwo\nthree\nfour\nfive\nsix\nseven\neight\nnine\nten";
#[test]
fn slices_basic_ascii_page() {
let rendered = render_text("hello world", None, 0, Some(5)).unwrap();
assert_eq!(rendered.text, "hello");
assert!(rendered.truncated);
assert_eq!(rendered.total_bytes, 11);
assert_eq!(rendered.next_offset, Some(5));
}
#[test]
fn offset_mid_codepoint_rounds_forward() {
// 'é' occupies bytes 1..3; offset 2 lands inside it.
let rendered = render_text("héllo", None, 2, None).unwrap();
assert_eq!(rendered.text, "llo");
assert!(!rendered.truncated);
assert_eq!(rendered.next_offset, None);
}
#[test]
fn end_mid_codepoint_rounds_backward() {
// 'é' occupies bytes 1..3; offset 0 + max_bytes 2 lands inside it.
let rendered = render_text("", None, 0, Some(2)).unwrap();
assert_eq!(rendered.text, "a");
assert!(rendered.truncated);
assert_eq!(rendered.total_bytes, 3);
assert_eq!(rendered.next_offset, Some(1));
let rest = render_text("", None, 1, Some(2)).unwrap();
assert_eq!(rest.text, "é");
assert!(!rest.truncated);
}
#[test]
fn max_bytes_below_one_codepoint_still_advances() {
// 'é' is 2 bytes; max_bytes 1 must not stall at next_offset == offset.
let rendered = render_text("éa", None, 0, Some(1)).unwrap();
assert_eq!(rendered.text, "é");
assert!(rendered.truncated);
assert_eq!(rendered.total_bytes, 3);
assert_eq!(rendered.next_offset, Some(2));
}
#[test]
fn offset_past_eof_returns_empty() {
let rendered = render_text("short", None, 100, None).unwrap();
assert_eq!(rendered.text, "");
assert!(!rendered.truncated);
assert_eq!(rendered.total_bytes, 5);
assert_eq!(rendered.next_offset, None);
}
#[test]
fn exact_fit_is_not_truncated() {
let rendered = render_text("exact", None, 0, Some(5)).unwrap();
assert_eq!(rendered.text, "exact");
assert!(!rendered.truncated);
assert_eq!(rendered.next_offset, None);
}
#[test]
fn default_max_bytes_is_default_text_max_bytes() {
let text = "a".repeat(DEFAULT_TEXT_MAX_BYTES + 1);
let rendered = render_text(&text, None, 0, None).unwrap();
assert_eq!(rendered.text.len(), DEFAULT_TEXT_MAX_BYTES);
assert!(rendered.truncated);
assert_eq!(rendered.next_offset, Some(DEFAULT_TEXT_MAX_BYTES));
}
#[test]
fn max_bytes_above_clamp_is_clamped() {
let text = "a".repeat(TEXT_MAX_BYTES_CLAMP + 1);
let rendered = render_text(&text, None, 0, Some(usize::MAX)).unwrap();
assert_eq!(rendered.text.len(), TEXT_MAX_BYTES_CLAMP);
assert!(rendered.truncated);
assert_eq!(rendered.next_offset, Some(TEXT_MAX_BYTES_CLAMP));
}
#[test]
fn truncate_utf8_rounds_back_to_char_boundary() {
// 'é' occupies bytes 1..3; a cut at byte 2 lands inside it.
assert_eq!(truncate_utf8("", 2), "a");
assert_eq!(truncate_utf8("", 3), "");
assert_eq!(truncate_utf8("abc", 10), "abc");
assert_eq!(truncate_utf8("abc", 0), "");
}
#[test]
fn clamp_metadata_appends_marker_only_when_oversized() {
assert_eq!(clamp_metadata("text/plain"), "text/plain");
let long = "u".repeat(METADATA_MAX_BYTES + 1);
let clamped = clamp_metadata(&long);
assert!(clamped.starts_with(&"u".repeat(METADATA_MAX_BYTES)));
assert!(clamped.contains("METADATA_MAX_BYTES"));
assert!(clamped.contains(&METADATA_MAX_BYTES.to_string()));
}
#[test]
fn pattern_emits_matches_with_context_and_line_numbers() {
let rendered = render_text(TEN_LINES, Some("^five$"), 0, None).unwrap();
assert_eq!(rendered.text, "3-three\n4-four\n5:five\n6-six\n7-seven");
assert!(!rendered.truncated);
assert_eq!(rendered.total_bytes, rendered.text.len());
}
#[test]
fn pattern_separates_disjoint_hunks() {
let rendered = render_text(TEN_LINES, Some("^(two|nine)$"), 0, None).unwrap();
assert_eq!(
rendered.text,
"1-one\n2:two\n3-three\n4-four\n--\n7-seven\n8-eight\n9:nine\n10-ten"
);
}
#[test]
fn pattern_merges_adjacent_hunks_without_duplicates() {
let rendered = render_text(TEN_LINES, Some("^(two|six)$"), 0, None).unwrap();
assert_eq!(
rendered.text,
"1-one\n2:two\n3-three\n4-four\n5-five\n6:six\n7-seven\n8-eight"
);
assert!(!rendered.text.contains(HUNK_SEPARATOR));
}
#[test]
fn pattern_paging_walks_the_filtered_stream() {
let full = render_text(TEN_LINES, Some("^t"), 0, None).unwrap();
assert!(!full.truncated);
let mut assembled = String::new();
let mut offset = 0;
loop {
let page = render_text(TEN_LINES, Some("^t"), offset, Some(7)).unwrap();
assert_eq!(page.total_bytes, full.text.len());
assembled.push_str(&page.text);
match page.next_offset {
Some(next) => offset = next,
None => break,
}
}
assert_eq!(assembled, full.text);
}
#[test]
fn pattern_with_no_matches_returns_empty() {
let rendered = render_text(TEN_LINES, Some("^zebra$"), 0, None).unwrap();
assert_eq!(rendered.text, "");
assert_eq!(rendered.total_bytes, 0);
assert!(!rendered.truncated);
assert_eq!(rendered.next_offset, None);
}
#[test]
fn invalid_pattern_is_a_teaching_error() {
let parse_error = Regex::new("(").unwrap_err().to_string();
let err = render_text("text", Some("("), 0, None).unwrap_err();
assert!(matches!(err, RenderError::InvalidPattern { .. }));
let message = err.to_string();
assert!(message.contains("'('"));
assert!(message.contains(&parse_error));
}
#[test]
fn utf8_blob_decodes_to_text_without_spilling() {
with_spill_base(|base| {
let b64 = STANDARD.encode("hello ✓ world");
let rendered = render_blob_at(&b64, Some("text/plain"), "srv", base).unwrap();
let RenderedBlob::Text(text) = rendered else {
panic!("expected text variant");
};
assert_eq!(text, "hello ✓ world");
assert_eq!(fs::read_dir(base).unwrap().count(), 0);
});
}
#[test]
fn binary_blob_spills_with_metadata_and_0600_perms() {
with_spill_base(|base| {
let data: &[u8] = &[0xff, 0xfe, 0x00, 0x88, 0x01];
let b64 = STANDARD.encode(data);
let rendered = render_blob_at(&b64, Some("application/pdf"), "docs", base).unwrap();
let RenderedBlob::Spilled(meta) = rendered else {
panic!("expected spilled variant");
};
let expected_sha = format!("{:x}", Sha256::digest(data));
assert_eq!(meta.sha256, expected_sha);
assert_eq!(
meta.path,
base.join("docs").join(format!("{expected_sha}.pdf"))
);
assert_eq!(meta.size_bytes, data.len() as u64);
assert_eq!(meta.mime_type.as_deref(), Some("application/pdf"));
assert!(!meta.sniffed);
assert!(meta.spilled);
assert_eq!(fs::read(&meta.path).unwrap(), data);
#[cfg(unix)]
{
let mode = fs::metadata(&meta.path).unwrap().permissions().mode();
assert_eq!(mode & 0o777, 0o600);
}
});
}
#[test]
fn decode_ceiling_rejects_oversized_blob() {
with_spill_base(|base| {
// base64 of 51 MiB of zero bytes is just a repeated-'A' string.
let encoded = "A".repeat(51 * 1024 * 1024 / 3 * 4);
let err = render_blob_at(&encoded, None, "srv", base).unwrap_err();
assert!(matches!(err, RenderError::DecodedSizeExceeded));
assert!(err.to_string().contains("BLOB_DECODE_CEILING_BYTES"));
});
}
#[test]
fn malformed_base64_is_rejected() {
with_spill_base(|base| {
let err = render_blob_at("!!!not base64!!!", None, "srv", base).unwrap_err();
assert!(matches!(err, RenderError::InvalidBase64(_)));
});
}
#[test]
fn spill_dedup_returns_same_path_without_rewriting() {
with_spill_base(|base| {
let data: &[u8] = &[0xff, 0x01, 0x02];
let b64 = STANDARD.encode(data);
let RenderedBlob::Spilled(first) = render_blob_at(&b64, None, "srv", base).unwrap()
else {
panic!("expected spilled variant");
};
fs::write(&first.path, b"sentinel").unwrap();
let RenderedBlob::Spilled(second) = render_blob_at(&b64, None, "srv", base).unwrap()
else {
panic!("expected spilled variant");
};
assert_eq!(second.path, first.path);
assert_eq!(second.sha256, first.sha256);
assert_eq!(fs::read(&second.path).unwrap(), b"sentinel");
});
}
#[test]
fn spill_metadata_serializes_spilled_true() {
with_spill_base(|base| {
let b64 = STANDARD.encode([0xffu8, 0x00]);
let RenderedBlob::Spilled(meta) =
render_blob_at(&b64, Some("image/png"), "srv", base).unwrap()
else {
panic!("expected spilled variant");
};
let value = serde_json::to_value(&meta).unwrap();
assert_eq!(value["spilled"], serde_json::Value::Bool(true));
assert_eq!(value["sniffed"], serde_json::Value::Bool(false));
assert_eq!(value["sha256"].as_str(), Some(meta.sha256.as_str()));
assert_eq!(value["mime_type"].as_str(), Some("image/png"));
});
}
#[test]
fn extension_allowlist_normalizes_and_defaults_to_bin() {
assert_eq!(extension_for_mime(Some("application/pdf")), "pdf");
assert_eq!(extension_for_mime(Some("image/png")), "png");
assert_eq!(extension_for_mime(Some(" TEXT/CSV ; charset=utf-8")), "csv");
assert_eq!(extension_for_mime(Some("../../evil")), "bin");
assert_eq!(extension_for_mime(Some("image/png/../../x")), "bin");
assert_eq!(extension_for_mime(Some("application/x-∞")), "bin");
assert_eq!(extension_for_mime(Some("text/plain")), "bin");
assert_eq!(extension_for_mime(None), "bin");
}
#[test]
fn sanitize_server_strips_path_separators() {
assert_eq!(sanitize_server("../evil/srv"), ".._evil_srv");
assert_eq!(sanitize_server("srv name!"), "srv_name_");
assert_eq!(sanitize_server(""), "_");
assert_eq!(sanitize_server("."), "_");
assert_eq!(sanitize_server(".."), "_");
assert_eq!(sanitize_server("good-server_1.0"), "good-server_1.0");
}
#[test]
fn sanitize_server_escapes_windows_reserved_names() {
assert_eq!(sanitize_server("con"), "_con");
assert_eq!(sanitize_server("CON"), "_CON");
assert_eq!(sanitize_server("nul.txt"), "_nul.txt");
assert_eq!(sanitize_server("COM1"), "_COM1");
assert_eq!(sanitize_server("lpt9"), "_lpt9");
assert_eq!(sanitize_server("com0"), "com0");
assert_eq!(sanitize_server("com10"), "com10");
assert_eq!(sanitize_server("consul"), "consul");
}
#[test]
fn sanitize_server_strips_trailing_dots_and_caps_length() {
assert_eq!(sanitize_server("srv."), "srv");
assert_eq!(sanitize_server("srv..."), "srv");
assert_eq!(sanitize_server("..."), "_");
let long = "a".repeat(100);
assert_eq!(sanitize_server(&long).len(), 64);
}
#[test]
fn spill_path_confines_crafted_server_and_mime() {
with_spill_base(|base| {
let b64 = STANDARD.encode([0xffu8, 0x00, 0x11]);
let RenderedBlob::Spilled(meta) =
render_blob_at(&b64, Some("../../evil"), "../evil/srv", base).unwrap()
else {
panic!("expected spilled variant");
};
assert!(meta.path.starts_with(base));
let dir_name = meta.path.parent().unwrap().file_name().unwrap();
assert_eq!(dir_name, ".._evil_srv");
assert_eq!(meta.path.extension().unwrap(), "bin");
});
}
#[test]
fn eviction_removes_oldest_files_first_across_server_dirs() {
with_spill_base(|base| {
let srv_a = base.join("srv-a");
let srv_b = base.join("srv-b");
fs::create_dir_all(&srv_a).unwrap();
fs::create_dir_all(&srv_b).unwrap();
let oldest = write_spill_file(&srv_a, "a.bin", 100, 100);
let middle = write_spill_file(&srv_b, "b.bin", 100, 200);
let newest = write_spill_file(&srv_b, "c.bin", 100, 300);
enforce_spill_bound(base, 150, &newest);
assert!(!oldest.exists());
assert!(!middle.exists());
assert!(newest.exists());
});
}
#[test]
fn eviction_skips_protected_file() {
with_spill_base(|base| {
let srv = base.join("srv");
fs::create_dir_all(&srv).unwrap();
let oldest = write_spill_file(&srv, "a.bin", 100, 100);
let middle = write_spill_file(&srv, "b.bin", 100, 200);
let newest = write_spill_file(&srv, "c.bin", 100, 300);
enforce_spill_bound(base, 250, &oldest);
assert!(oldest.exists());
assert!(!middle.exists());
assert!(newest.exists());
});
}
#[test]
fn eviction_under_bound_is_noop() {
with_spill_base(|base| {
let srv = base.join("srv");
fs::create_dir_all(&srv).unwrap();
let first = write_spill_file(&srv, "a.bin", 100, 100);
let second = write_spill_file(&srv, "b.bin", 100, 200);
enforce_spill_bound(base, 1000, &second);
assert!(first.exists());
assert!(second.exists());
});
}
#[test]
fn eviction_tolerates_already_removed_entries() {
with_spill_base(|base| {
let srv = base.join("srv");
fs::create_dir_all(&srv).unwrap();
let real = write_spill_file(&srv, "real.bin", 100, 200);
let entries = vec![
SpillEntry {
path: srv.join("ghost.bin"),
size: 100,
modified: SystemTime::UNIX_EPOCH + Duration::from_secs(100),
},
SpillEntry {
path: real.clone(),
size: 100,
modified: SystemTime::UNIX_EPOCH + Duration::from_secs(200),
},
];
evict_oldest(entries, 50, &base.join("untouched"));
assert!(!real.exists());
});
}
}
+68 -3
View File
@@ -1,11 +1,11 @@
use crate::function::{FunctionDeclaration, JsonSchema}; use crate::function::{self, FunctionDeclaration, JsonSchema};
use anyhow::{Context, Result, bail}; use anyhow::{Context, Result, bail};
use argc::{ChoiceValue, CommandValue, FlagOptionValue}; use argc::{ChoiceValue, CommandValue, FlagOptionValue};
use indexmap::IndexMap; use indexmap::IndexMap;
use std::env;
use std::fs::File; use std::fs::File;
use std::io::Read; use std::io::Read;
use std::path::Path; use std::path::Path;
use std::{env, fs};
pub fn generate_bash_declarations( pub fn generate_bash_declarations(
mut tool_file: File, mut tool_file: File,
@@ -23,7 +23,8 @@ pub fn generate_bash_declarations(
"", "",
env::var("TERM_WIDTH").ok().and_then(|v| v.parse().ok()), env::var("TERM_WIDTH").ok().and_then(|v| v.parse().ok()),
)?; )?;
fs::write(tools_file_path, &build_script) let build_script = allow_empty_required_values(&build_script);
function::write_file_atomic(tools_file_path, &build_script, Some(0o755))
.with_context(|| format!("Failed to write built script to '{tools_file_path:?}'"))?; .with_context(|| format!("Failed to write built script to '{tools_file_path:?}'"))?;
let command_value = argc::export(&build_script, file_name) let command_value = argc::export(&build_script, file_name)
@@ -74,6 +75,20 @@ fn underscore(s: &str) -> String {
s.replace('-', "_") s.replace('-', "_")
} }
/// argc's generated required-param check uses `-z "${!name:-}"`, which
/// conflates "not provided" with "provided but empty", so a required option
/// passed an explicit empty string (e.g. `fs_write --content=''` to create an
/// empty file) is rejected as "required arguments were not provided". The
/// JSON schema we advertise to models treats `required` as *presence*, so
/// rewrite the check to a set-ness test to keep runtime behavior consistent
/// with the schema. Applied post-build so it survives every regeneration.
fn allow_empty_required_values(build_script: &str) -> String {
build_script.replace(
r#"if [[ -z "${!name:-}" ]]; then"#,
r#"if [[ -z "${!name+x}" ]]; then"#,
)
}
fn schema_ty(t: &str) -> JsonSchema { fn schema_ty(t: &str) -> JsonSchema {
JsonSchema { JsonSchema {
type_value: Some(t.to_string()), type_value: Some(t.to_string()),
@@ -147,3 +162,53 @@ fn parse_parameters_schema(flags: &[FlagOptionValue]) -> JsonSchema {
required: Some(required), required: Some(required),
} }
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn rewrites_argc_required_check_to_setness_test() {
let src = "# @describe test tool\n# @option --content! The content\nmain() { :; }\n";
let built = argc::build(src, "", None).expect("argc build failed");
assert!(
built.contains(r#"if [[ -z "${!name:-}" ]]; then"#),
"argc changed its generated required-param template; update allow_empty_required_values()"
);
let fixed = allow_empty_required_values(&built);
assert!(fixed.contains(r#"if [[ -z "${!name+x}" ]]; then"#));
assert!(!fixed.contains(r#"if [[ -z "${!name:-}" ]]; then"#));
}
#[cfg(unix)]
#[test]
fn second_build_does_not_rewrite_tool_file_test() {
use std::fs;
use std::os::unix::fs::{MetadataExt, PermissionsExt};
let src = "# @describe test tool\n# @option --content! The content\nmain() { :; }\n";
let path = crate::utils::temp_file("bash-build", ".sh");
fs::write(&path, src).expect("failed to write temp script");
let declarations = generate_bash_declarations(File::open(&path).unwrap(), &path, "test")
.expect("first build failed");
assert!(!declarations.is_empty());
let first_ino = fs::metadata(&path).unwrap().ino();
let declarations = generate_bash_declarations(File::open(&path).unwrap(), &path, "test")
.expect("second build failed");
assert!(!declarations.is_empty());
let metadata = fs::metadata(&path).unwrap();
assert_eq!(
metadata.ino(),
first_ino,
"second build rewrote an already-built tool file"
);
assert_eq!(metadata.permissions().mode() & 0o777, 0o755);
fs::remove_file(&path).unwrap();
}
}
+37 -37
View File
@@ -151,7 +151,7 @@ impl Rag {
println!("⚙ Initializing RAG..."); println!("⚙ Initializing RAG...");
let mut data = Self::resolve_init_data(app, config)?; let mut data = Self::resolve_init_data(app, config)?;
data.driver = config.driver.clone().unwrap_or_else(|| "yaml".to_string()); data.driver = config.driver.clone().unwrap_or_else(|| "yaml".to_string());
let mut rag = Self::create(app, name, save_path, data)?; let mut rag = Self::create(app, name, save_path, data).await?;
let loaders = app.document_loaders.clone(); let loaders = app.document_loaders.clone();
let (spinner, spinner_rx) = Spinner::create(""); let (spinner, spinner_rx) = Spinner::create("");
abortable_run_with_spinner_rx( abortable_run_with_spinner_rx(
@@ -298,7 +298,7 @@ impl Rag {
}, },
); );
data.driver = driver; data.driver = driver;
let mut rag = Self::create(app, name, save_path, data)?; let mut rag = Self::create(app, name, save_path, data).await?;
let mut paths = doc_paths.to_vec(); let mut paths = doc_paths.to_vec();
if paths.is_empty() { if paths.is_empty() {
paths = add_documents()?; paths = add_documents()?;
@@ -317,12 +317,12 @@ impl Rag {
Ok(rag) Ok(rag)
} }
pub fn load(app: &AppConfig, name: &str, path: &Path) -> Result<Self> { pub async fn load(app: &AppConfig, name: &str, path: &Path) -> Result<Self> {
let err = || format!("Failed to load rag '{name}' at '{}'", path.display()); let err = || format!("Failed to load rag '{name}' at '{}'", path.display());
let content = fs::read_to_string(path).with_context(err)?; let content = fs::read_to_string(path).with_context(err)?;
let data: RagData = serde_yaml::from_str(&content).with_context(err)?; let data: RagData = serde_yaml::from_str(&content).with_context(err)?;
data.validate().with_context(err)?; data.validate().with_context(err)?;
Self::create(app, name, path, data) Self::create(app, name, path, data).await
} }
/// Loads a RAG from a YAML file. External drivers need an async constructor /// Loads a RAG from a YAML file. External drivers need an async constructor
@@ -372,7 +372,7 @@ impl Rag {
last_sources: RwLock::new(None), last_sources: RwLock::new(None),
}) })
} }
_ => Self::load(app, name, path), _ => Self::load(app, name, path).await,
} }
} }
@@ -537,14 +537,22 @@ impl Rag {
Ok(rag) Ok(rag)
} }
pub fn create(app: &AppConfig, name: &str, path: &Path, mut data: RagData) -> Result<Self> { pub async fn create(
app: &AppConfig,
name: &str,
path: &Path,
mut data: RagData,
) -> Result<Self> {
// Deliberately does NOT call rebuild_indexes: both callers construct the Rag // Deliberately does NOT call rebuild_indexes: both callers construct the Rag
// before any documents are added, so rebuilding empty data would be a no-op. // before any documents are added, so rebuilding empty data would be a no-op.
// Actual population happens later via sync_documents. // Actual population happens later via sync_documents.
let (provider, bm25): (Box<dyn RagProvider>, _) = match data.driver.as_str() { let (provider, bm25): (Box<dyn RagProvider>, _) = match data.driver.as_str() {
"duckdb" => { "duckdb" => {
let db_path = providers::duckdb_path_from_yaml(path); let db_path = providers::duckdb_path_from_yaml(path);
let dim = embedding_dim_for_model(&data.embedding_model); let dim = match DuckDbProvider::introspect_dim(&db_path)? {
Some(existing) => existing,
None => probe_embedding_dim(app, &data.embedding_model).await?,
};
let duck = DuckDbProvider::open(&db_path, dim)?; let duck = DuckDbProvider::open(&db_path, dim)?;
// HYDRATE — mandatory, not an optimization. The YAML file for a duckdb // HYDRATE — mandatory, not an optimization. The YAML file for a duckdb
// RAG deliberately omits `vectors`, so `data.vectors` arrives empty from // RAG deliberately omits `vectors`, so `data.vectors` arrives empty from
@@ -2119,21 +2127,25 @@ fn reciprocal_rank_fusion(
.collect() .collect()
} }
/// Map an embedding model id to its vector dimension. async fn probe_embedding_dim(app: &AppConfig, model_id: &str) -> Result<usize> {
/// let model = Model::retrieve_model(app, model_id, ModelType::Embedding)?;
/// The DuckDB `FLOAT[N]` column type and its HNSW index are fixed at schema-creation let client = init_client(&Arc::new(app.clone()), model)?;
/// time, so this value must be decided before the first insert. An unrecognized model let out = client
/// falls back to 1536; if that is wrong, DuckDB raises a dimension-mismatch error on .embeddings(&EmbeddingsData::new(vec!["dimension probe".into()], false))
/// the first insert rather than silently corrupting the schema, and the recovery is to .await
/// delete the sidecar and re-ingest from source. .with_context(|| {
fn embedding_dim_for_model(model_id: &str) -> usize { format!(
match model_id { "Failed to probe the embedding dimension of model '{model_id}'. \
m if m.contains("3-large") => 3072, Creating a duckdb RAG requires one call to the embedding endpoint."
m if m.contains("3-small") || m.contains("ada-002") => 1536, )
m if m.contains("nomic-embed-text") || m.contains("all-minilm") => 768, })?;
m if m.contains("jina-embeddings-v2") => 1024, let dim = out.first().map(|v| v.len()).unwrap_or(0);
_ => 1536,
if dim == 0 {
bail!("Embedding model '{model_id}' returned an empty vector during the dimension probe");
} }
Ok(dim)
} }
/// True only for "the vault does not hold this key". /// True only for "the vault does not hold this key".
@@ -2673,18 +2685,6 @@ mod tests {
assert_eq!(data.attached_source_label(), "[external collection]"); assert_eq!(data.attached_source_label(), "[external collection]");
} }
#[test]
fn embedding_dim_for_model_maps_known_models() {
assert_eq!(embedding_dim_for_model("text-embedding-3-large"), 3072);
assert_eq!(embedding_dim_for_model("text-embedding-3-small"), 1536);
assert_eq!(embedding_dim_for_model("text-embedding-ada-002"), 1536);
assert_eq!(embedding_dim_for_model("nomic-embed-text"), 768);
assert_eq!(embedding_dim_for_model("all-minilm"), 768);
assert_eq!(embedding_dim_for_model("jina-embeddings-v2-base-en"), 1024);
// Unknown models fall back to the OpenAI-compatible default.
assert_eq!(embedding_dim_for_model("some-unknown-model"), 1536);
}
#[test] #[test]
fn document_id_round_trip() { fn document_id_round_trip() {
let id = DocumentId::new(5, 17); let id = DocumentId::new(5, 17);
@@ -3044,7 +3044,7 @@ mod tests {
#[test] #[test]
fn reciprocal_rank_fusion_empty_lists() { fn reciprocal_rank_fusion_empty_lists() {
let result = super::reciprocal_rank_fusion(vec![], vec![], 5); let result = reciprocal_rank_fusion(vec![], vec![], 5);
assert!(result.is_empty(), "empty input should produce empty output"); assert!(result.is_empty(), "empty input should produce empty output");
} }
@@ -3052,7 +3052,7 @@ mod tests {
fn reciprocal_rank_fusion_deduplicates_across_signals() { fn reciprocal_rank_fusion_deduplicates_across_signals() {
let doc_a = DocumentId::new(0, 0); let doc_a = DocumentId::new(0, 0);
let doc_b = DocumentId::new(0, 1); let doc_b = DocumentId::new(0, 1);
let result = super::reciprocal_rank_fusion( let result = reciprocal_rank_fusion(
vec![vec![doc_a, doc_b], vec![doc_a, doc_b]], vec![vec![doc_a, doc_b], vec![doc_a, doc_b]],
vec![1.0, 1.0], vec![1.0, 1.0],
5, 5,
@@ -3069,7 +3069,7 @@ mod tests {
#[test] #[test]
fn reciprocal_rank_fusion_respects_top_k() { fn reciprocal_rank_fusion_respects_top_k() {
let docs: Vec<DocumentId> = (0..10).map(|i| DocumentId::new(0, i)).collect(); let docs: Vec<DocumentId> = (0..10).map(|i| DocumentId::new(0, i)).collect();
let result = super::reciprocal_rank_fusion(vec![docs], vec![1.0], 3); let result = reciprocal_rank_fusion(vec![docs], vec![1.0], 3);
assert_eq!(result.len(), 3, "result should be capped at top_k=3"); assert_eq!(result.len(), 3, "result should be capped at top_k=3");
} }
@@ -3077,7 +3077,7 @@ mod tests {
fn reciprocal_rank_fusion_weights_affect_ranking() { fn reciprocal_rank_fusion_weights_affect_ranking() {
let doc_a = DocumentId::new(0, 0); let doc_a = DocumentId::new(0, 0);
let doc_b = DocumentId::new(0, 1); let doc_b = DocumentId::new(0, 1);
let result = super::reciprocal_rank_fusion( let result = reciprocal_rank_fusion(
vec![vec![doc_a, doc_b], vec![doc_b, doc_a]], vec![vec![doc_a, doc_b], vec![doc_b, doc_a]],
vec![10.0, 1.0], vec![10.0, 1.0],
2, 2,
+211 -9
View File
@@ -5,7 +5,7 @@ use std::collections::HashMap;
use anyhow::{Context, Result, anyhow, bail}; use anyhow::{Context, Result, anyhow, bail};
use async_trait::async_trait; use async_trait::async_trait;
use duckdb::types::Value; use duckdb::types::Value;
use duckdb::{AccessMode, Config, Connection}; use duckdb::{AccessMode, Config, Connection, OptionalExt};
use indexmap::IndexMap; use indexmap::IndexMap;
use log::warn; use log::warn;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
@@ -30,6 +30,11 @@ pub(crate) fn duckdb_path_from_yaml(yaml_path: &Path) -> PathBuf {
yaml_path.with_extension("duckdb") yaml_path.with_extension("duckdb")
} }
fn parse_float_array_dim(data_type: &str) -> Option<usize> {
let inner = data_type.strip_prefix("FLOAT[")?.strip_suffix(']')?;
inner.parse::<usize>().ok().filter(|&n| n > 0)
}
/// The shared connection together with the access mode it was opened with. /// The shared connection together with the access mode it was opened with.
/// ///
/// `conn` is an `Option` only so that an upgrade can DROP the read-only connection /// `conn` is an `Option` only so that an upgrade can DROP the read-only connection
@@ -42,6 +47,11 @@ struct ConnHandle {
/// `duplicate()` clone sharing the `Arc` observes an upgrade performed through any /// `duplicate()` clone sharing the `Arc` observes an upgrade performed through any
/// other handle instead of keeping its own stale copy of the mode. /// other handle instead of keeping its own stale copy of the mode.
writable: bool, writable: bool,
/// Embedding dimension the `FLOAT[N]` column was opened (or last rebuilt) with.
/// Shared for the same reason as `writable`: a self-healing rebuild through one
/// handle updates the width, and every `duplicate()` clone must cast with the new
/// width instead of erroring on a healthy store with its stale copy.
dim: usize,
} }
impl ConnHandle { impl ConnHandle {
@@ -68,8 +78,6 @@ impl ConnHandle {
pub struct DuckDbProvider { pub struct DuckDbProvider {
path: PathBuf, path: PathBuf,
conn: Arc<Mutex<ConnHandle>>, conn: Arc<Mutex<ConnHandle>>,
/// Embedding dimension; fixed at open time because the `FLOAT[N]` column depends on it.
dim: usize,
/// True once an FTS index has been built on `documents`. Until then /// True once an FTS index has been built on `documents`. Until then
/// `fts_main_documents.match_bm25` does not exist and any keyword query would /// `fts_main_documents.match_bm25` does not exist and any keyword query would
/// fail with a DuckDB catalog error. Backs `has_native_keyword_search`. /// fail with a DuckDB catalog error. Backs `has_native_keyword_search`.
@@ -96,8 +104,8 @@ impl DuckDbProvider {
conn: Arc::new(Mutex::new(ConnHandle { conn: Arc::new(Mutex::new(ConnHandle {
conn: Some(conn), conn: Some(conn),
writable, writable,
})),
dim, dim,
})),
fts_ready: AtomicBool::new(fts_exists), fts_ready: AtomicBool::new(fts_exists),
}) })
} }
@@ -164,6 +172,35 @@ impl DuckDbProvider {
Ok(conn) Ok(conn)
} }
pub fn introspect_dim(db_path: &Path) -> Result<Option<usize>> {
if !db_path.exists() {
return Ok(None);
}
let conn = Self::open_read_only(db_path).with_context(|| {
format!(
"Cannot inspect the existing RAG store at '{}'",
db_path.display()
)
})?;
let ty: Option<String> = conn
.query_row(
"SELECT data_type FROM duckdb_columns() \
WHERE table_name = 'vectors' AND column_name = 'embedding'",
[],
|r| r.get(0),
)
.optional()
.with_context(|| {
format!(
"Failed to introspect the embedding dimension of the DuckDB store \
at '{}'",
db_path.display()
)
})?;
Ok(ty.and_then(|t| parse_float_array_dim(&t)))
}
/// Open the store read-write and make sure its schema exists. Exactly one process /// Open the store read-write and make sure its schema exists. Exactly one process
/// may hold such a handle, and no reader from another process may hold it meanwhile. /// may hold such a handle, and no reader from another process may hold it meanwhile.
fn open_read_write(db_path: &Path, dim: usize) -> Result<Connection> { fn open_read_write(db_path: &Path, dim: usize) -> Result<Connection> {
@@ -245,7 +282,7 @@ impl DuckDbProvider {
return Ok(()); return Ok(());
} }
drop(handle.conn.take()); drop(handle.conn.take());
match Self::open_read_write(&self.path, self.dim) { match Self::open_read_write(&self.path, handle.dim) {
Ok(conn) => { Ok(conn) => {
handle.conn = Some(conn); handle.conn = Some(conn);
handle.writable = true; handle.writable = true;
@@ -428,13 +465,33 @@ impl RagProvider for DuckDbProvider {
if embedding.iter().any(|f| !f.is_finite()) { if embedding.iter().any(|f| !f.is_finite()) {
bail!("Query embedding contains a non-finite value (NaN or infinity)"); bail!("Query embedding contains a non-finite value (NaN or infinity)");
} }
let handle = self.lock_conn()?;
let dim = handle.dim;
if embedding.len() != dim {
let rows: i64 = handle
.conn()?
.query_row("SELECT count(*) FROM vectors", [], |r| r.get(0))
.context("Failed to count vectors before a dimension-mismatch query")?;
if rows == 0 {
// A never-synced store answers "nothing", not a cast error.
return Ok(Vec::new());
}
bail!(
"RAG store at '{}' was built with {dim}-dim embeddings, but the \
embedding model now returns {}-dim vectors. The embedding model \
changed since ingestion. Re-embed the documents, or delete the \
sidecar file and re-ingest.",
self.path.display(),
embedding.len()
);
}
let vals: String = embedding let vals: String = embedding
.iter() .iter()
.map(|f| f.to_string()) .map(|f| f.to_string())
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join(", "); .join(", ");
let dim = self.dim;
let handle = self.lock_conn()?;
// array_cosine_distance requires a FLOAT[N] ARRAY, not the LIST type FLOAT[]. // array_cosine_distance requires a FLOAT[N] ARRAY, not the LIST type FLOAT[].
// ORDER BY distance ASC is required for the planner to use hnsw_idx; the // ORDER BY distance ASC is required for the planner to use hnsw_idx; the
// similarity form (DESC) does NOT trigger the ANN index. Distance is converted // similarity form (DESC) does NOT trigger the ANN index. Distance is converted
@@ -554,7 +611,28 @@ impl RagProvider for DuckDbProvider {
); );
} }
} }
let dim = self.dim; let dim = match data.vectors.first() {
None => self.lock_conn()?.dim,
Some((_, first)) => {
let dim = first.len();
if let Some((doc_id, other)) = data.vectors.iter().find(|(_, e)| e.len() != dim) {
let matching = data.vectors.values().filter(|e| e.len() == dim).count();
bail!(
"Refusing to rebuild the RAG store at '{}': the rebuild batch \
mixes {dim}-dim and {}-dim vectors ({matching} vs {} vectors; \
first mismatch: document {}). Re-embed the documents, or \
delete the sidecar file and re-ingest.",
self.path.display(),
other.len(),
data.vectors.len() - matching,
doc_id.0
);
}
dim
}
};
// THE write path. Everything above this line only reads, so the upgrade happens // THE write path. Everything above this line only reads, so the upgrade happens
// here, after both guards have had their say: a rebuild that is going to be // here, after both guards have had their say: a rebuild that is going to be
// refused must not first take the exclusive lock away from other processes. // refused must not first take the exclusive lock away from other processes.
@@ -664,6 +742,8 @@ impl RagProvider for DuckDbProvider {
// fall back to local BM25, which is also empty, and therefore correct. // fall back to local BM25, which is also empty, and therefore correct.
self.fts_ready.store(doc_count > 0, Ordering::Relaxed); self.fts_ready.store(doc_count > 0, Ordering::Relaxed);
handle.dim = dim;
Ok(()) Ok(())
} }
@@ -729,10 +809,11 @@ impl RagProvider for DuckDbProvider {
// Sharing the Arc also shares the ACCESS MODE, which lives inside the ConnHandle // Sharing the Arc also shares the ACCESS MODE, which lives inside the ConnHandle
// rather than beside it: when one handle upgrades itself to read-write, every // rather than beside it: when one handle upgrades itself to read-write, every
// clone is upgraded with it and none is left holding a stale "read-only" belief. // clone is upgraded with it and none is left holding a stale "read-only" belief.
// The embedding dimension lives there too, so a self-healing rebuild through
// one handle updates the width every clone casts with.
Box::new(DuckDbProvider { Box::new(DuckDbProvider {
path: self.path.clone(), path: self.path.clone(),
conn: Arc::clone(&self.conn), conn: Arc::clone(&self.conn),
dim: self.dim,
fts_ready: AtomicBool::new(self.fts_ready.load(Ordering::Relaxed)), fts_ready: AtomicBool::new(self.fts_ready.load(Ordering::Relaxed)),
}) })
} }
@@ -1074,6 +1155,127 @@ mod tests {
.expect("a fresh RAG with nothing indexed must rebuild cleanly"); .expect("a fresh RAG with nothing indexed must rebuild cleanly");
} }
#[test]
fn parse_float_array_dim_handles_arrays_lists_and_scalars() {
assert_eq!(parse_float_array_dim("FLOAT[768]"), Some(768));
assert_eq!(parse_float_array_dim("FLOAT[]"), None);
assert_eq!(parse_float_array_dim("VARCHAR"), None);
}
#[test]
fn introspect_dim_round_trips_the_open_dim() {
let db = TempDb::new("introspect");
assert_eq!(
DuckDbProvider::introspect_dim(&db.path).unwrap(),
None,
"a file that does not exist has no dim"
);
{
let _provider = DuckDbProvider::open(&db.path, 5).unwrap();
}
assert_eq!(DuckDbProvider::introspect_dim(&db.path).unwrap(), Some(5));
}
#[test]
fn introspect_dim_propagates_an_unopenable_existing_file() {
let db = TempDb::new("introspectgarbage");
fs::write(&db.path, b"not a duckdb database").unwrap();
let err = DuckDbProvider::introspect_dim(&db.path).unwrap_err();
assert!(
format!("{err:#}").contains(&format!(
"Cannot inspect the existing RAG store at '{}'",
db.path.display()
)),
"an existing-but-unopenable file must be an error naming the path; got: {err:#}"
);
}
#[tokio::test]
async fn rebuild_indexes_self_heals_dim_from_the_vectors_it_writes() {
let db = TempDb::new("selfheal");
let mut provider = DuckDbProvider::open(&db.path, 5).unwrap();
let mut data = minimal_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
provider.rebuild_indexes(&data, true).await.unwrap();
let results = provider
.vector_search(&[0.1, 0.2, 0.3], 5, 0.0)
.await
.unwrap();
assert_eq!(results.len(), 1);
drop(provider);
assert_eq!(DuckDbProvider::introspect_dim(&db.path).unwrap(), Some(3));
}
#[tokio::test]
async fn a_self_healed_dim_is_visible_through_duplicate_clones() {
let db = TempDb::new("dimdup");
let mut provider = DuckDbProvider::open(&db.path, 5).unwrap();
let dup = provider.duplicate(&minimal_rag_data());
let mut data = minimal_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
provider.rebuild_indexes(&data, true).await.unwrap();
let results = dup.vector_search(&[0.1, 0.2, 0.3], 5, 0.0).await.unwrap();
assert_eq!(
results.len(),
1,
"a duplicate() clone must observe the dim written by a rebuild through \
the original"
);
}
#[tokio::test]
async fn rebuild_indexes_rejects_mixed_dim_vectors() {
let db = TempDb::new("mixeddim");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
let mut data = minimal_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
data.vectors.insert(DocumentId(1), vec![0.1, 0.2, 0.3, 0.4]);
let err = provider.rebuild_indexes(&data, true).await.unwrap_err();
assert!(
err.to_string().contains("rebuild batch mixes"),
"got: {err}"
);
}
#[tokio::test]
async fn vector_search_dim_mismatch_on_empty_store_returns_nothing() {
let db = TempDb::new("dimempty");
let provider = DuckDbProvider::open(&db.path, 3).unwrap();
let results = provider.vector_search(&[0.1, 0.2], 5, 0.0).await.unwrap();
assert!(
results.is_empty(),
"a never-synced store must answer 'nothing', not a cast error"
);
}
#[tokio::test]
async fn vector_search_dim_mismatch_on_populated_store_errors() {
let db = TempDb::new("dimfull");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
let mut data = minimal_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
provider.rebuild_indexes(&data, true).await.unwrap();
let err = provider
.vector_search(&[0.1, 0.2], 5, 0.0)
.await
.unwrap_err();
assert!(err.to_string().contains("was built with"), "got: {err}");
}
#[tokio::test] #[tokio::test]
async fn duplicate_shares_the_same_connection() { async fn duplicate_shares_the_same_connection() {
let db = TempDb::new("dup"); let db = TempDb::new("dup");
+34 -2
View File
@@ -39,8 +39,10 @@ static INLINE_CODE_RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"`([^`\n]+
static IMAGE_RE: LazyLock<Regex> = static IMAGE_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"!\[([^\]]*)\]\(([^)]+)\)").unwrap()); LazyLock::new(|| Regex::new(r"!\[([^\]]*)\]\(([^)]+)\)").unwrap());
static LINK_RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"\[([^\]]+)\]\(([^)]+)\)").unwrap()); static LINK_RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"\[([^\]]+)\]\(([^)]+)\)").unwrap());
static BOLD_AST_RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"\*\*([^*\n]+)\*\*").unwrap()); static BOLD_AST_RE: LazyLock<Regex> =
static BOLD_US_RE: LazyLock<Regex> = LazyLock::new(|| Regex::new(r"__([^_\n]+)__").unwrap()); LazyLock::new(|| Regex::new(r"\*\*((?:[^*\n]|\*(?!\*))+?)\*\*").unwrap());
static BOLD_US_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"__((?:[^_\n]|_(?!_))+?)__").unwrap());
static ITALIC_AST_RE: LazyLock<Regex> = static ITALIC_AST_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?<![*\w])\*(?!\s)([^*\n]+?)(?<!\s)\*(?!\*)").unwrap()); LazyLock::new(|| Regex::new(r"(?<![*\w])\*(?!\s)([^*\n]+?)(?<!\s)\*(?!\*)").unwrap());
static ITALIC_US_RE: LazyLock<Regex> = static ITALIC_US_RE: LazyLock<Regex> =
@@ -2419,6 +2421,36 @@ std::error::Error>> {
assert!(result.contains("\x1b[9m"), "strikethrough SGR: {result:?}"); assert!(result.contains("\x1b[9m"), "strikethrough SGR: {result:?}");
} }
#[test]
fn bold_asterisk_wraps_italic_asterisk() {
let styles = test_styles();
let result = apply_inline("**loud *soft* loud**", &styles);
assert!(
!result.contains("**"),
"outer bold markers stripped: {result:?}"
);
assert!(result.contains("\x1b[1m"), "bold SGR present: {result:?}");
assert!(result.contains("\x1b[3m"), "italic SGR present: {result:?}");
assert!(result.contains("soft"));
}
#[test]
fn bold_underscore_wraps_italic_underscore() {
let styles = test_styles();
let result = apply_inline("__loud _soft_ loud__", &styles);
assert!(
!result.contains("__"),
"outer bold markers stripped: {result:?}"
);
assert!(result.contains("\x1b[1m"), "bold SGR present: {result:?}");
assert!(result.contains("\x1b[3m"), "italic SGR present: {result:?}");
assert!(result.contains("soft"));
}
#[test] #[test]
fn bold_wraps_inline_code() { fn bold_wraps_inline_code() {
let styles = test_styles(); let styles = test_styles();
+312 -2
View File
@@ -1,11 +1,17 @@
use super::{REPL_COMMANDS, ReplCommand}; use super::{REPL_COMMANDS, ReplCommand};
use crate::{config::RequestContext, utils::fuzzy_filter}; use crate::config::{McpPromptCompletion, RequestContext, sanitize_display_text};
use crate::mcp::ConnectedServer;
use crate::utils::fuzzy_filter;
use parking_lot::RwLock; use parking_lot::RwLock;
use reedline::{Completer, Span, Suggestion}; use reedline::{Completer, Span, Suggestion};
use rmcp::model::Prompt;
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration;
const PROMPT_COMPLETION_RPC_TIMEOUT: Duration = Duration::from_secs(2);
impl Completer for ReplCompleter { impl Completer for ReplCompleter {
fn complete(&mut self, line: &str, pos: usize) -> Vec<Suggestion> { fn complete(&mut self, line: &str, pos: usize) -> Vec<Suggestion> {
@@ -29,6 +35,22 @@ impl Completer for ReplCompleter {
return suggestions; return suggestions;
} }
if cmd == ".prompt" && parts_len > 1 {
let span = Span::new(parts[parts_len - 1].1, pos);
let args: Vec<&str> = parts.iter().skip(1).map(|(v, _)| *v).collect();
let filter = args.last().copied().unwrap_or_default().to_string();
let stage = {
let ctx = self.ctx.read();
ctx.mcp_prompt_completion(&args)
};
return complete_prompt_stage(stage, &filter, PROMPT_COMPLETION_RPC_TIMEOUT)
.iter()
.map(|(value, description)| {
create_suggestion(value, description.as_deref().unwrap_or_default(), span)
})
.collect();
}
let ctx = self.ctx.read(); let ctx = self.ctx.read();
let state = ctx.state(); let state = ctx.state();
let model_has_reasoning = !ctx.current_model().reasoning_levels().is_empty(); let model_has_reasoning = !ctx.current_model().reasoning_levels().is_empty();
@@ -74,7 +96,25 @@ impl Completer for ReplCompleter {
format!("{name} ") format!("{name} ")
}; };
create_suggestion(&name, description, span) create_suggestion(&name, description, span)
})) }));
let macros: Vec<(String, Option<String>)> = ctx
.visible_macro_completions()
.into_iter()
.map(|(name, description)| (format!(".{name}"), description))
.filter(|(name, _)| {
command_filter.len() == 1
|| name.starts_with(command_filter.get(..2).unwrap_or(&command_filter))
})
.collect();
let macros = fuzzy_filter(macros, |(name, _)| name.as_str(), &command_filter);
suggestions.extend(macros.iter().map(|(name, description)| {
create_suggestion(
&format!("{name} "),
description.as_deref().unwrap_or_default(),
span,
)
}));
} }
suggestions suggestions
} }
@@ -123,6 +163,71 @@ fn create_suggestion(value: &str, description: &str, span: Span) -> Suggestion {
} }
} }
fn complete_prompt_stage(
stage: McpPromptCompletion,
filter: &str,
rpc_timeout: Duration,
) -> Vec<(String, Option<String>)> {
let values = match stage {
McpPromptCompletion::Ready(values) => values,
McpPromptCompletion::PromptNames { server } => list_prompts_blocking(server, rpc_timeout)
.unwrap_or_default()
.into_iter()
.map(|prompt| {
(
sanitize_display_text(&prompt.name),
prompt
.description
.map(|description| sanitize_display_text(&description)),
)
})
.collect(),
McpPromptCompletion::ArgumentKeys {
server,
prompt,
typed_keys,
} => list_prompts_blocking(server, rpc_timeout)
.unwrap_or_default()
.into_iter()
.find(|candidate| candidate.name == prompt)
.and_then(|candidate| candidate.arguments)
.unwrap_or_default()
.into_iter()
.filter(|arg| !typed_keys.contains(&arg.name))
.map(|arg| {
let description = arg
.description
.map(|description| sanitize_display_text(&description));
let description = match (arg.required == Some(true), description) {
(true, Some(description)) => Some(format!("{description} (required)")),
(true, None) => Some("(required)".to_string()),
(false, description) => description,
};
(
format!("{}=", sanitize_display_text(&arg.name)),
description,
)
})
.collect(),
};
fuzzy_filter(values, |(value, _)| value.as_str(), filter)
}
fn list_prompts_blocking(
server: Arc<ConnectedServer>,
rpc_timeout: Duration,
) -> Option<Vec<Prompt>> {
let fut = async move { tokio::time::timeout(rpc_timeout, server.list_all_prompts()).await };
// block_in_place is only sound because the REPL's read_line runs inside the
// main-thread block_on of the multi-thread runtime.
let result = match tokio::runtime::Handle::try_current().ok() {
Some(handle) => tokio::task::block_in_place(|| handle.block_on(fut)),
None => tokio::runtime::Runtime::new().ok()?.block_on(fut),
};
result.ok()?.ok()
}
fn split_line(line: &str) -> Vec<(&str, usize)> { fn split_line(line: &str) -> Vec<(&str, usize)> {
let mut parts = vec![]; let mut parts = vec![];
let mut part_start = None; let mut part_start = None;
@@ -160,3 +265,208 @@ fn test_split_line() {
vec![(".set", 0), ("highlight", 5), ("t", 15)], vec![(".set", 0), ("highlight", 5), ("t", 15)],
); );
} }
#[cfg(test)]
mod prompt_completion_tests {
use super::*;
use crate::config::test_fixtures::{FixtureServer, fixture_runtime};
use std::sync::atomic::Ordering;
fn prompts_fixture() -> FixtureServer {
FixtureServer {
prompts_capability: true,
..Default::default()
}
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stage_two_lists_prompt_names_with_descriptions() {
let (runtime, _server) = fixture_runtime(prompts_fixture()).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::PromptNames { server },
"",
Duration::from_secs(2),
);
assert_eq!(
values,
vec![(
"summarize".to_string(),
Some("Summarize a document".to_string())
)]
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stage_three_suggests_argument_keys_with_required_marker() {
let (runtime, _server) = fixture_runtime(prompts_fixture()).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::ArgumentKeys {
server,
prompt: "summarize".to_string(),
typed_keys: vec![],
},
"",
Duration::from_secs(2),
);
assert_eq!(
values,
vec![
(
"path=".to_string(),
Some("Document path (required)".to_string())
),
("style=".to_string(), None),
]
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stage_three_excludes_typed_keys_and_fuzzy_filters() {
let (runtime, _server) = fixture_runtime(prompts_fixture()).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::ArgumentKeys {
server: Arc::clone(&server),
prompt: "summarize".to_string(),
typed_keys: vec!["path".to_string()],
},
"",
Duration::from_secs(2),
);
assert_eq!(values, vec![("style=".to_string(), None)]);
let values = complete_prompt_stage(
McpPromptCompletion::ArgumentKeys {
server,
prompt: "summarize".to_string(),
typed_keys: vec![],
},
"sty",
Duration::from_secs(2),
);
assert_eq!(values, vec![("style=".to_string(), None)]);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn stage_three_unknown_prompt_is_empty() {
let (runtime, _server) = fixture_runtime(prompts_fixture()).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::ArgumentKeys {
server,
prompt: "ghost".to_string(),
typed_keys: vec![],
},
"",
Duration::from_secs(2),
);
assert!(values.is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn hostile_prompt_strings_are_sanitized_in_suggestions() {
let fixture = FixtureServer {
hostile_prompt: true,
..prompts_fixture()
};
let (runtime, _server) = fixture_runtime(fixture).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::PromptNames {
server: Arc::clone(&server),
},
"evil",
Duration::from_secs(2),
);
assert_eq!(
values,
vec![(
"summarize-evil".to_string(),
Some("Runs hostile text".to_string())
)]
);
let values = complete_prompt_stage(
McpPromptCompletion::ArgumentKeys {
server,
prompt: "sum\u{1b}[31mmarize-evil".to_string(),
typed_keys: vec![],
},
"",
Duration::from_secs(2),
);
assert_eq!(
values,
vec![("path=".to_string(), Some("Doc path (required)".to_string()))]
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn slow_listing_times_out_to_empty() {
let fixture = FixtureServer {
prompt_delay: Some(Duration::from_millis(200)),
..prompts_fixture()
};
let (runtime, _server) = fixture_runtime(fixture).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::PromptNames { server },
"",
Duration::from_millis(20),
);
assert!(values.is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn failed_listing_is_swallowed_without_retry() {
let fixture = FixtureServer {
fail_prompt_listings: true,
..prompts_fixture()
};
let list_prompts_calls = Arc::clone(&fixture.list_prompts_calls);
let (runtime, _server) = fixture_runtime(fixture).await;
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::PromptNames { server },
"",
Duration::from_secs(2),
);
assert!(values.is_empty());
assert_eq!(list_prompts_calls.load(Ordering::SeqCst), 1);
}
#[test]
fn bridge_without_ambient_runtime_uses_fallback_runtime() {
let rt = tokio::runtime::Runtime::new().unwrap();
let (runtime, _server) = rt.block_on(fixture_runtime(prompts_fixture()));
let server = runtime.get("fixture").cloned().unwrap();
let values = complete_prompt_stage(
McpPromptCompletion::PromptNames { server },
"",
Duration::from_secs(2),
);
assert_eq!(
values,
vec![(
"summarize".to_string(),
Some("Summarize a document".to_string())
)]
);
}
}
+445 -36
View File
@@ -12,15 +12,15 @@ use crate::client::{
oauth, oauth,
}; };
use crate::config::{ use crate::config::{
AgentVariables, AppConfig, AssertState, Input, LastMessage, RequestContext, StateFlags, AgentVariables, AppConfig, AssertState, Input, LastMessage, MacroState, RequestContext,
macro_execute, StateFlags, flatten_prompt_messages, macro_execute, resolve_prompt_args, sanitize_display_text,
}; };
use crate::config::{AssetCategory, paths}; use crate::config::{AssetCategory, paths};
use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail}; use crate::function::supervisor::{GuardrailAction, check_pending_agents_guardrail};
use crate::render::render_error; use crate::render::render_error;
use crate::utils::{ use crate::utils::{
AbortSignal, SHELL, abortable_run_with_spinner, create_abort_signal, dimmed_text, run_command, AbortSignal, SHELL, abortable_run_with_spinner, create_abort_signal, dimmed_text,
set_text, temp_file, drain_stale_tty_input, run_command, set_text, temp_file,
}; };
use crate::sandbox::SANDBOX_ENV_FLAG; use crate::sandbox::SANDBOX_ENV_FLAG;
@@ -29,6 +29,7 @@ use anyhow::{Context, Result, bail};
use crossterm::cursor::SetCursorStyle; use crossterm::cursor::SetCursorStyle;
use fancy_regex::Regex; use fancy_regex::Regex;
use indoc::indoc; use indoc::indoc;
use inquire::Text;
use log::warn; use log::warn;
use parking_lot::RwLock; use parking_lot::RwLock;
use reedline::CursorConfig; use reedline::CursorConfig;
@@ -38,6 +39,7 @@ use reedline::{
default_emacs_keybindings, default_vi_insert_keybindings, default_vi_normal_keybindings, default_emacs_keybindings, default_vi_insert_keybindings, default_vi_normal_keybindings,
}; };
use reedline::{MenuBuilder, Signal}; use reedline::{MenuBuilder, Signal};
use std::collections::HashMap;
use std::sync::LazyLock; use std::sync::LazyLock;
use std::{env, process, sync::Arc}; use std::{env, process, sync::Arc};
use tokio::task; use tokio::task;
@@ -53,7 +55,7 @@ pub const DEFAULT_CONTINUATION_PROMPT: &str = indoc! {"
4. Continue with the next pending item now. Call tools immediately." 4. Continue with the next pending item now. Call tools immediately."
}; };
static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| { static REPL_COMMANDS: LazyLock<[ReplCommand; 62]> = LazyLock::new(|| {
[ [
ReplCommand::new(".help", "Show this help guide", AssertState::pass()), ReplCommand::new(".help", "Show this help guide", AssertState::pass()),
ReplCommand::new(".info", "Show system info", AssertState::pass()), ReplCommand::new(".info", "Show system info", AssertState::pass()),
@@ -105,6 +107,11 @@ static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| {
ReplCommand::new(".model", "Switch LLM model", AssertState::pass()), ReplCommand::new(".model", "Switch LLM model", AssertState::pass()),
ReplCommand::new( ReplCommand::new(
".prompt", ".prompt",
"Invoke an MCP prompt and submit the result as chat input",
AssertState::pass(),
),
ReplCommand::new(
".temp-role",
"Set a temporary role using a prompt", "Set a temporary role using a prompt",
AssertState::False(StateFlags::SESSION | StateFlags::AGENT), AssertState::False(StateFlags::SESSION | StateFlags::AGENT),
), ),
@@ -307,7 +314,7 @@ static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| {
), ),
ReplCommand::new( ReplCommand::new(
".list", ".list",
"List roles, sessions, agents, RAGs, macros, skills, tools, or MCP servers", "List roles, sessions, agents, RAGs, macros, skills, prompts, tools, MCP servers, or bundles",
AssertState::pass(), AssertState::pass(),
), ),
ReplCommand::new( ReplCommand::new(
@@ -317,7 +324,12 @@ static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| {
), ),
ReplCommand::new( ReplCommand::new(
".install", ".install",
"Reinstall bundled assets, or install assets from a remote git repo (.install remote <url>)", "Reinstall bundled assets, install a bundle from a git repo, or update an installed bundle",
AssertState::pass(),
),
ReplCommand::new(
".uninstall",
"Uninstall an installed bundle (delete its owned files and MCP entries)",
AssertState::pass(), AssertState::pass(),
), ),
ReplCommand::new( ReplCommand::new(
@@ -411,6 +423,10 @@ Type ".help" for additional help.
} }
} }
// Discard any stray terminal-query reply bytes (e.g. late colorsaurus
// OSC 11 / DA1 responses) so they don't get injected into the prompt.
drain_stale_tty_input();
loop { loop {
if self.abort_signal.aborted_ctrld() { if self.abort_signal.aborted_ctrld() {
break; break;
@@ -765,12 +781,47 @@ pub async fn run_repl_command(
.tool disable <name> # Disable a single tool in the current context"# .tool disable <name> # Disable a single tool in the current context"#
), ),
}, },
".prompt" => match args { ".prompt" => {
let (words, _) = split_args_text(args.unwrap_or_default(), cfg!(windows));
match words.as_slice() {
[server, name, rest @ ..] => {
let provided = parse_prompt_call_args(rest)?;
let prompts = ctx.tool_scope.mcp_runtime.list_prompts(server).await?;
let declared = prompts
.into_iter()
.find(|prompt| prompt.name == *name)
.with_context(|| {
format!("Prompt '{name}' not found on MCP server '{server}'")
})?
.arguments
.unwrap_or_default();
let (mut arguments, missing) = resolve_prompt_args(&declared, provided);
for key in missing {
let value = Text::new(&prompt_arg_inquire_label(server, name, &key))
.prompt()
.with_context(|| {
format!("Failed to read prompt argument '{key}'")
})?;
arguments.insert(key, value);
}
let result = ctx
.tool_scope
.mcp_runtime
.prompt(server, name, arguments)
.await?;
let flattened = flatten_prompt_messages(&result.messages);
let input = Input::from_str(ctx, &flattened, None)?;
ask(ctx, abort_signal.clone(), input, true).await?;
}
_ => println!("Usage: .prompt <server> <name> [key=value ...]"),
}
}
".temp-role" => match args {
Some(text) => { Some(text) => {
let app = Arc::clone(&ctx.app.config); let app = Arc::clone(&ctx.app.config);
ctx.use_prompt(app.as_ref(), text)?; ctx.use_temp_role(app.as_ref(), text)?;
} }
None => println!("Usage: .prompt <text>..."), None => println!("Usage: .temp-role <text>..."),
}, },
".role" => match args { ".role" => match args {
Some(args) => match args.split_once(['\n', ' ']) { Some(args) => match args.split_once(['\n', ' ']) {
@@ -861,27 +912,18 @@ pub async fn run_repl_command(
replay::render(app.as_ref(), &compressed, &active)?; replay::render(app.as_ref(), &compressed, &active)?;
} }
} }
".install" => { ".install" => match parse_repl_install(args) {
let trimmed = args.map(str::trim).unwrap_or(""); ReplInstallDispatch::Builtins(category) => config::install_assets(category)?,
let mut parts = trimmed.splitn(2, char::is_whitespace); ReplInstallDispatch::Unified(value) => {
match parts.next() { config::install_or_update_from_repl_args(value)?;
Some("remote") => {
let rest = parts.next().unwrap_or("").trim();
config::install_remote_from_repl_args(rest)?;
} }
Some(name) if !name.is_empty() => match AssetCategory::parse(name) { ReplInstallDispatch::Help => println!("{}", repl_install_help()),
Some(category) => config::install_assets(category)?, ReplInstallDispatch::Usage => println!(
None => println!( "Usage: .install <{}> | .install <git-url|owner/repo|installed-bundle> \
"Unknown asset category '{name}'. Valid categories: {}", [--git-host <host>] [--filter <cat>] [--force] (see `.install --help`)",
AssetCategory::NAMES.join(", ")
),
},
_ => println!(
"Usage: .install <{}> | .install remote <git-url>",
AssetCategory::NAMES.join("|") AssetCategory::NAMES.join("|")
), ),
} },
}
".update" => { ".update" => {
if ctx.macro_flag { if ctx.macro_flag {
bail!("Cannot perform this operation because you are in a macro") bail!("Cannot perform this operation because you are in a macro")
@@ -1069,15 +1111,44 @@ pub async fn run_repl_command(
} }
}, },
".macro" => match split_first_arg(args) { ".macro" => match split_first_arg(args) {
Some((sub @ ("enable" | "disable"), rest)) => {
match rest.and_then(|v| v.split_whitespace().next()) {
Some(name) => ctx.macro_toggle(name, sub == "enable")?,
None => println!("Usage: .macro {sub} <name>"),
}
}
Some((name, extra)) => { Some((name, extra)) => {
let policy = ctx.macro_policy();
match policy.find(name).map(|row| &row.state) {
Some(state) if state.is_invocable() => {
macro_execute(ctx, name, extra, abort_signal.clone()).await?;
}
Some(MacroState::DisabledRuntime) => bail!(
r#"Macro '{name}' is disabled. Re-enable it with ".macro enable {name}""#
),
Some(MacroState::Locked { level }) => bail!(
"Macro '{name}' is restricted by {} enabled_macros",
ctx.macro_lock_owner(*level)
),
Some(MacroState::Invalid { reason }) => {
bail!("Macro '{name}' is invalid: {reason}")
}
Some(_) | None => {
if extra.is_none() {
let app = Arc::clone(&ctx.app.config); let app = Arc::clone(&ctx.app.config);
if !paths::has_macro(name) && extra.is_none() {
ctx.new_macro(app.as_ref(), name)?; ctx.new_macro(app.as_ref(), name)?;
} else { } else {
macro_execute(ctx, name, extra, abort_signal.clone()).await?; macro_execute(ctx, name, extra, abort_signal.clone()).await?;
} }
} }
None => println!("Usage: .macro <name> <text>..."), }
}
None => println!(
r#"Usage:
.macro <name> [text]... # Execute a macro
.macro enable <name> # Re-enable a runtime-disabled macro
.macro disable <name> # Disable a macro for the rest of this process"#
),
}, },
".file" => match args { ".file" => match args {
Some(args) => { Some(args) => {
@@ -1169,13 +1240,25 @@ pub async fn run_repl_command(
println!("Usage: .delete <role|session|rag|macro|skill|agent-data>") println!("Usage: .delete <role|session|rag|macro|skill|agent-data>")
} }
}, },
".list" => match args { ".uninstall" => match parse_repl_uninstall(args) {
ReplUninstallDispatch::Run(name, assume_yes) => {
config::uninstall_bundle(&name, assume_yes)?
}
ReplUninstallDispatch::Help => println!("{}", repl_uninstall_help()),
ReplUninstallDispatch::Usage => {
println!("Usage: .uninstall <bundle-name> [--yes] (see `.uninstall --help`)")
}
},
".list" => match args.map(str::trim) {
Some("prompts") => {
ctx.list_mcp_prompts().await?;
}
Some(args) => { Some(args) => {
ctx.list_assets(args.trim())?; ctx.list_assets(args)?;
} }
_ => { _ => {
println!( println!(
"Usage: .list <roles|sessions|agents|rags|macros|skills|tools|mcp-servers>" "Usage: .list <roles|sessions|agents|rags|macros|skills|prompts|tools|mcp-servers|bundles>"
) )
} }
}, },
@@ -1290,7 +1373,26 @@ pub async fn run_repl_command(
println!("Usage: .vault <add|get|update|delete|list> [name]") println!("Usage: .vault <add|get|update|delete|list> [name]")
} }
}, },
_ => {
let name = cmd.strip_prefix('.').unwrap_or(cmd);
let policy = ctx.macro_policy();
match policy.find(name).map(|row| &row.state) {
Some(MacroState::Enabled) => {
macro_execute(ctx, name, args, abort_signal.clone()).await?;
}
Some(MacroState::DisabledRuntime) => bail!(
r#"Macro '{name}' is disabled. Re-enable it with ".macro enable {name}""#
),
Some(MacroState::Locked { level }) => bail!(
"Macro '{name}' is restricted by {} enabled_macros",
ctx.macro_lock_owner(*level)
),
Some(MacroState::Invalid { reason }) => {
bail!("Macro '{name}' is invalid: {reason}")
}
_ => unknown_command()?, _ => unknown_command()?,
}
}
}, },
None => { None => {
if let Some(cmd) = try_extract_shell_command(line) { if let Some(cmd) = try_extract_shell_command(line) {
@@ -1518,6 +1620,109 @@ fn unknown_command() -> Result<()> {
bail!(r#"Unknown command. Type ".help" for additional help."#); bail!(r#"Unknown command. Type ".help" for additional help."#);
} }
#[derive(Debug, PartialEq)]
enum ReplInstallDispatch<'a> {
Builtins(AssetCategory),
Unified(&'a str),
Help,
Usage,
}
fn parse_repl_install(args: Option<&str>) -> ReplInstallDispatch<'_> {
let trimmed = args.map(str::trim).unwrap_or("");
if trimmed
.split_whitespace()
.any(|token| token == "--help" || token == "-h")
{
return ReplInstallDispatch::Help;
}
let mut parts = trimmed.splitn(2, char::is_whitespace);
match parts.next() {
Some(name) if !name.is_empty() => {
let rest = parts.next().map(str::trim).unwrap_or("");
match AssetCategory::parse(name) {
Some(category) if rest.is_empty() => ReplInstallDispatch::Builtins(category),
Some(_) => ReplInstallDispatch::Usage,
None => ReplInstallDispatch::Unified(trimmed),
}
}
_ => ReplInstallDispatch::Usage,
}
}
fn repl_install_help() -> String {
format!(
r#"Install built-in assets, install a bundle from a Git source, or update an installed bundle.
Usage:
.install <category> Reinstall built-in assets ({categories})
.install <owner/repo>[#ref] Install a bundle from {default_host} (change the host with --git-host)
.install <git-url>[#ref] Install a bundle from any Git URL, scp-style path, or local path
.install <installed-bundle>[#ref] Update an installed bundle from its recorded source
Flags:
--git-host <host> Host the <owner/repo> shorthand expands against (default {default_host})
--filter <cat> Restrict a remote install to one category ({filters})
--force Overwrite all conflicts without prompting (remote installs only)
Suffix #<ref> to pin a branch, tag, or commit. List installed bundles with
`.list bundles`; remove one with `.uninstall <name>`."#,
categories = AssetCategory::NAMES.join("|"),
default_host = config::DEFAULT_GIT_HOST,
filters = config::InstallFilter::NAMES.join("|"),
)
}
fn repl_uninstall_help() -> String {
r#"Remove an installed bundle: delete the files it owns and the mcp.json entries it added.
Usage:
.uninstall <bundle-name> [--yes]
Flags:
--yes, -y Skip the confirmation prompt
Files you modified after install are prompted for individually and kept by
default; --yes never deletes modified files. List installed bundles with
`.list bundles`."#
.to_string()
}
#[derive(Debug, PartialEq)]
enum ReplUninstallDispatch {
Run(String, bool),
Help,
Usage,
}
fn parse_repl_uninstall(args: Option<&str>) -> ReplUninstallDispatch {
let mut assume_yes = false;
let mut names = Vec::new();
for token in args.unwrap_or("").split_whitespace() {
match token {
"--help" | "-h" => return ReplUninstallDispatch::Help,
"--yes" | "-y" => assume_yes = true,
other if other.starts_with('-') => return ReplUninstallDispatch::Usage,
other => names.push(other),
}
}
match names.as_slice() {
[name] => ReplUninstallDispatch::Run(name.to_string(), assume_yes),
_ => ReplUninstallDispatch::Usage,
}
}
pub fn builtin_command_names() -> Vec<&'static str> {
let mut names: Vec<&'static str> = REPL_COMMANDS
.iter()
.filter_map(|cmd| cmd.name.split_whitespace().next())
.filter_map(|name| name.strip_prefix('.'))
.collect();
names.sort_unstable();
names.dedup();
names
}
fn dump_repl_help() { fn dump_repl_help() {
let head = REPL_COMMANDS let head = REPL_COMMANDS
.iter() .iter()
@@ -1528,6 +1733,10 @@ fn dump_repl_help() {
r###"{head} r###"{head}
{:<24} Run an arbitrary shell command (stdout/stderr stream to your terminal; Ctrl+C interrupts) {:<24} Run an arbitrary shell command (stdout/stderr stream to your terminal; Ctrl+C interrupts)
Custom commands (macros): macros are coyote's custom commands. An enabled
macro <name> runs top-level as .<name> [args...], equivalent to ".macro <name>".
List them with ".list macros"; toggle them with ".macro enable|disable <name>".
Type ::: to start multi-line editing, type ::: to finish it. Type ::: to start multi-line editing, type ::: to finish it.
Press Ctrl+O to open an editor for editing the input buffer. Press Ctrl+O to open an editor for editing the input buffer.
Press Ctrl+C to cancel the response, Ctrl+D to exit the REPL."###, Press Ctrl+C to cancel the response, Ctrl+D to exit the REPL."###,
@@ -1573,6 +1782,37 @@ fn split_first_arg(args: Option<&str>) -> Option<(&str, Option<&str>)> {
}) })
} }
fn parse_prompt_call_args(words: &[String]) -> Result<HashMap<String, String>> {
let mut args = HashMap::new();
for word in words {
let Some((key, value)) = word.split_once('=') else {
bail!("Invalid prompt argument '{word}': arguments must be key=value pairs");
};
args.insert(key.to_string(), unquote_prompt_value(value).to_string());
}
Ok(args)
}
fn unquote_prompt_value(value: &str) -> &str {
let quoted = value.len() >= 2
&& ((value.starts_with('"') && value.ends_with('"'))
|| (value.starts_with('\'') && value.ends_with('\'')));
if quoted {
&value[1..value.len() - 1]
} else {
value
}
}
fn prompt_arg_inquire_label(server: &str, prompt: &str, arg: &str) -> String {
format!(
"Prompt '{}' on '{}' requires '{}':",
sanitize_display_text(prompt),
sanitize_display_text(server),
sanitize_display_text(arg)
)
}
pub fn split_args_text(line: &str, is_win: bool) -> (Vec<String>, &str) { pub fn split_args_text(line: &str, is_win: bool) -> (Vec<String>, &str) {
let mut words = Vec::new(); let mut words = Vec::new();
let mut word = String::new(); let mut word = String::new();
@@ -1724,8 +1964,165 @@ mod tests {
} }
#[test] #[test]
fn repl_commands_has_60_entries() { fn repl_commands_has_62_entries() {
assert_eq!(REPL_COMMANDS.len(), 60); assert_eq!(REPL_COMMANDS.len(), 62);
}
#[test]
fn parse_prompt_call_args_splits_on_first_equals_and_unquotes() {
let words = vec![
"path=notes.txt".to_string(),
r#"style="a b""#.to_string(),
"expr=a=b".to_string(),
];
let args = parse_prompt_call_args(&words).unwrap();
assert_eq!(args["path"], "notes.txt");
assert_eq!(args["style"], "a b");
assert_eq!(args["expr"], "a=b");
}
#[test]
fn parse_prompt_call_args_rejects_words_without_equals() {
let err = parse_prompt_call_args(&["positional".to_string()])
.unwrap_err()
.to_string();
assert_eq!(
err,
"Invalid prompt argument 'positional': arguments must be key=value pairs"
);
}
#[test]
fn prompt_arg_inquire_label_sanitizes_all_components() {
assert_eq!(
prompt_arg_inquire_label("srv", "summarize", "path"),
"Prompt 'summarize' on 'srv' requires 'path':"
);
assert_eq!(
prompt_arg_inquire_label("s\u{1b}[31mrv", "sum\u{1b}]0;x\u{7}marize", "pa\u{7}th"),
"Prompt 'summarize' on 'srv' requires 'pa th':"
);
}
#[test]
fn parse_repl_install_routes_categories_to_builtins() {
assert_eq!(
parse_repl_install(Some("agents")),
ReplInstallDispatch::Builtins(AssetCategory::Agents)
);
}
#[test]
fn parse_repl_install_routes_other_values_to_unified_dispatch() {
assert_eq!(
parse_repl_install(Some("https://github.com/x/y")),
ReplInstallDispatch::Unified("https://github.com/x/y")
);
assert_eq!(
parse_repl_install(Some("my-bundle")),
ReplInstallDispatch::Unified("my-bundle")
);
}
#[test]
fn parse_repl_install_empty_args_ask_for_usage() {
assert_eq!(parse_repl_install(None), ReplInstallDispatch::Usage);
assert_eq!(parse_repl_install(Some(" ")), ReplInstallDispatch::Usage);
}
#[test]
fn parse_repl_install_rejects_extra_tokens_after_a_category() {
assert_eq!(
parse_repl_install(Some("agents --force")),
ReplInstallDispatch::Usage
);
assert_eq!(
parse_repl_install(Some("agents extra")),
ReplInstallDispatch::Usage
);
}
#[test]
fn parse_repl_install_routes_help_from_any_position() {
assert_eq!(
parse_repl_install(Some("--help")),
ReplInstallDispatch::Help
);
assert_eq!(parse_repl_install(Some("-h")), ReplInstallDispatch::Help);
assert_eq!(
parse_repl_install(Some("agents --help")),
ReplInstallDispatch::Help
);
assert_eq!(
parse_repl_install(Some("owner/repo --help")),
ReplInstallDispatch::Help
);
}
#[test]
fn parse_repl_uninstall_routes_run_help_and_usage() {
assert_eq!(
parse_repl_uninstall(Some("my-bundle --yes")),
ReplUninstallDispatch::Run("my-bundle".to_string(), true)
);
assert_eq!(
parse_repl_uninstall(Some("my-bundle")),
ReplUninstallDispatch::Run("my-bundle".to_string(), false)
);
assert_eq!(
parse_repl_uninstall(Some("--help")),
ReplUninstallDispatch::Help
);
assert_eq!(
parse_repl_uninstall(Some("my-bundle -h")),
ReplUninstallDispatch::Help
);
assert_eq!(
parse_repl_uninstall(Some("--force my-bundle")),
ReplUninstallDispatch::Usage
);
assert_eq!(parse_repl_uninstall(None), ReplUninstallDispatch::Usage);
}
#[test]
fn repl_install_and_uninstall_help_text_cover_the_full_surface() {
let install = repl_install_help();
for needle in [
"--git-host",
"--filter",
"--force",
"#ref",
"owner/repo",
".list bundles",
] {
assert!(install.contains(needle), "install help missing {needle}");
}
let uninstall = repl_uninstall_help();
for needle in ["--yes", ".list bundles", "<bundle-name>"] {
assert!(
uninstall.contains(needle),
"uninstall help missing {needle}"
);
}
}
#[test]
fn builtin_command_names_are_sorted_deduped_first_words_without_dots() {
let names = builtin_command_names();
assert!(!names.is_empty());
for name in &names {
assert!(!name.starts_with('.'), "'{name}' should not keep the dot");
assert!(!name.contains(' '), "'{name}' should be a single word");
}
assert!(
names.windows(2).all(|w| w[0] < w[1]),
"names should be sorted and deduplicated: {names:?}"
);
assert!(names.contains(&"help"));
assert!(names.contains(&"macro"));
} }
#[test] #[test]
@@ -1823,10 +2220,22 @@ mod tests {
} }
#[test] #[test]
fn repl_commands_prompt_blocked_in_session_or_agent() { fn repl_commands_prompt_always_available() {
let cmd = REPL_COMMANDS.iter().find(|c| c.name == ".prompt").unwrap(); let cmd = REPL_COMMANDS.iter().find(|c| c.name == ".prompt").unwrap();
assert!(cmd.is_valid(StateFlags::empty())); assert!(cmd.is_valid(StateFlags::empty()));
assert!(cmd.is_valid(StateFlags::ROLE)); assert!(cmd.is_valid(StateFlags::ROLE));
assert!(cmd.is_valid(StateFlags::SESSION));
assert!(cmd.is_valid(StateFlags::AGENT));
}
#[test]
fn repl_commands_temp_role_blocked_in_session_or_agent() {
let cmd = REPL_COMMANDS
.iter()
.find(|c| c.name == ".temp-role")
.unwrap();
assert!(cmd.is_valid(StateFlags::empty()));
assert!(cmd.is_valid(StateFlags::ROLE));
assert!(!cmd.is_valid(StateFlags::SESSION)); assert!(!cmd.is_valid(StateFlags::SESSION));
assert!(!cmd.is_valid(StateFlags::AGENT)); assert!(!cmd.is_valid(StateFlags::AGENT));
} }
+296 -15
View File
@@ -80,6 +80,8 @@ pub(crate) struct CredentialSpec {
pub env_var: String, pub env_var: String,
pub proxy_managed: bool, pub proxy_managed: bool,
pub inject: Vec<InjectRule>, pub inject: Vec<InjectRule>,
pub custom_hosts: Vec<String>,
pub custom_allow_entries: Vec<String>,
pub servers: Vec<String>, pub servers: Vec<String>,
} }
@@ -110,8 +112,13 @@ pub(crate) fn collect_credentials(
} }
let mut by_secret: BTreeMap<String, Aggregate> = BTreeMap::new(); let mut by_secret: BTreeMap<String, Aggregate> = BTreeMap::new();
let mut hosts_by_server: BTreeMap<String, ServerSecretHosts> = BTreeMap::new();
for (server_name, config) in servers { for (server_name, config) in servers {
for occurrence in collect_server_occurrences(config)? { let occurrences = collect_server_occurrences(config)?;
if !occurrences.is_empty() {
hosts_by_server.insert(server_name.clone(), server_secret_hosts(config));
}
for occurrence in occurrences {
let agg = by_secret let agg = by_secret
.entry(occurrence.secret_name) .entry(occurrence.secret_name)
.or_insert_with(|| Aggregate { .or_insert_with(|| Aggregate {
@@ -152,8 +159,9 @@ pub(crate) fn collect_credentials(
eprintln!( eprintln!(
"MCP secrets {} all target the '{header}' header for '{domain}'. The \ "MCP secrets {} all target the '{header}' header for '{domain}'. The \
sandbox proxy cannot tell which one a given request needs, so these \ sandbox proxy cannot tell which one a given request needs, so these \
secrets will be resolved from environment variables inside the \ secrets will be provisioned as placeholder-based custom secrets \
sandbox instead.", instead: each env var holds a unique placeholder that the proxy \
swaps for the real value in outbound request headers.",
quoted_list(secrets) quoted_list(secrets)
); );
slot_conflicted.extend(secrets.iter().cloned()); slot_conflicted.extend(secrets.iter().cloned());
@@ -191,6 +199,21 @@ pub(crate) fn collect_credentials(
} }
let proxy_managed = agg.all_injectable && !slot_conflicted.contains(&secret_name); let proxy_managed = agg.all_injectable && !slot_conflicted.contains(&secret_name);
let (custom_hosts, custom_allow_entries) = if proxy_managed {
(Vec::new(), Vec::new())
} else {
let mut targets: BTreeSet<String> = BTreeSet::new();
let mut allow: BTreeSet<String> = BTreeSet::new();
for hosts in agg
.servers
.iter()
.filter_map(|server| hosts_by_server.get(server))
{
targets.extend(hosts.targets.iter().cloned());
allow.extend(hosts.allow_entries.iter().cloned());
}
(targets.into_iter().collect(), allow.into_iter().collect())
};
credentials.push(CredentialSpec { credentials.push(CredentialSpec {
secret_name, secret_name,
service_id, service_id,
@@ -201,6 +224,8 @@ pub(crate) fn collect_credentials(
} else { } else {
Vec::new() Vec::new()
}, },
custom_hosts,
custom_allow_entries,
servers: agg.servers.into_iter().collect(), servers: agg.servers.into_iter().collect(),
}); });
} }
@@ -365,6 +390,98 @@ fn parse_https_domain(raw: &str) -> Option<String> {
} }
} }
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct ServerSecretHosts {
pub targets: BTreeSet<String>,
pub allow_entries: BTreeSet<String>,
}
pub(crate) fn server_secret_hosts(server: &Value) -> ServerSecretHosts {
let mut hosts = ServerSecretHosts::default();
scrape_hosts_value(server, &mut hosts);
if let Some(host) = server
.get("url")
.and_then(Value::as_str)
.and_then(|raw| Url::parse(raw).ok())
.and_then(|url| url.host_str().map(str::to_string))
.filter(|host| is_usable_host(host))
{
hosts.targets.insert(host);
}
hosts
}
fn scrape_hosts_value(value: &Value, out: &mut ServerSecretHosts) {
match value {
Value::String(s) => {
for url in urls_in_text(s) {
let Some(host) = url.host_str().filter(|h| is_usable_host(h)) else {
continue;
};
out.targets.insert(host.to_string());
if let Some(entry) = allow_entry_for_parsed(&url) {
out.allow_entries.insert(entry);
}
}
}
Value::Object(map) => {
for v in map.values() {
scrape_hosts_value(v, out);
}
}
Value::Array(arr) => {
for v in arr {
scrape_hosts_value(v, out);
}
}
_ => {}
}
}
/// A host usable in the sbx target and allow grammars: non-empty, not a
/// bracketed IPv6 literal (no spelling in either grammar), and not an
/// unresolved `{{placeholder}}` fragment (would register garbage targets
/// and could fail kit validation at create).
fn is_usable_host(host: &str) -> bool {
!host.is_empty() && !host.starts_with('[') && !host.contains('{') && !host.contains('}')
}
/// Every http(s) URL embedded in `text`. Tokens end at whitespace, quotes,
/// or URL-hostile punctuation so comma- or bracket-separated lists don't
/// bleed into one another.
fn urls_in_text(text: &str) -> Vec<Url> {
const TERMINATORS: &[char] = &['"', '\'', ',', ';', '(', ')', '[', ']', '{', '}', '<', '>'];
let mut out = Vec::new();
for (idx, _) in text.match_indices("http") {
let candidate = &text[idx..];
if !candidate.starts_with("http://") && !candidate.starts_with("https://") {
continue;
}
let end = candidate
.find(|c: char| c.is_whitespace() || TERMINATORS.contains(&c))
.unwrap_or(candidate.len());
if let Ok(url) = Url::parse(&candidate[..end]) {
out.push(url);
}
}
out
}
/// Extracts the host of every http(s) URL embedded in `text`, ports stripped.
/// Bracketed IPv6 hosts and placeholder fragments are skipped.
pub(crate) fn hosts_in_text(text: &str) -> BTreeSet<String> {
urls_in_text(text)
.iter()
.filter_map(|url| url.host_str())
.filter(|host| is_usable_host(host))
.map(str::to_string)
.collect()
}
/// Collects a network allow-list entry for every remote MCP server `url` /// Collects a network allow-list entry for every remote MCP server `url`
/// (http/https), so user-configured servers are reachable regardless of how, /// (http/https), so user-configured servers are reachable regardless of how,
/// or whether, their credentials are provisioned. Https on the default port /// or whether, their credentials are provisioned. Https on the default port
@@ -394,13 +511,14 @@ pub(crate) fn allow_entry_for_url(raw: &str) -> Option<String> {
return None; return None;
} }
let host = url.host_str()?; allow_entry_for_parsed(&url)
if host.is_empty() || host.starts_with('[') { }
return None;
} fn allow_entry_for_parsed(url: &Url) -> Option<String> {
let host = url.host_str().filter(|h| is_usable_host(h))?;
match url.port_or_known_default() { match url.port_or_known_default() {
Some(443) if scheme == "https" => Some(host.to_string()), Some(443) if url.scheme() == "https" => Some(host.to_string()),
Some(port) => Some(format!("{host}:{port}")), Some(port) => Some(format!("{host}:{port}")),
None => None, None => None,
} }
@@ -489,12 +607,17 @@ pub(crate) fn render_mixin_document(
serde_yaml::to_string(&mixin).context("Failed to serialize generated sandbox mixin") serde_yaml::to_string(&mixin).context("Failed to serialize generated sandbox mixin")
} }
/// Renders the generated `coyote-mcp` mixin. Only proxy-managed credentials
/// are declared. sbx does not materialize env vars for `proxyManaged: false`
/// mixin credentials, so the rest are provisioned as custom secrets outside
/// the mixin, and only their target hosts join the network allow list here.
pub(crate) fn render_mixin_yaml( pub(crate) fn render_mixin_yaml(
credentials: &[CredentialSpec], credentials: &[CredentialSpec],
server_allow_entries: &[String], server_allow_entries: &[String],
) -> Result<String> { ) -> Result<String> {
let entries = credentials let entries = credentials
.iter() .iter()
.filter(|c| c.proxy_managed)
.map(|c| CredentialEntry { .map(|c| CredentialEntry {
service: c.service_id.clone(), service: c.service_id.clone(),
description: format!( description: format!(
@@ -510,14 +633,20 @@ pub(crate) fn render_mixin_yaml(
}) })
.collect(); .collect();
let mut allow_entries: Vec<String> = server_allow_entries.to_vec();
for credential in credentials.iter().filter(|c| !c.proxy_managed) {
allow_entries.extend(credential.custom_allow_entries.iter().cloned());
}
render_mixin_document( render_mixin_document(
MCP_MIXIN_NAME, MCP_MIXIN_NAME,
"Auto-generated by Coyote at launch: allows network egress to the user's remote MCP \ "Auto-generated by Coyote at launch: allows network egress to the user's remote MCP \
servers and declares their credentials so Docker Sandboxes binds them (bindings are \ servers and declares their credentials so Docker Sandboxes binds them (bindings are \
approved on first interactive run). Values are pre-seeded from Coyote's vault via \ approved on first interactive run). Proxy-injectable values are pre-seeded from \
`sbx secret set`.", Coyote's vault via `sbx secret set`; the remaining secrets are provisioned as \
placeholder-based custom secrets via `sbx secret set-custom`.",
entries, entries,
server_allow_entries, &allow_entries,
) )
} }
@@ -603,6 +732,11 @@ mod tests {
assert_eq!(cred.service_id, "github-pat"); assert_eq!(cred.service_id, "github-pat");
assert_eq!(cred.env_var, "COYOTE_SECRET_GITHUB_PAT"); assert_eq!(cred.env_var, "COYOTE_SECRET_GITHUB_PAT");
assert!(cred.proxy_managed); assert!(cred.proxy_managed);
assert!(
cred.custom_hosts.is_empty(),
"proxy-managed credentials are provisioned via `sbx secret set`, \
not set-custom, so they carry no custom hosts"
);
assert_eq!( assert_eq!(
cred.inject, cred.inject,
vec![InjectRule { vec![InjectRule {
@@ -839,6 +973,12 @@ mod tests {
"secrets sharing an inject domain must both fall back to env" "secrets sharing an inject domain must both fall back to env"
); );
assert!(creds.iter().all(|c| c.inject.is_empty())); assert!(creds.iter().all(|c| c.inject.is_empty()));
assert!(
creds
.iter()
.all(|c| c.custom_hosts == vec!["api.githubcopilot.com".to_string()]),
"demoted secrets must derive their set-custom targets from the server url"
);
} }
#[test] #[test]
@@ -1018,21 +1158,162 @@ mod tests {
} }
#[test] #[test]
fn rendered_mixin_omits_inject_and_permissions_for_env_based_secrets() { fn rendered_mixin_drops_env_based_credentials() {
let servers = servers(json!({ let servers = servers(json!({
"local": { "command": "run", "env": { "KEY": "{{NOTION_TOKEN}}" } } "local": { "command": "run", "env": { "KEY": "{{NOTION_TOKEN}}" } }
})); }));
let creds = collect_credentials(&servers).unwrap(); let creds = collect_credentials(&servers).unwrap();
assert_eq!(creds.len(), 1);
assert!(!creds[0].proxy_managed, "precondition");
assert!(
creds[0].custom_hosts.is_empty(),
"a stdio server with no URLs anywhere yields no derivable hosts"
);
let yaml = render_mixin_yaml(&creds, &[]).unwrap(); let yaml = render_mixin_yaml(&creds, &[]).unwrap();
let value: serde_yaml::Value = serde_yaml::from_str(&yaml).unwrap(); let value: serde_yaml::Value = serde_yaml::from_str(&yaml).unwrap();
let cred = &value["credentials"][0]; assert!(
assert_eq!(cred["apiKey"]["proxyManaged"].as_bool(), Some(false)); value.get("credentials").is_none(),
assert!(cred["apiKey"].get("inject").is_none()); "sbx does not materialize env vars for proxyManaged:false mixin \
credentials; declaring them would be dead config"
);
assert!(value.get("permissions").is_none()); assert!(value.get("permissions").is_none());
} }
#[test]
fn hosts_in_text_extracts_hosts_and_strips_ports() {
assert_eq!(
hosts_in_text("https://api.example.com:8443/v1"),
BTreeSet::from(["api.example.com".to_string()])
);
assert_eq!(
hosts_in_text("see http://a.example.com/x and https://b.example.com/y"),
BTreeSet::from(["a.example.com".to_string(), "b.example.com".to_string()])
);
assert_eq!(
hosts_in_text("https://api.example.com/mcp?key={{KEY}}"),
BTreeSet::from(["api.example.com".to_string()])
);
assert!(hosts_in_text("ws://sock.example.com/mcp").is_empty());
assert!(hosts_in_text("qdrant.example.com:6333").is_empty());
assert!(hosts_in_text("https://[::1]:8443/mcp").is_empty());
assert!(hosts_in_text("httpserver is not a scheme").is_empty());
}
#[test]
fn hosts_in_text_terminates_urls_at_punctuation() {
assert_eq!(
hosts_in_text("https://a.example.com,https://b.example.com;https://c.example.com"),
BTreeSet::from([
"a.example.com".to_string(),
"b.example.com".to_string(),
"c.example.com".to_string()
])
);
assert_eq!(
hosts_in_text("(see https://docs.example.com)"),
BTreeSet::from(["docs.example.com".to_string()])
);
}
#[test]
fn server_secret_hosts_unions_url_host_and_scraped_urls() {
let server = json!({
"command": "run",
"args": ["--endpoint", "https://api.vendor.example/v2"],
"env": { "BASE_URL": "http://internal.example.com:8080/api" }
});
let hosts = server_secret_hosts(&server);
assert_eq!(
hosts.targets,
BTreeSet::from([
"api.vendor.example".to_string(),
"internal.example.com".to_string()
]),
"set-custom targets are port-stripped"
);
assert_eq!(
hosts.allow_entries,
BTreeSet::from([
"api.vendor.example".to_string(),
"internal.example.com:8080".to_string()
]),
"allow entries keep non-default ports so the hosts stay reachable"
);
}
#[test]
fn server_secret_hosts_includes_non_http_url_host() {
let server = json!({ "url": "ws://sock.example.com/mcp" });
let hosts = server_secret_hosts(&server);
assert_eq!(
hosts.targets,
BTreeSet::from(["sock.example.com".to_string()])
);
assert!(
hosts.allow_entries.is_empty(),
"non-http(s) urls have no spelling in the allow grammar"
);
}
#[test]
fn server_secret_hosts_skips_placeholder_hosts() {
let server = json!({
"url": "https://{{TENANT}}.example.com/mcp",
"env": { "API_URL": "https://{{REGION}}.api.example.com/v1" }
});
let hosts = server_secret_hosts(&server);
assert!(
hosts.targets.is_empty() && hosts.allow_entries.is_empty(),
"a host containing an unresolved placeholder must never reach \
set-custom argv or the mixin allow list: {hosts:?}"
);
}
#[test]
fn demoted_credential_hosts_are_unioned_into_allow() {
let servers = servers(json!({
"local": {
"command": "run",
"env": {
"KEY": "{{TOKEN}}",
"API_URL": "https://api.internal.example.com:8443/v1"
}
}
}));
let creds = collect_credentials(&servers).unwrap();
assert_eq!(creds.len(), 1);
assert!(!creds[0].proxy_managed, "precondition");
assert_eq!(
creds[0].custom_hosts,
vec!["api.internal.example.com".to_string()]
);
assert_eq!(
creds[0].custom_allow_entries,
vec!["api.internal.example.com:8443".to_string()],
"allow entries keep the port that set-custom targets must strip"
);
let yaml = render_mixin_yaml(&creds, &[]).unwrap();
let value: serde_yaml::Value = serde_yaml::from_str(&yaml).unwrap();
assert!(value.get("credentials").is_none());
assert_eq!(
value["permissions"]["network"]["allow"][0].as_str(),
Some("api.internal.example.com:8443"),
"a demoted credential's derived hosts must still be reachable"
);
}
#[test] #[test]
fn rendered_mixin_is_deterministic() { fn rendered_mixin_is_deterministic() {
let servers = servers(json!({ let servers = servers(json!({
+807 -64
View File
@@ -2,12 +2,12 @@ use anyhow::{Context, Result, anyhow, bail};
use rust_embed::RustEmbed; use rust_embed::RustEmbed;
use serde_json::Value; use serde_json::Value;
use sha2::{Digest, Sha256}; use sha2::{Digest, Sha256};
use std::collections::HashSet; use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
use std::env;
use std::fs; use std::fs;
use std::io::Write; use std::io::Write;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::{Command, Stdio}; use std::process::{Command, Stdio};
use std::{env, io};
use which::which; use which::which;
pub(crate) mod mcp_credentials; pub(crate) mod mcp_credentials;
@@ -52,25 +52,38 @@ pub fn launch(name: Option<String>, fresh: bool) -> Result<()> {
..AppConfig::default() ..AppConfig::default()
}; };
let vault = Vault::init(&bootstrap)?; let vault = Vault::init(&bootstrap)?;
let registered = sbx_registered_services()?; let registered = sbx_registered_secrets()?;
inject_llm_secret(&config_content, &vault, &registered)?; inject_llm_secret(&config_content, &vault, &registered.services)?;
let mut custom_plans: BTreeMap<String, CustomSecretPlan> = BTreeMap::new();
if !fresh { if !fresh {
inject_rag_secrets(&vault, &registered)?; collect_rag_custom_secrets(&mut custom_plans)?;
} }
let credentials_mixin = if fresh { let credentials_mixin = if fresh {
None None
} else { } else {
inject_mcp_secrets(&vault, &registered)? inject_mcp_secrets(&vault, &registered, &mut custom_plans)?
}; };
let new_custom_envs = provision_custom_secrets(&vault, &registered, custom_plans)?;
let discovered = mixins::discover()?; let discovered = mixins::discover()?;
if sandbox_exists(&name)? { if sandbox_exists(&name)? {
info!("Re-attaching to existing sandbox '{name}'"); info!("Re-attaching to existing sandbox '{name}'");
if !fresh {
warn_if_mixin_drifted(&name, credentials_mixin.as_deref());
if !new_custom_envs.is_empty() {
eprintln!(
"Custom secret env var(s) {} were just registered; restart sandbox \
'{name}' for them to appear in its environment.",
mcp_credentials::quoted_list(&new_custom_envs)
);
}
}
} else { } else {
mixins::log_discovery(&discovered, false); mixins::log_discovery(&discovered, false);
create_sandbox(&name, &kit_path, &discovered, credentials_mixin.as_deref())?; create_sandbox(&name, &kit_path, &discovered, credentials_mixin.as_deref())?;
persist_mixin_hash(&name, credentials_mixin.as_deref());
if !fresh { if !fresh {
copy_host_files(&name)?; copy_host_files(&name)?;
} }
@@ -214,6 +227,50 @@ fn compute_kit_hash() -> Result<String> {
Ok(format!("{:x}", hasher.finalize())) Ok(format!("{:x}", hasher.finalize()))
} }
/// The generated `coyote-mcp` mixin is baked into a sandbox at create time and
/// never re-applied on re-attach, so its hash is persisted per sandbox to
/// detect when the MCP config drifts from the rules the sandbox runs with.
/// An absent mixin hashes as the empty string, keeping the comparison total.
fn credentials_mixin_hash(mixin: Option<&str>) -> String {
let mut hasher = Sha256::new();
hasher.update(mixin.unwrap_or("").as_bytes());
format!("{:x}", hasher.finalize())
}
fn persist_mixin_hash(name: &str, mixin: Option<&str>) {
let path = paths::sandbox_mixin_hash_file(name);
let write = |path: &Path| -> io::Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
fs::write(path, credentials_mixin_hash(mixin))
};
if let Err(e) = write(&path) {
eprintln!(
"Warning: failed to record the sandbox mixin hash at {} ({e}); \
stale-rule detection is disabled for sandbox '{name}'.",
path.display()
);
}
}
fn warn_if_mixin_drifted(name: &str, mixin: Option<&str>) {
let path = paths::sandbox_mixin_hash_file(name);
let Ok(stored) = fs::read_to_string(&path) else {
return;
};
if stored.trim() != credentials_mixin_hash(mixin) {
eprintln!(
"Warning: the MCP config changed since sandbox '{name}' was created; its \
baked-in network and credential rules are stale. Remove and re-create \
the sandbox to apply the new rules: sbx rm {name}"
);
}
}
fn inject_llm_secret( fn inject_llm_secret(
config_content: &str, config_content: &str,
vault: &Vault, vault: &Vault,
@@ -262,7 +319,17 @@ fn inject_llm_secret(
/// and returns the generated schema-v2 `coyote-mcp` mixin (network egress for /// and returns the generated schema-v2 `coyote-mcp` mixin (network egress for
/// every remote MCP server + credential declarations), or `None` when the MCP /// every remote MCP server + credential declarations), or `None` when the MCP
/// config references no remote servers and no secrets. /// config references no remote servers and no secrets.
fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Option<String>> { ///
/// Proxy-managed credentials go through `sbx secret set` and are declared in
/// the mixin. The rest (slot-conflicted or non-header secrets) go through
/// `sbx secret set-custom`; they are only accumulated into `custom_plans`
/// here. `provision_custom_secrets` registers each env var once with the
/// union of targets from every source (MCP and RAG) that needs it.
fn inject_mcp_secrets(
vault: &Vault,
registered: &SbxSecrets,
custom_plans: &mut BTreeMap<String, CustomSecretPlan>,
) -> Result<Option<String>> {
let mcp_path = paths::mcp_config_file(); let mcp_path = paths::mcp_config_file();
if !mcp_path.exists() { if !mcp_path.exists() {
return Ok(None); return Ok(None);
@@ -284,7 +351,8 @@ fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Opt
} }
for credential in &credentials { for credential in &credentials {
if registered.contains(credential.service_id.as_str()) { if credential.proxy_managed {
if registered.services.contains(credential.service_id.as_str()) {
eprintln!( eprintln!(
"Secret for '{}' already registered with sbx. \ "Secret for '{}' already registered with sbx. \
To update it, run: sbx secret set --force {}", To update it, run: sbx secret set --force {}",
@@ -295,17 +363,22 @@ fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Opt
let secret_value = vault let secret_value = vault
.get_secret(&credential.secret_name, false) .get_secret(&credential.secret_name, false)
.with_context(|| { .with_context(|| mcp_secret_missing_hint(credential))?;
format!(
"Secret '{}' referenced by MCP server(s) {} not found \
in vault. Add it with: coyote --add-secret {}",
credential.secret_name,
mcp_credentials::quoted_list(&credential.servers),
credential.secret_name
)
})?;
sbx_secret_set(&credential.service_id, &secret_value)?; sbx_secret_set(&credential.service_id, &secret_value)?;
continue;
}
add_custom_secret_plan(
custom_plans,
&credential.secret_name,
credential.custom_hosts.iter().cloned(),
format!(
"MCP server(s) {}",
mcp_credentials::quoted_list(&credential.servers)
),
true,
);
} }
Ok(Some(mcp_credentials::render_mixin_yaml( Ok(Some(mcp_credentials::render_mixin_yaml(
@@ -314,7 +387,216 @@ fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Opt
)?)) )?))
} }
fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()> { fn mcp_secret_missing_hint(credential: &mcp_credentials::CredentialSpec) -> String {
format!(
"Secret '{}' referenced by MCP server(s) {} not found \
in vault. Add it with: coyote --add-secret {}",
credential.secret_name,
mcp_credentials::quoted_list(&credential.servers),
credential.secret_name
)
}
/// One custom secret to provision, accumulated across every source (RAG
/// driver configs, demoted MCP credentials) before anything is registered,
/// so an env var shared by several sources gets exactly one registration
/// with the union of their target hosts.
#[derive(Debug, PartialEq, Eq)]
struct CustomSecretPlan {
secret_name: String,
hosts: BTreeSet<String>,
/// Human labels ("RAG 'docs'", "MCP server(s) 'kong'") for notices.
sources: BTreeSet<String>,
/// When false a missing vault secret only warns (RAG behavior); any
/// strict source (MCP) upgrades the whole plan to a hard error.
strict: bool,
}
fn add_custom_secret_plan(
plans: &mut BTreeMap<String, CustomSecretPlan>,
secret_name: &str,
hosts: impl IntoIterator<Item = String>,
source: String,
strict: bool,
) {
let plan = plans
.entry(sandbox_secret_env_var(secret_name))
.or_insert_with(|| CustomSecretPlan {
secret_name: secret_name.to_string(),
hosts: BTreeSet::new(),
sources: BTreeSet::new(),
strict: false,
});
plan.hosts.extend(hosts);
plan.sources.insert(source);
plan.strict |= strict;
}
/// Target hosts for a custom secret; falls back to the `'**'` wildcard (match
/// any host) when none could be derived, so the secret is still provisioned.
fn custom_secret_targets(plan: &CustomSecretPlan) -> Vec<String> {
if plan.hosts.is_empty() {
eprintln!(
"Warning: no target host could be derived for secret '{}'; \
registering its sandbox custom secret with the wildcard target '**', \
so the proxy replaces its placeholder in headers sent to ANY host.",
plan.secret_name
);
return vec!["**".to_string()];
}
plan.hosts.iter().cloned().collect()
}
#[derive(Debug, PartialEq, Eq)]
enum CustomSecretAction {
/// No custom secret is registered for this env var yet.
Register {
targets: Vec<String>,
},
Covered,
/// Targets drifted. sbx cannot update targets in place, so the existing
/// registration is removed (by placeholder) and re-registered with the
/// union of old and new targets. A union so that scope widened outside
/// Coyote is never narrowed. Values are re-seeded from the vault, so
/// this is an update, not a deletion.
Replace {
placeholder: String,
targets: Vec<String>,
},
}
fn plan_custom_secret_action(
existing: Option<&CustomSecret>,
wanted: &[String],
) -> CustomSecretAction {
let Some(existing) = existing else {
return CustomSecretAction::Register {
targets: wanted.to_vec(),
};
};
let covered =
existing.targets.contains("**") || wanted.iter().all(|t| existing.targets.contains(t));
if covered {
return CustomSecretAction::Covered;
}
let targets: Vec<String> = existing
.targets
.iter()
.chain(wanted.iter())
.cloned()
.collect::<BTreeSet<_>>()
.into_iter()
.collect();
CustomSecretAction::Replace {
placeholder: existing.placeholder.clone(),
targets,
}
}
/// Registers every accumulated custom secret with sbx and returns the env
/// vars that were newly (re-)registered. Never re-registers on a value
/// change (values are write-once here) only on target drift.
fn provision_custom_secrets(
vault: &Vault,
registered: &SbxSecrets,
plans: BTreeMap<String, CustomSecretPlan>,
) -> Result<Vec<String>> {
let mut new_envs = Vec::new();
for (env_var, plan) in plans {
let targets = custom_secret_targets(&plan);
let sources = plan.sources.iter().cloned().collect::<Vec<_>>().join(", ");
eprintln!(
"Secret '{}' (used by {sources}) resolves to a proxy placeholder inside \
the sandbox (env var {env_var}); the real value is only injected into \
HTTP(S) request headers sent to: {}.",
plan.secret_name,
targets.join(", ")
);
let action = plan_custom_secret_action(registered.custom.get(&env_var), &targets);
if let CustomSecretAction::Covered = action {
eprintln!("Custom secret '{env_var}' already registered with sbx.");
let existing = &registered.custom[&env_var];
if existing.targets.contains("**") && targets != ["**"] {
eprintln!(
"Note: the existing registration targets the wildcard '**', wider \
than the derived host(s) {}. To re-scope it, remove it with \
`sbx secret rm --placeholder {} -f` and re-launch.",
mcp_credentials::quoted_list(&targets),
existing.placeholder
);
}
continue;
}
// Resolve the value BEFORE any removal so a missing vault secret
// never destroys an existing registration.
let secret_value = match vault.get_secret(&plan.secret_name, false) {
Ok(value) => value,
Err(e) if !plan.strict => {
eprintln!(
"Warning: could not load secret '{}' (used by {sources}): {e}. \
Requests that need it will fail inside the sandbox. \
Run `coyote --add-secret {}` to fix.",
plan.secret_name, plan.secret_name
);
continue;
}
Err(e) => {
return Err(e).with_context(|| {
format!(
"Secret '{}' (used by {sources}) not found in vault. \
Add it with: coyote --add-secret {}",
plan.secret_name, plan.secret_name
)
});
}
};
let (targets, replaced) = match action {
CustomSecretAction::Register { targets } => (targets, false),
CustomSecretAction::Replace {
placeholder,
targets,
} => {
eprintln!(
"Updating the sbx custom secret for '{env_var}' to cover target \
host(s) {}.",
mcp_credentials::quoted_list(&targets)
);
if !sbx_secret_rm_custom(&placeholder)? {
continue;
}
(targets, true)
}
CustomSecretAction::Covered => unreachable!("handled above"),
};
if sbx_secret_set_custom(&env_var, &targets, &secret_value)? {
new_envs.push(env_var);
} else if replaced {
eprintln!(
"Warning: the old registration for '{env_var}' was removed but \
re-registering it failed; it will be re-registered from the vault \
on the next launch."
);
}
}
Ok(new_envs)
}
/// Accumulates every attached RAG's driver_config secrets into `custom_plans`
/// (bound to `COYOTE_SECRET_<NAME>`, the env var `interpolate_secrets`
/// resolves inside the sandbox). Non-strict: a missing vault secret warns at
/// provisioning time instead of failing the launch, meaning only that RAG's
/// queries would fail.
fn collect_rag_custom_secrets(custom_plans: &mut BTreeMap<String, CustomSecretPlan>) -> Result<()> {
let rags_dir = paths::rags_dir(); let rags_dir = paths::rags_dir();
if !rags_dir.exists() { if !rags_dir.exists() {
return Ok(()); return Ok(());
@@ -338,20 +620,19 @@ fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()>
continue; continue;
} }
let secret_names = driver_config_secret_names(&data); let secret_names = driver_config_secret_names(&data);
let Some((primary, extra)) = secret_names.split_first() else { if secret_names.is_empty() {
continue; continue;
};
let service_id = mcp_credentials::secret_service_id(&stem);
if !service_id.is_empty() && !registered.contains(&service_id) {
bind_rag_secret(vault, &service_id, primary, &stem)?;
} }
for name in extra { let hosts = rag_driver_hosts(&data);
let id = mcp_credentials::secret_service_id(name); for secret_name in &secret_names {
if !id.is_empty() && !registered.contains(&id) { add_custom_secret_plan(
bind_rag_secret(vault, &id, name, &stem)?; custom_plans,
} secret_name,
hosts.iter().cloned(),
format!("RAG '{stem}'"),
false,
);
} }
} }
@@ -378,21 +659,46 @@ fn driver_config_secret_names(data: &RagData) -> Vec<String> {
names names
} }
fn bind_rag_secret(vault: &Vault, service_id: &str, secret_name: &str, stem: &str) -> Result<()> { /// Derives custom-secret target hosts from a RAG's driver_config: any http(s)
match vault.get_secret(secret_name, false) { /// URL in a value contributes its host, and the `host`/`url` keys also accept
Ok(secret_value) => { /// a bare `host[:port]` value (e.g. `qdrant.example.com:6333`). Ports are
sbx_secret_set(service_id, &secret_value) /// stripped (sbx custom-secret targets are host-only).
.context("Failed to register RAG secret with sbx")?; fn rag_driver_hosts(data: &RagData) -> Vec<String> {
} let mut hosts: BTreeSet<String> = BTreeSet::new();
Err(e) => { for (key, value) in &data.driver_config {
eprintln!( let trimmed = value.trim();
"Warning: could not load secret '{secret_name}' for RAG '{stem}': {e}. \ hosts.extend(mcp_credentials::hosts_in_text(trimmed));
Queries to this RAG will fail inside the sandbox. \ if (key == "host" || key == "url")
Run `coyote --add-secret {secret_name}` to fix." && let Some(host) = bare_host(trimmed)
); {
hosts.insert(host);
} }
} }
Ok(())
hosts.into_iter().collect()
}
fn bare_host(value: &str) -> Option<String> {
if value.is_empty()
|| value.contains("://")
|| value.contains("{{")
|| value.contains('/')
|| value.contains(char::is_whitespace)
{
return None;
}
let host = match value.rsplit_once(':') {
Some((host, port)) if !port.is_empty() && port.bytes().all(|b| b.is_ascii_digit()) => host,
Some(_) => value,
None => value,
};
if host.is_empty() || host.contains(':') || host.starts_with('[') {
return None;
}
Some(host.to_string())
} }
fn provider_to_sbx_service(provider_type: &str, client_name: Option<&str>) -> String { fn provider_to_sbx_service(provider_type: &str, client_name: Option<&str>) -> String {
@@ -405,29 +711,112 @@ fn provider_to_sbx_service(provider_type: &str, client_name: Option<&str>) -> St
} }
} }
fn sbx_registered_services() -> Result<HashSet<String>> { #[derive(Debug, Default)]
let (success, stdout, _) = run_command_with_output(SBX_BINARY, &["secret", "ls"], None) struct SbxSecrets {
services: HashSet<String>,
custom: HashMap<String, CustomSecret>,
}
#[derive(Debug)]
struct CustomSecret {
targets: BTreeSet<String>,
placeholder: String,
}
fn sbx_registered_secrets() -> Result<SbxSecrets> {
let (success, stdout, stderr) = run_command_with_output(SBX_BINARY, &["secret", "ls"], None)
.context("Failed to run `sbx secret ls`")?; .context("Failed to run `sbx secret ls`")?;
if !success { if !success {
return Ok(HashSet::new()); eprintln!(
"Warning: `sbx secret ls` failed ({}); Coyote cannot tell which secrets \
are already registered and may attempt to re-register existing ones.",
stderr.trim()
);
return Ok(SbxSecrets::default());
} }
Ok(stdout Ok(parse_sbx_secret_ls(&stdout))
.lines() }
.skip(1)
.filter_map(|line| { fn parse_sbx_secret_ls(stdout: &str) -> SbxSecrets {
let mut parts = line.split_whitespace(); let mut secrets = SbxSecrets::default();
let scope = parts.next()?; let mut in_custom = false;
let _kind = parts.next()?; let mut in_header = true;
let name = parts.next()?; let mut custom_body_lines = 0usize;
if scope == "(global)" { let mut custom_rows = 0usize;
Some(name.to_string()) for line in stdout.lines() {
} else { let trimmed = line.trim();
None if trimmed.is_empty() {
continue;
} }
}) if trimmed == "CUSTOM SECRETS" {
.collect()) in_custom = true;
in_header = true;
continue;
}
if in_header {
in_header = false;
continue;
}
if in_custom {
custom_body_lines += 1;
let cols = split_columns(line);
let [scope, targets, env, placeholder, ..] = cols.as_slice() else {
continue;
};
custom_rows += 1;
if *scope != "(global)" {
continue;
}
secrets.custom.insert(
(*env).to_string(),
CustomSecret {
targets: targets.split(',').map(|t| t.trim().to_string()).collect(),
placeholder: (*placeholder).to_string(),
},
);
} else {
let mut parts = line.split_whitespace();
let (Some(scope), Some(_kind), Some(name)) = (parts.next(), parts.next(), parts.next())
else {
continue;
};
if scope == "(global)" {
secrets.services.insert(name.to_string());
}
}
}
if custom_body_lines > 0 && custom_rows == 0 {
eprintln!(
"Warning: no rows could be parsed from the CUSTOM SECRETS section of \
`sbx secret ls`; its output format may have changed. Custom-secret \
idempotency checks are disabled for this launch."
);
}
secrets
}
fn split_columns(line: &str) -> Vec<&str> {
let mut out = Vec::new();
let mut rest = line.trim();
while !rest.is_empty() {
match rest.find(" ") {
Some(idx) => {
out.push(&rest[..idx]);
rest = rest[idx..].trim_start();
}
None => {
out.push(rest);
break;
}
}
}
out
} }
fn sbx_secret_set(service: &str, secret_value: &str) -> Result<()> { fn sbx_secret_set(service: &str, secret_value: &str) -> Result<()> {
@@ -439,10 +828,11 @@ fn sbx_secret_set(service: &str, secret_value: &str) -> Result<()> {
.spawn() .spawn()
.context("Failed to spawn `sbx secret set`")?; .context("Failed to spawn `sbx secret set`")?;
if let Some(mut stdin_handle) = child.stdin.take() { if let Some(mut stdin_handle) = child.stdin.take()
stdin_handle && let Err(e) = stdin_handle.write_all(secret_value.as_bytes())
.write_all(secret_value.as_bytes()) && e.kind() != io::ErrorKind::BrokenPipe
.context("Failed to write secret to `sbx secret set` stdin")?; {
return Err(anyhow!(e).context("Failed to write secret to `sbx secret set` stdin"));
} }
let status = child let status = child
@@ -453,13 +843,79 @@ fn sbx_secret_set(service: &str, secret_value: &str) -> Result<()> {
eprintln!( eprintln!(
"Warning: failed to register sbx secret '{service}' \ "Warning: failed to register sbx secret '{service}' \
(`sbx secret set {service}` exited with {status}). \ (`sbx secret set {service}` exited with {status}). \
Set it manually with: echo '<value>' | sbx secret set {service}" Set it manually with: sbx secret set {service} \
(the value is read from the prompt)"
); );
} }
Ok(()) Ok(())
} }
fn sbx_secret_set_custom(env_var: &str, targets: &[String], secret_value: &str) -> Result<bool> {
let mut args: Vec<&str> = vec!["secret", "set-custom", "--env", env_var];
for target in targets {
args.push("--host");
args.push(target);
}
debug!(
"sbx secret set-custom --env {env_var} (targets: {})",
targets.join(", ")
);
let mut child = Command::new(SBX_BINARY)
.args(&args)
.stdin(Stdio::piped())
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.spawn()
.context("Failed to spawn `sbx secret set-custom`")?;
if let Some(mut stdin_handle) = child.stdin.take()
&& let Err(e) = stdin_handle.write_all(secret_value.as_bytes())
&& e.kind() != io::ErrorKind::BrokenPipe
{
return Err(anyhow!(e).context("Failed to write secret to `sbx secret set-custom` stdin"));
}
let status = child
.wait()
.context("Failed to wait for `sbx secret set-custom`")?;
if !status.success() {
let host_flags: String = targets.iter().map(|t| format!(" --host '{t}'")).collect();
eprintln!(
"Warning: failed to register sbx custom secret '{env_var}' \
(`sbx secret set-custom` exited with {status}). Set it manually with: \
sbx secret set-custom --env {env_var}{host_flags} \
(the value is read from the prompt)"
);
}
Ok(status.success())
}
fn sbx_secret_rm_custom(placeholder: &str) -> Result<bool> {
debug!("sbx secret rm --placeholder {placeholder} -f");
let status = Command::new(SBX_BINARY)
.args(["secret", "rm", "--placeholder", placeholder, "-f"])
.stdin(Stdio::inherit())
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.status()
.context("Failed to spawn `sbx secret rm`")?;
if !status.success() {
eprintln!(
"Warning: failed to remove the outdated sbx custom secret \
(`sbx secret rm --placeholder {placeholder} -f` exited with {status}); \
its targets were left unchanged."
);
}
Ok(status.success())
}
fn sandbox_exists(name: &str) -> Result<bool> { fn sandbox_exists(name: &str) -> Result<bool> {
let (success, stdout, stderr) = let (success, stdout, stderr) =
run_command_with_output(SBX_BINARY, &["ls"], None).context("Failed to run `sbx ls`")?; run_command_with_output(SBX_BINARY, &["ls"], None).context("Failed to run `sbx ls`")?;
@@ -732,6 +1188,293 @@ mod tests {
"order follows driver_config, and a repeat is not registered twice" "order follows driver_config, and a repeat is not registered twice"
); );
} }
/// Pinned to the `sbx secret ls` output shape of sbx v0.38.0.
const SECRET_LS_SAMPLE: &str = "\
SCOPE TYPE NAME SECRET
(global) service github (stored)
(global) service kong-prod-pat (stored)
(global) service anthropic (oauth configured)
CUSTOM SECRETS
SCOPE TARGETS ENV PLACEHOLDER SECRET
(global) api.stripe.com STRIPE_API_KEY sbx-cs-97BPiO11AS93Tlo5 rk_liv******...******JT6n
";
#[test]
fn parse_sbx_secret_ls_reads_both_sections() {
let secrets = parse_sbx_secret_ls(SECRET_LS_SAMPLE);
assert_eq!(
secrets.services,
HashSet::from([
"github".to_string(),
"kong-prod-pat".to_string(),
"anthropic".to_string()
])
);
let custom = secrets.custom.get("STRIPE_API_KEY").unwrap();
assert_eq!(
custom.targets,
BTreeSet::from(["api.stripe.com".to_string()])
);
assert_eq!(custom.placeholder, "sbx-cs-97BPiO11AS93Tlo5");
}
#[test]
fn parse_sbx_secret_ls_splits_comma_separated_targets() {
// Pinned to a live capture: multiple --host targets render as one
// comma+space-separated TARGETS field, columns padded to 2+ spaces.
let output = "\
SCOPE TYPE NAME SECRET
(global) service github (stored)
CUSTOM SECRETS
SCOPE TARGETS ENV PLACEHOLDER SECRET
(global) probe.invalid, other.probe.invalid, a-quite-long-hostname.subdomain.probe.invalid COYOTE_TEST_PROBE sbx-cs-TdaC76ZA3MYAfNAt probe-*******
";
let secrets = parse_sbx_secret_ls(output);
let custom = secrets.custom.get("COYOTE_TEST_PROBE").unwrap();
assert_eq!(
custom.targets,
BTreeSet::from([
"probe.invalid".to_string(),
"other.probe.invalid".to_string(),
"a-quite-long-hostname.subdomain.probe.invalid".to_string()
])
);
assert_eq!(custom.placeholder, "sbx-cs-TdaC76ZA3MYAfNAt");
}
#[test]
fn parse_sbx_secret_ls_without_custom_section() {
let output = "\
SCOPE TYPE NAME SECRET
(global) service github (stored)
";
let secrets = parse_sbx_secret_ls(output);
assert_eq!(secrets.services, HashSet::from(["github".to_string()]));
assert!(secrets.custom.is_empty());
}
#[test]
fn parse_sbx_secret_ls_ignores_non_global_rows() {
let output = "\
SCOPE TYPE NAME SECRET
my-box service github (stored)
CUSTOM SECRETS
SCOPE TARGETS ENV PLACEHOLDER SECRET
my-box api.stripe.com API_KEY sbx-cs-abc12 sk-***
";
let secrets = parse_sbx_secret_ls(output);
assert!(secrets.services.is_empty());
assert!(secrets.custom.is_empty());
}
#[test]
fn parse_sbx_secret_ls_handles_empty_output() {
let secrets = parse_sbx_secret_ls("");
assert!(secrets.services.is_empty());
assert!(secrets.custom.is_empty());
}
fn plan_for(secret_name: &str, hosts: &[&str]) -> CustomSecretPlan {
CustomSecretPlan {
secret_name: secret_name.to_string(),
hosts: hosts.iter().map(|h| h.to_string()).collect(),
sources: BTreeSet::from(["test".to_string()]),
strict: true,
}
}
fn strs(items: &[&str]) -> Vec<String> {
items.iter().map(|s| s.to_string()).collect()
}
#[test]
fn custom_secret_targets_falls_back_to_wildcard() {
assert_eq!(
custom_secret_targets(&plan_for("KEY", &[])),
vec!["**".to_string()]
);
assert_eq!(
custom_secret_targets(&plan_for("KEY", &["api.example.com"])),
vec!["api.example.com".to_string()]
);
}
#[test]
fn plan_action_registers_when_no_secret_exists() {
assert_eq!(
plan_custom_secret_action(None, &strs(&["api.example.com"])),
CustomSecretAction::Register {
targets: strs(&["api.example.com"])
}
);
}
#[test]
fn plan_action_skips_a_covering_registration() {
let existing = CustomSecret {
targets: BTreeSet::from(["a.example.com".to_string(), "b.example.com".to_string()]),
placeholder: "sbx-cs-x".to_string(),
};
assert_eq!(
plan_custom_secret_action(Some(&existing), &strs(&["a.example.com"])),
CustomSecretAction::Covered
);
assert_eq!(
plan_custom_secret_action(Some(&existing), &strs(&["a.example.com", "b.example.com"])),
CustomSecretAction::Covered
);
}
#[test]
fn plan_action_treats_wildcard_as_covering_everything() {
let existing = CustomSecret {
targets: BTreeSet::from(["**".to_string()]),
placeholder: "sbx-cs-x".to_string(),
};
assert_eq!(
plan_custom_secret_action(Some(&existing), &strs(&["any.example.com"])),
CustomSecretAction::Covered,
"a wildcard registration is never narrowed, only noted"
);
}
#[test]
fn plan_action_replaces_on_target_drift_with_the_union() {
let existing = CustomSecret {
targets: BTreeSet::from(["a.example.com".to_string()]),
placeholder: "sbx-cs-x".to_string(),
};
assert_eq!(
plan_custom_secret_action(Some(&existing), &strs(&["b.example.com"])),
CustomSecretAction::Replace {
placeholder: "sbx-cs-x".to_string(),
targets: strs(&["a.example.com", "b.example.com"]),
},
"drift removes the old registration (by placeholder) and re-registers \
with the union so scope widened outside Coyote is never narrowed"
);
}
#[test]
fn shared_rag_and_mcp_secret_accumulates_one_plan_with_unioned_hosts() {
let mut plans: BTreeMap<String, CustomSecretPlan> = BTreeMap::new();
add_custom_secret_plan(
&mut plans,
"OPENAI_KEY",
strs(&["qdrant.example.com"]),
"RAG 'docs'".to_string(),
false,
);
add_custom_secret_plan(
&mut plans,
"OPENAI_KEY",
strs(&["api.openai.com"]),
"MCP server(s) 'assistant'".to_string(),
true,
);
assert_eq!(plans.len(), 1, "one env var must yield one registration");
let plan = &plans["COYOTE_SECRET_OPENAI_KEY"];
assert_eq!(
plan.hosts,
BTreeSet::from([
"api.openai.com".to_string(),
"qdrant.example.com".to_string()
]),
"targets from every source are unioned, not last-writer-wins"
);
assert_eq!(
plan.sources,
BTreeSet::from([
"MCP server(s) 'assistant'".to_string(),
"RAG 'docs'".to_string()
])
);
assert!(
plan.strict,
"any strict source upgrades the whole plan to a hard error on a missing secret"
);
}
#[test]
fn rag_driver_hosts_strips_port_from_bare_host() {
let data = rag_with(&[("host", "qdrant.example.com:6333"), ("collection", "docs")]);
assert_eq!(rag_driver_hosts(&data), vec!["qdrant.example.com"]);
}
#[test]
fn rag_driver_hosts_scrapes_urls_and_bare_url_key() {
let data = rag_with(&[
("url", "https://qdrant.example.com:6333"),
("proxy", "endpoint http://edge.example.com/v1"),
]);
assert_eq!(
rag_driver_hosts(&data),
vec!["edge.example.com", "qdrant.example.com"]
);
}
#[test]
fn rag_driver_hosts_ignores_placeholders_and_non_host_values() {
let data = rag_with(&[
("host", "{{QDRANT_HOST}}"),
("api_key", "{{QDRANT_KEY}}"),
("collection", "docs"),
]);
assert!(rag_driver_hosts(&data).is_empty());
}
#[test]
fn bare_host_accepts_host_and_host_port_only() {
assert_eq!(
bare_host("qdrant.example.com"),
Some("qdrant.example.com".to_string())
);
assert_eq!(bare_host("localhost:6333"), Some("localhost".to_string()));
assert_eq!(bare_host("127.0.0.1:6333"), Some("127.0.0.1".to_string()));
assert_eq!(bare_host("https://a.example.com"), None);
assert_eq!(bare_host("{{HOST}}"), None);
assert_eq!(bare_host("host/path"), None);
assert_eq!(bare_host("two words"), None);
assert_eq!(bare_host("::1"), None);
assert_eq!(bare_host("[::1]:6333"), None);
assert_eq!(bare_host(""), None);
}
#[test]
fn credentials_mixin_hash_distinguishes_content_and_absence() {
assert_eq!(
credentials_mixin_hash(Some("kind: mixin\n")),
credentials_mixin_hash(Some("kind: mixin\n"))
);
assert_eq!(
credentials_mixin_hash(None),
credentials_mixin_hash(Some(""))
);
assert_ne!(
credentials_mixin_hash(None),
credentials_mixin_hash(Some("kind: mixin\n"))
);
}
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{SystemTime, UNIX_EPOCH};
#[test] #[test]
+29 -1
View File
@@ -35,6 +35,7 @@ use nu_ansi_term::Color;
use serde_json::Value; use serde_json::Value;
use std::borrow::Cow; use std::borrow::Cow;
use std::collections::VecDeque; use std::collections::VecDeque;
use std::io;
use std::sync::atomic::AtomicBool; use std::sync::atomic::AtomicBool;
use std::sync::{LazyLock, Mutex, OnceLock}; use std::sync::{LazyLock, Mutex, OnceLock};
use std::{cmp, env, path::PathBuf, process}; use std::{cmp, env, path::PathBuf, process};
@@ -45,7 +46,7 @@ pub static CODE_BLOCK_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?ms)```\w*(.*)```").unwrap()); LazyLock::new(|| Regex::new(r"(?ms)```\w*(.*)```").unwrap());
pub static THINK_TAG_RE: LazyLock<Regex> = pub static THINK_TAG_RE: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"(?s)^\s*<think>.*?</think>(\s*|$)").unwrap()); LazyLock::new(|| Regex::new(r"(?s)^\s*<think>.*?</think>(\s*|$)").unwrap());
pub static IS_STDOUT_TERMINAL: LazyLock<bool> = LazyLock::new(|| std::io::stdout().is_terminal()); pub static IS_STDOUT_TERMINAL: LazyLock<bool> = LazyLock::new(|| io::stdout().is_terminal());
pub static HEADLESS: AtomicBool = AtomicBool::new(false); pub static HEADLESS: AtomicBool = AtomicBool::new(false);
pub static ACP_SERVER: AtomicBool = AtomicBool::new(false); pub static ACP_SERVER: AtomicBool = AtomicBool::new(false);
@@ -133,6 +134,33 @@ pub fn parse_bool(value: &str) -> Option<bool> {
} }
} }
pub fn drain_stale_tty_input() {
use crossterm::event::{poll, read};
use std::time::{Duration, Instant};
if !io::stdin().is_terminal() {
return;
}
if crossterm::terminal::enable_raw_mode().is_err() {
return;
}
let deadline = Instant::now() + Duration::from_millis(100);
while Instant::now() < deadline {
match poll(Duration::from_millis(10)) {
Ok(true) => {
if read().is_err() {
break;
}
}
_ => break,
}
}
let _ = crossterm::terminal::disable_raw_mode();
}
pub fn estimate_token_length(text: &str) -> usize { pub fn estimate_token_length(text: &str) -> usize {
let weighted: usize = text.chars().map(|c| if c.is_ascii() { 1 } else { 2 }).sum(); let weighted: usize = text.chars().map(|c| if c.is_ascii() { 1 } else { 2 }).sum();
weighted.div_ceil(4) weighted.div_ceil(4)
-14
View File
@@ -54,20 +54,6 @@ pub async fn expand_glob_paths<T: AsRef<str>>(
Ok(new_paths) Ok(new_paths)
} }
pub fn clear_dir(dir: &Path) -> Result<()> {
for entry in fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
if path.is_dir() {
fs::remove_dir_all(&path)?;
} else {
fs::remove_file(&path)?;
}
}
Ok(())
}
pub fn list_file_names<T: AsRef<Path>>(dir: T, ext: &str) -> Vec<String> { pub fn list_file_names<T: AsRef<Path>>(dir: T, ext: &str) -> Vec<String> {
match fs::read_dir(dir.as_ref()) { match fs::read_dir(dir.as_ref()) {
Ok(rd) => { Ok(rd) => {
+3
View File
@@ -12,6 +12,7 @@ pub use utils::prompt_provider_choice;
use crate::cli::Cli; use crate::cli::Cli;
use crate::config::AppConfig; use crate::config::AppConfig;
use crate::utils::drain_stale_tty_input;
use crate::vault::utils::ensure_password_file_initialized; use crate::vault::utils::ensure_password_file_initialized;
use anyhow::{Context, Result, anyhow, bail}; use anyhow::{Context, Result, anyhow, bail};
use fancy_regex::Regex; use fancy_regex::Regex;
@@ -151,6 +152,7 @@ impl Vault {
"Vault management is disabled in sandbox mode. Use `coyote --add-secret` on your host." "Vault management is disabled in sandbox mode. Use `coyote --add-secret` on your host."
); );
} }
drain_stale_tty_input();
let secret_value = Password::new("Enter the secret value:") let secret_value = Password::new("Enter the secret value:")
.with_validator(required!()) .with_validator(required!())
.with_display_mode(PasswordDisplayMode::Masked) .with_display_mode(PasswordDisplayMode::Masked)
@@ -190,6 +192,7 @@ impl Vault {
"Vault management is disabled in sandbox mode. Use `coyote --add-secret` on your host." "Vault management is disabled in sandbox mode. Use `coyote --add-secret` on your host."
); );
} }
drain_stale_tty_input();
let secret_value = Password::new("Enter the secret value:") let secret_value = Password::new("Enter the secret value:")
.with_validator(required!()) .with_validator(required!())
.with_display_mode(PasswordDisplayMode::Masked) .with_display_mode(PasswordDisplayMode::Masked)
+4
View File
@@ -1,5 +1,6 @@
use crate::config::ensure_parent_exists; use crate::config::ensure_parent_exists;
use crate::sandbox::{SANDBOX_ENV_FLAG, sandbox_secret_env_var}; use crate::sandbox::{SANDBOX_ENV_FLAG, sandbox_secret_env_var};
use crate::utils::drain_stale_tty_input;
use crate::vault::{SECRET_RE, Vault}; use crate::vault::{SECRET_RE, Vault};
use anyhow::Result; use anyhow::Result;
use anyhow::anyhow; use anyhow::anyhow;
@@ -68,6 +69,7 @@ pub fn create_vault_password_file(vault: &mut Vault) -> Result<()> {
} }
} }
drain_stale_tty_input();
let ans = Confirm::new( let ans = Confirm::new(
format!( format!(
"The configured password file '{}' is empty. Create a password?", "The configured password file '{}' is empty. Create a password?",
@@ -107,6 +109,7 @@ pub fn create_vault_password_file(vault: &mut Vault) -> Result<()> {
} }
} }
} else { } else {
drain_stale_tty_input();
let ans = Confirm::new("No password file configured. Do you want to create one now?") let ans = Confirm::new("No password file configured. Do you want to create one now?")
.with_default(true) .with_default(true)
.prompt()?; .prompt()?;
@@ -185,6 +188,7 @@ pub fn create_vault_password_file(vault: &mut Vault) -> Result<()> {
} }
pub fn prompt_provider_choice() -> Result<Option<SupportedProvider>> { pub fn prompt_provider_choice() -> Result<Option<SupportedProvider>> {
drain_stale_tty_input();
let choices = vec![ let choices = vec![
"local - encrypted file on this machine", "local - encrypted file on this machine",
"aws_secrets_manager - AWS Secrets Manager", "aws_secrets_manager - AWS Secrets Manager",