Compare commits

...
Author SHA1 Message Date
Dark-Alex-17 ae96a7e031 fmt: applied formatting 2026-08-12 19:35:27 -06:00
Dark-Alex-17 2658ca776e feat: Installed duckdb into the coyote image
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 19:29:26 -06:00
Dark-Alex-17 f8682102a0 docs: Added duckdb prerequisite
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 19:07:31 -06:00
Dark-Alex-17 3fa0f5c428 feat: Support managing MCP servers from the CLI directly
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 17:55:33 -06:00
Dark-Alex-17 68135b97d1 feat: append new built-in rag__query function to RAG contexts to allow further querying by LLMs
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 16:25:50 -06:00
Dark-Alex-17 b87a3460c4 fix: detect duplicate tool call IDs client-side before sending to Claude
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 13:33:24 -06:00
Dark-Alex-17 cb23da6490 feat: support static file bundling with sbx-mixins
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 13:18:38 -06:00
Alex Clarke 2a40a5a81d Merge pull request #14 from Dark-Alex-17/feat/rag-driver-abstraction-v3
CI / All (ubuntu-latest) (push) Failing after 31s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
feat(rag): pluggable RagProvider abstraction with DuckDB and Qdrant drivers
2026-08-12 12:45:12 -06:00
Dark-Alex-17 ebba976a27 fmt: applied formatting 2026-08-12 12:07:38 -06:00
Dark-Alex-17 c84f9522e9 feat(rag): offer the storage driver when an agent initializes its RAG
Agent startup and graph rag nodes both run an interactive wizard when their
knowledge base has not been built, but neither offered the driver choice that
interactive named-RAG creation has, so both silently produced a yaml store.

A plain agent was the worse of the two: AgentConfig carries only documents, so
there was no way to get a duckdb RAG for one, interactively or declaratively. A
graph node could at least declare driver: in the workflow.

Agent startup now passes prompt_for_driver, and a rag node whose wizard runs is
asked too. The prompt is skipped when the node already declares a driver, and
sits inside the not-fully-specified branch after the non-interactive bail, so
declarative workflows and headless runs are unchanged. Temp RAGs still pass
false: they are deleted on the next run, so a persistent store would only leave
a sidecar behind.

The prompt moves to select_rag_driver rather than being duplicated.
2026-08-12 12:02:15 -06:00
Dark-Alex-17 81ed769f8a fix(rag): warn when a duckdb store is empty but files are indexed
A duckdb RAG is two files. The .yaml deliberately carries no vectors, and
open() runs CREATE TABLE IF NOT EXISTS, so a .yaml copied or synced without its
.duckdb sidecar produces a fresh empty store, hydrates to nothing, and answers
every query with nothing while .info rag still lists every indexed file.

Neither existing guard catches it: the anti-wipe check in rebuild_indexes needs
existing > 0, and the mandatory ? on hydration needs a genuine error, while an
absent store is the same Ok(empty) as a RAG with nothing indexed yet.

Warn rather than bail, so a store deleted on purpose still loads and can be
rebuilt.
2026-08-12 11:26:16 -06:00
Dark-Alex-17 6f7defe25f style: removed redundant comment 2026-08-12 10:56:34 -06:00
Dark-Alex-17 b837f82d7e fix(rag): keep a local Qdrant off an ambient proxy
Reverts the global proxy rework in 54685be and narrows it to the provider.

That commit took over proxy detection for every client in order to exempt
loopback and private ranges. Too broad: reqwest's detection also reads macOS
System Settings and the Windows registry behind its system-proxy feature, which
sits in its default set. Coyote disables default features today, so hand-rolling
the environment lookup happened to match — but re-enabling defaults later would
silently restore that support for main and not for the hand-rolled version. It
also made an explicitly configured proxy skip local hosts, which nobody asked
for: a proxy named for a LAN endpoint should be used.

build_client and utils are byte-identical to main again. The bypass now lives in
QdrantProvider::make_client, which is the only place that knows the target host,
and applies solely when that host is loopback, link-local, private or .local. A
public or cloud-hosted store keeps whatever the environment configures.

Also drops apply_proxy: with build_client reverted there was one caller left, and
set_proxy already covers it.

Both #[ignore]d live tests still pass against a Qdrant on loopback while an
ambient proxy that rejects it is in force.
2026-08-12 10:52:31 -06:00
Dark-Alex-17 54685be9a2 fix: keep loopback and LAN traffic off an ambient proxy
Pre-existing on main, not introduced by the driver work, but it makes a local
RAG backend unusable so it belongs with this change.

build_client only called set_proxy when a client had configured one of its own.
With nothing configured, reqwest's own detection applied, which sends every
request through a *_PROXY variable including ones bound for 127.0.0.1 or a LAN
address. A proxy cannot usefully forward those, and anything that intercepts
proxied traffic answers on behalf of a service that is running perfectly well,
so the error names the proxy rather than the store and reads as a Coyote fault.

Concretely, an installed Socket Firewall exports HTTP_PROXY to the processes it
wraps and rejects hosts outside its allow list. That turned a healthy Ollama on
the LAN into 'error decoding response body: expected value at line 2 column 1' —
its HTML refusal page parsed as JSON — and a loopback Qdrant into an HTTP 405.

Proxy handling is now always applied and always exempts loopback and private
ranges, with NO_PROXY merged in since replacing reqwest's detection also
replaces its handling of that variable. HTTP_PROXY and HTTPS_PROXY are kept
separate because they are allowed to differ. An explicitly configured proxy
still wins, and '-' still means none.

This also supersedes the unconditional no_proxy() added to the Qdrant client in
af9622d: that made it the only client to ignore a proxy outright, on a
justification I got wrong. It now shares this path, so a remote store behind a
real proxy keeps working.
2026-08-12 10:33:03 -06:00
Dark-Alex-17 4dd6e794b2 docs: removed redundant comment 2026-08-11 22:19:14 -06:00
Dark-Alex-17 af9622d31c fix(rag): stop routing Qdrant requests through an ambient proxy
make_client used a bare reqwest builder, which honours whatever proxy the
environment advertises. That made it the only HTTP client in Coyote to do so:
utils::set_proxy discards ambient settings and applies only Coyote's configured
proxy, and every other client goes through it.

The symptom is that a perfectly healthy Qdrant is unreachable and the error
belongs to the interposing proxy, not the store, so it reads as a Coyote or
Qdrant fault. Locally an installed Socket Firewall answered `.rag attach`
against 127.0.0.1:6333 with an HTML 'Connection Required' page and HTTP 405.

Both #[ignore]d live tests now pass against a real Qdrant; they failed with that
same 405 before this change, which is the first time either has run green.

A remote store that genuinely needs Coyote's configured proxy is a follow-up:
that means threading the proxy config into the provider.
2026-08-11 22:15:04 -06:00
Dark-Alex-17 6f586bd535 style: cleanup 2026-08-11 22:07:35 -06:00
Dark-Alex-17 78740db170 chore: ignore the .coyote workspace directory
It holds generated workspace state and should never be committed.
2026-08-11 22:03:45 -06:00
Dark-Alex-17 64d594f4ee refactor(rag): discover driver_config secrets by grammar, not field name
Sandbox provisioning only ever looked at driver_config["api_key"], so a driver
whose credential is called anything else would have been silently unprovisioned
inside a sandbox. It now scans every driver_config value and treats any that is
a secret placeholder as a credential, which is the same rule resolve_driver_config
already used at point of use.

The first one binds to the RAG's own service id, which is what the generated
mixin declares; any others register under their own names, as MCP secrets do.
The mixin still carries a single credential entry, so a driver needing two bound
secrets remains a follow-up.

Also drops the placeholder parser added in 74bc613. crate::vault::SECRET_RE is
already the canonical definition and was already imported here, so that was a
third implementation of the same grammar. Requiring the whole value to match is
what keeps a literal key from being read as a secret name and printed.

The api_key check is gone from RagData::validate: a generic config validator
should not know a provider's field names.
2026-08-11 22:03:45 -06:00
Dark-Alex-17 1322d73c7b style: further cleanup 2026-08-11 21:50:46 -06:00
Dark-Alex-17 de91ffa517 fix(rag): treat a zero min_score as no floor on Qdrant searches
parse_search_hits filtered on score > min_score, and the only caller passes
0.0. Qdrant Euclid collections score by negative distance, so every hit was
dropped and an attached Euclid collection returned nothing at all, silently.

This is the same trap the surrounding code already documents: score_threshold
is deliberately not sent because it is metric-aware and a 0.0 floor filters
everything out on Euclid. The local filter then reproduced it exactly. Only a
positive floor is now treated as a floor.
2026-08-11 21:07:20 -06:00
Dark-Alex-17 74bc613d94 fix(rag): address Copilot review findings on the driver abstraction
Five review comments, all real:

- hybrid_search ran its vector and keyword legs sequentially after the
  provider refactor; main ran them under tokio::join!. Restores the
  concurrency while keeping the degrade-on-error keyword behaviour, so a
  remote provider no longer pays two serial round trips per query.

- inject_rag_secrets derived a vault secret name by trimming braces, which
  leaves a literal key untouched. A RAG holding a plaintext api_key therefore
  looked the secret up by its own value and printed it to stderr on failure.
  Parsing is now strict and a non-placeholder is skipped with a warning that
  names no credential.

- validate() now refuses a driver_config.api_key that is not a {{NAME}}
  placeholder, so a plaintext key cannot reach the RAG YAML at all.

- Rag::create's catch-all arm treated any unrecognised driver as yaml. A typo
  built a yaml store, paid to embed the corpus, persisted the bad driver and
  only failed on the next run. Unknown drivers now fail immediately.

- The qdrant arm's error was written for a developer; it now tells the user
  that only attached collections are readable and points at .rag attach.
2026-08-11 21:04:21 -06:00
Dark-Alex-17 6d0a5550fe style: Removed some redundant comments 2026-08-11 20:56:51 -06:00
Dark-Alex-17 7b1c0342b4 fix(rag): delete the DuckDB write-ahead log alongside the store
Deleting a RAG removed its .duckdb file but left the sibling .duckdb.wal
behind. DuckDB only removes that log on a clean close, so any RAG whose
process was killed left one on disk, and creating a new RAG under the same
name let it inherit a write-ahead log describing someone else's data.

The test helper already cleaned the log up after itself, which is why no
test noticed the production path did not.
2026-08-11 16:55:19 -06:00
Dark-Alex-17 d6c114fe58 feat: simplified the duckdb selection prompt 2026-08-11 16:50:50 -06:00
Dark-Alex-17 912e00a627 docs(rag): correct the duckdb concurrency note in the driver prompt
The driver picker still told users a duckdb RAG can only be open in one Coyote
process at a time. That stopped being true once the store began opening
read-only for queries: any number of processes can now query it concurrently.

The restriction that remains is narrower and only bites while writing, so the
prompt now states that instead — several processes can query at once, but an
ingest or rebuild locks the others out until it finishes.
2026-08-11 16:38:55 -06:00
Dark-Alex-17 e006e29ff1 feat(rag): let several Coyote processes query one duckdb RAG at once
The DuckDB store was always opened read-write, which takes an exclusive file
lock, so a second Coyote process could not even read the RAG. Querying does not
write, and DuckDB permits many concurrent readers as long as no writer is
attached, so the store is now opened read-only whenever it already carries a
complete schema.

Creating or initializing the store still writes, as does rebuilding, so those
paths take the exclusive handle. The rebuild path upgrades a read-only
connection in place, which every clone of the handle observes because the mode
lives behind the shared mutex rather than beside it. Extension loads and the
HNSW persistence setting are per-connection and are re-established on the
upgraded connection.

An upgrade that loses the race for the write lock reports that another process
holds the RAG and that nothing was written, then reopens read-only so the
session can keep querying. A read-only handle also refuses writes outright, so a
missed upgrade cannot silently discard an ingest.
2026-08-11 14:58:18 -06:00
Dark-Alex-17 c0067d387c refactor(rag): drop the hardcoded embedding model hint from attach
The attach wizard mapped a collection's vector dimension to a hardcoded list of
model names and printed them as likely candidates. The list was never checked
against the models the user actually has configured, so it could recommend a
model they cannot select, and one entry was a parenthetical note rather than a
model id and so could never match anything. Any list like this rots as models
are released.

The dimension itself comes from the server and is worth stating, so it is still
printed, as is the warning that a mismatched embedding model returns bad
results. Deriving real candidates would need a dimension recorded against each
configured model, which the model config does not carry today.
2026-08-11 14:06:57 -06:00
Dark-Alex-17 118c346345 feat(rag): support string and UUID Qdrant point IDs
Point ids were read with as_u64() inside a filter_map, so a string id was
silently dropped and a UUID-keyed collection returned zero hits with no error.
The attach wizard therefore refused such collections and told the user to
rebuild with integer ids, which defeats the purpose of attaching to a
collection someone else already built. LangChain, a common way to populate
Qdrant, uses UUIDs by default.

The integer id was never load-bearing for this driver. DocumentId is a packed
(file, chunk) pair used positionally by the local drivers, but an attached RAG
holds no local files or vectors and every positional consumer already returns
early on it, so the id only has to survive the round trip from search back to
the content fetch. Ids that cannot make that trip as a u64 are interned behind
a synthetic handle and restored when the fetch is issued, leaving collections
that already use integer ids on exactly the path they used before.
2026-08-11 14:05:38 -06:00
Dark-Alex-17 dc677a2529 fix(sandbox): discover agent-scoped RAG mixin sidecars
An agent-scoped RAG writes its config to <data>/agents/<agent>/<rag>.yaml, so
its sbx mixin sidecar lands beside it as <rag>.sbx-mixin.yaml. Discovery scanned
the agents directory only for a file named exactly sbx-mixin.yaml, and scanned
for suffixed sidecars only in the top-level rags directory, so a RAG attached
while an agent was active contributed no network allow rule and no credential to
the sandbox. The failure was silent: the sandbox launched and the RAG was simply
unreachable from inside it.

The two collectors differed only in the filename shape they matched, so they are
now one scan that takes the set of layouts to look for. The agents directory
asks for both its own sbx-mixin.yaml and the suffixed sidecars one level in,
which is the shape that was missing. Discovery order is unchanged, and it is
load-bearing: each mixin becomes a --kit in list order and later ones layer over
earlier ones, so the workspace mixin must stay last.
2026-08-11 14:05:31 -06:00
Dark-Alex-17 5e2b9c98ad fix(rag): stop the attach wizard from silently accepting an empty collection
`sample_point_id` returns `None` for a collection with no points, so the
UUID guard's `if let Some(..)` fell straight through and the wizard attached
happily. The result is a RAG that answers every query with zero hits and
never says why.

Sample once, then check for emptiness explicitly. This warns and asks rather
than hard-failing: an empty collection is not necessarily a mistake, since
another tool may be about to populate it, and none of the wizard's remaining
probes can distinguish that from a misconfiguration. The confirmation
defaults to "no" so it cannot be walked past by accident, and `attach`
already refuses to run non-interactively, so no unattended path reaches it.
2026-08-11 13:55:59 -06:00
Dark-Alex-17 c458ca93a9 feat(rag): create the API key secret inline in the attach wizard
The wizard hard-errored with "Secret 'X' not found in vault. Run
`coyote --add-secret X` first.", throwing away every answer the user had
already given it. Offer to create the secret in place instead, deferring to
`Vault::add_secret` for the masked prompt, the provider write and the
confirmation line, then read it back.

Only a genuine `SecretError::NotFound` triggers the offer. An auth failure,
a provider outage, or the vault being disabled inside a sandbox all
propagate with their own message, because prompting for a value that cannot
be stored would fail one step later and bury the real cause. Declining the
offer fails with both ways out spelled: add the secret up front, or answer
"no" to the API-key question.
2026-08-11 13:50:33 -06:00
Dark-Alex-17 860566bf50 feat: let workflow rag nodes select a RAG driver
`RagNode` gains an optional `driver`, forwarded into `RagInitConfig` so a
graph node can build its knowledge base on duckdb instead of yaml. Nodes
that name no driver forward `None`, which still resolves to yaml, so
existing workflows are unaffected.

An unknown driver is rejected up front rather than at construction time.
`Rag::create` dispatches unknown drivers to its yaml catch-all, so a typo
would otherwise embed every document and persist the bogus string, after
which every subsequent load fails validation and the agent cannot start.
The check asks `RagData::validate()` through a probe value instead of
restating the list of valid drivers, so the two cannot drift.
2026-08-11 13:46:59 -06:00
Dark-Alex-17 7f90710427 refactor(rag): interpolate every driver_config value, not just api_key
Only `driver_config["api_key"]` was interpolated, so any credential-bearing
driver field added later would have shipped its raw `{{PLACEHOLDER}}` to the
server. Resolve every value instead, via `resolve_driver_config`.

Resolution still happens into a function-local copy and never touches
`RagData`: `save()` serializes `self.data` and is called by `.set rag_top_k`
and friends, so a resolved credential parked there would be written to the
RAG's YAML in plaintext. The literal `{{NAME}}` also has to survive on disk
because sandbox credential provisioning parses it back out to learn which
vault secret to bind. Scope stays `driver_config` deliberately: the rest of
a RAG file is ingested document text, where `{{...}}` is ordinary content.

Covered by a test that saves after a load and asserts the placeholder, not
the secret, is what reaches the file.
2026-08-11 13:46:32 -06:00
Dark-Alex-17 93a934439b fix(rag): fail loudly when a RAG's vault secret is missing
`interpolate_secrets` does not error on a secret the vault cannot resolve:
it substitutes the empty string and returns the name in its second tuple
element. `load_async` discarded that vec, so a typo'd or deleted vault
secret produced `api_key = ""` and an unexplained 401 from Qdrant.

Bail instead, naming the RAG and the missing secrets, matching what global
config loading already does.
2026-08-11 13:44:00 -06:00
Dark-Alex-17 ecda258d3a style: Cleaned up some minor styling issues 2026-08-11 13:04:27 -06:00
Dark-Alex-17 3e598065f8 fix(rag): serialize DuckDB extension installs to stop a Windows race
`ensure_extension` fell back to `INSTALL` whenever `LOAD` failed. With a cold
extension cache every thread's `LOAD` fails at once, so every thread ran
`INSTALL` concurrently for the same extension. DuckDB installs by downloading
to a temp file and then MOVING it into `~/.duckdb/extensions/...`; POSIX allows
replacing a file other handles hold open, so Linux and macOS survived, but
Windows rejects that move with "Access is denied" and the losing threads failed.

Guard the install step with a process-global mutex and re-check `LOAD` after
acquiring it. The re-check is what bounds the work to a single install: without
it every thread queued behind the winner would still run a redundant `INSTALL`
and repeat the same move over a file that is now open.

`LOAD` is per-connection, so it still runs on every connection; only `INSTALL`
is serialized. An already-installed extension takes the pre-lock fast path and
costs neither a lock nor network. The lock is never held across the connection
mutex, so it cannot invert lock order.

Traced with strace on a cold cache under default test parallelism: before, 17
threads moved files into the store (13 racing on vss alone); after, exactly one
rename per extension.
2026-08-10 16:13:36 -06:00
Dark-Alex-17 3abc30d633 fix(rag): emit an sbx kit v2 mixin and declare RAG credentials to the proxy
The RAG attach sidecar was written against the sbx kit v1 spec and still emitted schemaVersion "1" with network.allowedDomains, network.serviceDomains, network.serviceAuth, credentials.sources.<n>.env and environment.proxyManaged. Every one of those keys was removed in kit v2. Coyote does not validate mixins, it copies them byte-for-byte into spec.yaml, so the invalid document surfaced only as an opaque sbx failure with no indication of which mixin caused it.

generate_rag_sbx_mixin now builds the document from the shared serializer structs instead of a format! string, which is how the envelope drifted unnoticed in the first place. render_mixin_yaml and the RAG sidecar both go through a new render_mixin_document, giving one definition of the envelope and one enforcement point for the rule that every inject domain must also appear in permissions.network.allow.

Fix an auth bug the port exposed: inject_rag_secrets bound the API key with sbx secret set, but nothing ever emitted a matching credentials entry, so the proxy held a value with no inject rule and never rewrote the auth header. An attached RAG credential silently did not work inside the sandbox. The sidecar now declares that credential; a RAG with no API key declares none while still receiving egress.

Fix the service id: the bind passed the raw file stem instead of routing it through secret_service_id, so a RAG named My_Docs produced an illegal id. The bind and the generated credentials service now share that derivation and cannot disagree.

Retire sbx_domain_forms in favour of allow_entry_for_url, now pub(crate). It emitted both a bare host and host:port because v1 serviceDomains needed a bare key; v2 has no such need, so the extra entry is simply wrong. It also defaulted a schemeless host to port 6333 while normalize_base_url resolves it to http and port 80, meaning the allow entry named a port the client never dialled.
2026-08-10 15:58:40 -06:00
Dark-Alex-17 f68937611e fix(rag): install DuckDB vss and fts extensions when they are missing
The DuckDB schema init loaded the vss and fts extensions but nothing ever
installed them, so any machine without them already present failed with
'IO Error: Extension "vss.duckdb_extension" not found'. This surfaced as 13
failing tests in CI while passing locally, because local runs had the
extensions installed already.

Loading is attempted first so an extension that is already present costs
nothing and never touches the network; INSTALL is reached only once, on a
machine seeing the extension for the first time, and reports an actionable
message if it cannot download.

CI cached the extension directory but nothing populated it, so the cache
saved an empty directory forever. The cache key now derives from Cargo.lock
rather than a hardcoded DuckDB version, and a step on cache miss installs the
extensions so the post-job save has something to store.
2026-08-10 15:47:15 -06:00
Dark-Alex-17 a12cf84eb6 Merge remote-tracking branch 'refs/remotes/origin/main' 2026-08-10 15:41:50 -06:00
Dark-Alex-17 7a732436aa feat(rag): add attach-only Qdrant provider, attach wizard and sandbox wiring
Adds QdrantProvider as a read-only driver for pre-existing remote Qdrant
collections, an interactive '.rag attach' wizard, and the sandbox credential
and domain-whitelisting wiring that lets an attached RAG work inside a sandbox.

Attach-only by design: rebuild_indexes bails for both the attached and the
unattached case rather than silently succeeding. Coyote never writes to Qdrant
in this change.

Vectors are never hydrated back from Qdrant. Cosine collections L2-normalize
stored vectors on write, so reading them back returns unit-length copies of the
originals; the YAML vector copy is authoritative and the serialization guard
stays scoped to the duckdb driver alone.

Collections keyed by string or UUID point IDs are rejected at attach time. The
read path parses point ids as u64 inside a filter_map, so such a collection
would otherwise yield zero results with no error.

The three response-shape parsers are pure functions over an already-parsed JSON
body, unit-tested against captured fixtures, with the async wrappers delegating
to them rather than duplicating the logic.

'.rag' now splits its first argument, so '.rag attach <name>' no longer tries to
load a RAG literally named 'attach <name>'.
2026-08-10 13:40:22 -06:00
Dark-Alex-17 98d3ba4a83 feat(rag): add DuckDB provider behind the RAG driver abstraction
Phase 3 of the RAG driver abstraction. Adds a `DuckDbProvider` that keeps
vectors and document content in a `.duckdb` sidecar next to the existing
YAML metadata, selected by the `driver: duckdb` field.

- `src/rag/providers/duckdb.rs` (new): vector search via the vss extension
  and keyword search via fts, an all-or-nothing hydration path (a partial
  read is an error, never a shorter map), and an anti-wipe guard that
  refuses the destructive `CREATE OR REPLACE TABLE` when `data.vectors` is
  empty while `data.files` is not and the store still holds rows.
- `src/rag/mod.rs`: `sync_documents` now refreshes `bm25`/`node_to_docs`
  BEFORE the fallible `provider.rebuild_indexes`. `self.data` is already
  mutated by that point, so propagating a provider error afterwards would
  leave the derived in-memory state describing the previous corpus while
  `data` describes the new one. Both rebuilds are pure functions of
  `self.data` and cannot fail, so running them first is always safe.
- `src/config/paths.rs`: sidecar path helpers.
- `src/rag/providers/mod.rs`, `src/config/agent.rs`: driver dispatch and
  RAG cache keying.

Also keeps `RequestContext::rag_key` in lockstep with `rag` at the two
sites that were still missing it, so that a cache insert and its matching
invalidate are structurally incapable of disagreeing:

- `use_agent` assigned `self.rag` from the agent but never set `rag_key`.
  This one was live. Agent RAGs are inserted under `RagKey::Agent(<name>)`,
  so with `rag_key == None` the invalidation guards in `rebuild_rag` and
  `edit_rag_docs` matched nothing and `.rebuild rag` left the stale cache
  entry in place. Worse, a preceding `.rag <name>` left a stale
  `Named(<name>)` key attached to the agent's RAG, pointing the
  invalidation at an unrelated RAG's cache entry. Now mirrors the insert
  key exactly, yielding `None` when the agent has no RAG.
- `exit_agent` cleared `self.rag` but left `rag_key` behind. Latent rather
  than live, since `rebuild_rag`/`edit_rag_docs` both bail on
  `rag.is_none()` before reaching the invalidate guards, but the guards
  that make it unobservable are not the kind of thing to depend on.

Covered by `use_agent_does_not_carry_stale_rag_key`, and by a new
assertion in `exit_agent_clears_all_agent_state`.
2026-08-10 12:51:56 -06:00
Dark-Alex-17 d734276927 build(rag): add duckdb dependency and pin comfy-table to 7.1.4
Adds the duckdb crate with the bundled feature ahead of any provider
code, so the dependency and build surface can be proven on every CI
target on its own.

duckdb constrains comfy-table to ~7.1, so comfy-table moves from 7.2.2
to 7.1.4 while keeping custom_styling. That feature swaps
measure_text_width for an ANSI-stripping implementation that
render_table's pre-styled cells depend on; dropping it still compiles
and still passes every other test, and only corrupts column widths. A
regression test now renders a styled table at a fixed wrap width and
asserts every line has an equal ANSI-stripped display width.

comfy-table 7.1.4 pins crossterm 0.28 while coyote pins 0.29, so both
now build side by side. comfy_table::Color, Attribute and Cell are
consequently crossterm 0.28 types and must not be used; table styling
stays ANSI-string based.

Caches the DuckDB extension directory in CI, keyed on the runner OS and
the DuckDB version, so vss and fts survive an upstream outage.

Also switches merge_vector_results to f32::total_cmp. The previous
partial_cmp().unwrap_or(Equal) comparator is not total under NaN, which
sort_by is permitted to answer with a panic in release builds.
2026-08-10 11:59:57 -06:00
Dark-Alex-17 5049143fcc refactor(rag): extract RagProvider trait and add YamlProvider
Introduce a narrow `RagProvider` trait covering vector search and content
retrieval, and make `Rag` delegate to a boxed provider instead of owning an
HNSW index directly. `YamlProvider` is the sole implementation for now.

The trait deliberately stays narrow: embeddings, chunking, BM25 keyword
search, graph RAG, entity extraction, RRF merging and persistence all remain
on `Rag`/`RagData`, so a new storage backend does not have to reimplement
Coyote's indexing logic.

Notable points:

- `fetch_content`'s ordering contract is part of the trait, not an accident.
  Implementations must return results in input-`ids` order; `hybrid_search`
  passes an RRF-ranked list straight to the prompt builder, so a provider
  returning storage order would silently discard the ranking.
- The content store is keyed on `data.files`, never `data.vectors`. Both the
  content map and BM25 now route through the new `RagData::iter_documents()`
  so the two key spaces match by construction. `RagData::add` zips document
  ids with embeddings and truncates silently, so ids in `files \ vectors` are
  genuinely reachable.
- A provider keyword-search failure degrades to an empty ranker with a
  warning rather than failing the whole query; it is one of three RRF inputs.
  It deliberately does not fall back to the local BM25, which would be a
  silent ranking-algorithm swap once a provider with native FTS exists.
- The rerank path builds its text and id vectors from a single `fetch_content`
  result in one pass, so the reranker's positional indices cannot desync.

This is not a bit-for-bit no-op. `vector_search` now dedups by best score and
sorts globally instead of concatenating per-chunk hit lists. Single-chunk
queries (the common case) are unaffected. Multi-chunk queries get corrected
rank assignment and no longer let a document that matched several query
chunks accumulate multiple RRF contributions. There is no overall cap on the
merged pool — truncation remains `reciprocal_rank_fusion`'s job.

`RagData::get()` is removed: its only two callers were the content lookups
replaced here, and an unused private-module method fails the build under
`--deny warnings`. Its three tests were rewritten against `iter_documents()`,
one of which now guards the files-vs-vectors keying directly.

Implements Phase 2 of the RAG driver abstraction design (§6).
2026-08-10 11:41:37 -06:00
Dark-Alex-17 a968c3228d feat(rag): add driver/attached fields, validation floors and force-reingest
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).

Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
  `attached: bool` as the first two fields of RagData, so driver metadata sits
  at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
  YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
  map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
  It is deliberately not derived: a derived impl yields `driver: ""`, which is
  not a valid driver string.

Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
  It enforces the (driver, attached) matrix and, critically, numeric floors
  that the new defaults would otherwise mask: `top_k >= 1` unconditionally
  (a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
  `chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
  divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
  this, the new load-time floor turns one keystroke into an unloadable RAG:
  the setter saves immediately and no dot-command can reach the file again.

Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
  nothing, because the content-hash skip fired regardless of the refresh flag.
  Extract that decision into a module-level `find_hash_skip()` free function and
  thread a `force_reingest` flag through `sync_documents()` and
  `refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
  stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
  now prints a one-line file-count warning first (no prompt: the path is
  reachable from a non-interactive CLI flag).

Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
  which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
  driver/attached through `Rag::export()` so `.info rag` shows them.

Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
2026-08-10 11:04:51 -06:00
Dark-Alex-17 f404acdbca Merge branch 'main' 2026-08-06 16:39:02 -06:00
Dark-Alex-17 9aeb9e6e2e Merge branch 'main' 2026-08-06 13:13:03 -06:00
29 changed files with 7002 additions and 343 deletions
+11
View File
@@ -36,6 +36,17 @@ jobs:
- uses: Swatinem/rust-cache@v2
- name: Cache DuckDB Extensions
id: duckdb-extensions
uses: actions/cache@v4
with:
path: ~/.duckdb/extensions
key: duckdb-ext-${{ matrix.os }}-${{ hashFiles('Cargo.lock') }}
- name: Install DuckDB Extensions
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
run: cargo test --all duckdb
- name: Test
run: cargo test --all
+1
View File
@@ -5,3 +5,4 @@
.idea/
/coyote.iml
/.idea/
.coyote
Generated
+566 -141
View File
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -17,8 +17,9 @@ exclude = [".github", "CONTRIBUTING.md"]
anyhow = "1.0.69"
bytes = "1.4.0"
clap = { version = "4.5.40", features = ["cargo", "derive", "wrap_help"] }
comfy-table = { version = "7.2.2", features = ["custom_styling"] }
comfy-table = { version = "7.1.4", features = ["custom_styling"] }
dirs = "6.0.0"
duckdb = { version = "1.10505.0", features = ["bundled"] }
dunce = "1.0.5"
futures-util = "0.3.29"
inquire = "0.9.4"
+15
View File
@@ -36,6 +36,21 @@ RUN set -euo pipefail; \
install -m 0755 "$TMPDIR/usql_static" /usr/local/bin/usql; \
rm -rf "$TMPDIR"
RUN set -euo pipefail; \
DUCKDB_VERSION=1.5.5; \
case "${TARGETARCH}" in \
amd64) DUCKDB_ARCH=amd64 ;; \
arm64) DUCKDB_ARCH=arm64 ;; \
*) echo "Unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
esac; \
TMPDIR=$(mktemp -d); \
curl -fsSL --retry 3 \
"https://github.com/duckdb/duckdb/releases/download/v${DUCKDB_VERSION}/duckdb_cli-linux-${DUCKDB_ARCH}.gz" \
-o "$TMPDIR/duckdb.gz"; \
gunzip "$TMPDIR/duckdb.gz"; \
install -m 0755 "$TMPDIR/duckdb" /usr/local/bin/duckdb; \
rm -rf "$TMPDIR"
USER 1000
RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
+2
View File
@@ -70,6 +70,8 @@ Coyote requires the following tools to be installed on your system:
* **Cargo:** `cargo install ast-grep --locked`
* **npm:** `npm i -g @ast-grep/cli`
* Optional: if `ast-grep` is not installed, the `ast_grep` tool reports it and agents fall back to `fs_grep`
* [duckdb](https://duckdb.org/) (for fast, local RAGs)
* `curl https://install.duckdb.org | sh`
These tools are used to provide various functionalities within Coyote, such as document processing, JSON manipulation,
etc., and they are used within agents and tools.
+161 -1
View File
@@ -13,6 +13,20 @@ use is_terminal::IsTerminal;
use std::collections::HashSet;
use std::io::{Read, stdin};
#[derive(clap::ValueEnum, Debug, Clone, Copy, PartialEq, Eq)]
pub enum McpTransportArg {
Stdio,
Http,
Sse,
}
#[derive(clap::ValueEnum, Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum McpScopeArg {
#[default]
User,
Workspace,
}
#[derive(Parser, Debug)]
#[command(author, version, about, long_about = None)]
#[command(
@@ -41,10 +55,15 @@ use std::io::{Read, stdin};
"list_skills", "skill", "tail_logs", "completions", "update",
])
),
group(
ArgGroup::new("mcp-action")
.args(["mcp_add", "mcp_remove", "mcp_list", "mcp_get"])
.multiple(false)
),
)]
pub struct Cli {
/// Input text
#[arg(trailing_var_arg = true)]
#[arg(allow_hyphen_values = true)]
text: Vec<String>,
/// Select a LLM model
@@ -224,6 +243,57 @@ pub struct Cli {
#[arg(long, exclusive = true, value_name = "SERVER_NAME", help_heading = "Authentication", add = ArgValueCompleter::new(mcp_server_completer))]
pub auth_mcp: Option<String>,
/// Add an MCP server. Use `-- <cmd> [args...]` for stdio, or `--url <URL>` for http/sse.
#[arg(long, value_name = "NAME", help_heading = "MCP Servers")]
pub mcp_add: Option<String>,
/// Remove an MCP server by name
#[arg(long, value_name = "NAME", help_heading = "MCP Servers", add = ArgValueCompleter::new(mcp_server_completer))]
pub mcp_remove: Option<String>,
/// List all configured MCP servers (user + workspace scopes)
#[arg(long, help_heading = "MCP Servers")]
pub mcp_list: bool,
/// Show the JSON config for one MCP server
#[arg(long, value_name = "NAME", help_heading = "MCP Servers", add = ArgValueCompleter::new(mcp_server_completer))]
pub mcp_get: Option<String>,
/// Transport for --mcp-add: stdio (default when `--` present), http, or sse
#[arg(
long,
value_enum,
value_name = "TRANSPORT",
help_heading = "MCP Servers"
)]
pub transport: Option<McpTransportArg>,
/// URL for http/sse MCP server (used with --mcp-add)
#[arg(long, value_name = "URL", help_heading = "MCP Servers")]
pub url: Option<String>,
/// Scope for MCP config: user (~/.config/coyote/functions/mcp.json) or workspace (./.coyote/mcp.json). Default: user
#[arg(long, value_enum, value_name = "SCOPE", help_heading = "MCP Servers")]
pub scope: Option<McpScopeArg>,
/// Environment variable for stdio MCP server (repeatable): --env KEY=VALUE
#[arg(long, value_name = "KEY=VALUE", help_heading = "MCP Servers")]
pub env: Vec<String>,
/// HTTP header for http/sse MCP server (repeatable): --header "Name: Value"
#[arg(long, value_name = "HEADER", help_heading = "MCP Servers")]
pub header: Vec<String>,
/// Working directory for stdio MCP server
#[arg(long, value_name = "PATH", value_hint = ValueHint::AnyPath, help_heading = "MCP Servers")]
pub cwd: Option<String>,
/// OAuth client ID for http/sse MCP server
#[arg(long, value_name = "ID", help_heading = "MCP Servers")]
pub client_id: Option<String>,
/// OAuth client secret for http/sse MCP server (use {{NAME}} to reference a vault secret)
#[arg(long, value_name = "SECRET", help_heading = "MCP Servers")]
pub client_secret: Option<String>,
/// OAuth callback port for http/sse MCP server
#[arg(long, value_name = "PORT", help_heading = "MCP Servers")]
pub callback_port: Option<u16>,
/// OAuth redirect host for http/sse MCP server
#[arg(long, value_name = "HOST", help_heading = "MCP Servers")]
pub redirect_host: Option<String>,
/// Overwrite an existing MCP server (with --mcp-add) or skip confirmation (with --mcp-remove)
#[arg(long, help_heading = "MCP Servers")]
pub mcp_force: bool,
/// Launch Coyote inside a Docker sandbox (via `sbx`); name defaults to current directory basename
#[arg(long, value_name = "NAME", help_heading = "Sandbox")]
pub sandbox: Option<Option<String>>,
@@ -254,6 +324,15 @@ pub struct Cli {
/// Generate static shell completion scripts
#[arg(long, value_name = "SHELL", value_enum, help_heading = "Shell")]
pub completions: Option<ShellCompletion>,
/// Stdio command for --mcp-add: everything after `--` is passed to the server verbatim
#[arg(
last = true,
allow_hyphen_values = true,
value_name = "CMD",
help_heading = "MCP Servers"
)]
pub mcp_command: Vec<String>,
}
impl Cli {
@@ -633,4 +712,85 @@ mod tests {
fn parse_sandbox_is_exclusive() {
assert!(Cli::try_parse_from(["coyote", "--sandbox", "--agent", "foo"]).is_err());
}
#[test]
fn parse_mcp_add_stdio_with_trailing_command() {
let cli = parse(&[
"--mcp-add",
"myserver",
"--",
"npx",
"some-server",
"--flag",
"arg1",
]);
assert_eq!(cli.mcp_add, Some("myserver".to_string()));
assert_eq!(
cli.mcp_command,
vec!["npx", "some-server", "--flag", "arg1"]
);
assert!(cli.text.is_empty());
}
#[test]
fn parse_mcp_add_stdio_with_env_and_command() {
let cli = parse(&[
"--mcp-add",
"s",
"--env",
"API_KEY={{API_KEY}}",
"--env",
"MODE=dev",
"--",
"npx",
"srv",
]);
assert_eq!(cli.mcp_add, Some("s".to_string()));
assert_eq!(cli.env, vec!["API_KEY={{API_KEY}}", "MODE=dev"]);
assert_eq!(cli.mcp_command, vec!["npx", "srv"]);
}
#[test]
fn parse_mcp_add_http_with_header() {
let cli = parse(&[
"--mcp-add",
"notion",
"--transport",
"http",
"--url",
"https://mcp.notion.com/mcp",
"--header",
"Authorization: Bearer {{NOTION_TOKEN}}",
]);
assert_eq!(cli.mcp_add, Some("notion".to_string()));
assert!(matches!(cli.transport, Some(McpTransportArg::Http)));
assert_eq!(cli.url, Some("https://mcp.notion.com/mcp".to_string()));
assert_eq!(cli.header, vec!["Authorization: Bearer {{NOTION_TOKEN}}"]);
assert!(cli.mcp_command.is_empty());
}
#[test]
fn parse_mcp_list_flag() {
let cli = parse(&["--mcp-list"]);
assert!(cli.mcp_list);
}
#[test]
fn parse_mcp_scope_workspace() {
let cli = parse(&["--mcp-list", "--scope", "workspace"]);
assert!(cli.mcp_list);
assert!(matches!(cli.scope, Some(McpScopeArg::Workspace)));
}
#[test]
fn parse_mcp_action_group_is_exclusive() {
assert!(Cli::try_parse_from(["coyote", "--mcp-list", "--mcp-get", "foo"]).is_err());
}
#[test]
fn parse_trailing_text_unchanged_without_dash_dash() {
let cli = parse(&["hello", "world"]);
assert_eq!(cli.text, vec!["hello", "world"]);
assert!(cli.mcp_command.is_empty());
}
}
+115 -5
View File
@@ -1,3 +1,4 @@
use std::collections::HashSet;
use std::mem;
use super::access_token::get_access_token;
@@ -368,17 +369,32 @@ pub fn claude_build_chat_completions_body(
]
} else {
// One pair per round: Claude can reuse tool_use IDs across API calls.
// A round boundary is detected by the presence of round text, but
// rounds where the model emitted only tool calls (no narration)
// carry no text marker. As a backstop, also split whenever a
// tool_use ID would repeat within the current assistant message —
// the API rejects duplicate tool_use IDs in a single message.
let mut messages = vec![];
let mut assistant_parts: Vec<serde_json::Value> = vec![];
let mut user_parts: Vec<serde_json::Value> = vec![];
let mut chunk_ids: HashSet<&str> = HashSet::new();
for (index, tool_result) in tool_results.iter().enumerate() {
if index > 0 && tool_result.text.is_some() {
let id_collision = tool_result
.call
.id
.as_deref()
.is_some_and(|id| chunk_ids.contains(id));
if index > 0 && (tool_result.text.is_some() || id_collision) {
messages.push(
json!({ "role": "assistant", "content": assistant_parts }),
);
messages.push(json!({ "role": "user", "content": user_parts }));
assistant_parts = vec![];
user_parts = vec![];
chunk_ids.clear();
}
if let Some(id) = tool_result.call.id.as_deref() {
chunk_ids.insert(id);
}
for block in &tool_result.thinking {
assistant_parts.push(json!(block));
@@ -485,10 +501,7 @@ pub fn claude_extract_chat_completions(data: &Value) -> Result<ChatCompletionsOu
if let Some(v) = item["thinking"].as_str() {
thinking.push(ThinkingBlock::Thinking {
thinking: v.to_string(),
signature: item["signature"]
.as_str()
.unwrap_or_default()
.to_string(),
signature: item["signature"].as_str().unwrap_or_default().to_string(),
});
}
}
@@ -535,3 +548,100 @@ pub fn claude_extract_chat_completions(data: &Value) -> Result<ChatCompletionsOu
};
Ok(output)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::function::{ToolCall, ToolResult};
fn tool_result(id: &str, text: Option<&str>) -> ToolResult {
ToolResult {
call: ToolCall::new("fs_read".into(), json!({"path": "x"}), Some(id.into())),
output: json!("ok"),
text: text.map(|t| t.to_string()),
thinking: vec![],
}
}
fn build_body(tool_results: Vec<ToolResult>) -> Value {
let data = ChatCompletionsData {
messages: vec![
Message::new(MessageRole::User, MessageContent::Text("hello".to_string())),
Message::new(
MessageRole::Assistant,
MessageContent::ToolCalls(MessageContentToolCalls {
tool_results,
text: String::new(),
sequence: true,
}),
),
],
temperature: None,
top_p: None,
reasoning_effort: None,
functions: None,
stream: false,
};
claude_build_chat_completions_body(data, &Model::new("claude", "claude-test")).unwrap()
}
fn assert_unique_tool_use_ids_per_message(body: &Value) {
for message in body["messages"].as_array().unwrap() {
let Some(content) = message["content"].as_array() else {
continue;
};
let mut seen = HashSet::new();
for block in content {
if block["type"] == "tool_use" {
let id = block["id"].as_str().unwrap();
assert!(
seen.insert(id.to_string()),
"duplicate tool_use id `{id}` within a single assistant message: {message}"
);
}
}
}
}
#[test]
fn sequence_splits_on_round_text() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", Some("running another tool")),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 5, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn sequence_splits_on_reused_id_in_textless_round() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_A", None),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 5, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
#[test]
fn sequence_keeps_textless_rounds_merged_when_ids_are_unique() {
let body = build_body(vec![
tool_result("toolu_A", None),
tool_result("toolu_B", None),
tool_result("toolu_C", None),
]);
let messages = body["messages"].as_array().unwrap();
assert_eq!(messages.len(), 3, "body: {body}");
assert_unique_tool_use_ids_per_message(&body);
}
}
+100 -14
View File
@@ -4,6 +4,7 @@ use crate::{
client::Model,
config::memory,
function::{Functions, run_llm_function},
graph, rag,
};
use super::rag_cache::RagKey;
@@ -12,6 +13,7 @@ use crate::config::prompts::{
DEFAULT_SPAWN_INSTRUCTIONS, DEFAULT_TEAMMATE_INSTRUCTIONS, DEFAULT_TODO_INSTRUCTIONS,
DEFAULT_USER_INTERACTION_INSTRUCTIONS,
};
use crate::graph::types::RagNode;
use crate::graph::{Graph, GraphParser, NodeType};
use crate::rag::RagInitConfig;
use crate::vault::SECRET_RE;
@@ -146,11 +148,18 @@ impl Agent {
let rag = if rag_path.exists() {
let key = RagKey::Agent(name.to_string());
let app_clone = app.clone();
let vault_clone = app_state.vault.clone();
let rag_path_clone = rag_path.clone();
let rag = app_state
.rag_cache
.load_with(key, || async move {
Rag::load(&app_clone, DEFAULT_AGENT_NAME, &rag_path_clone)
Rag::load_async(
&app_clone,
&vault_clone,
DEFAULT_AGENT_NAME,
&rag_path_clone,
)
.await
})
.await?;
Some(rag)
@@ -171,7 +180,15 @@ impl Agent {
let rag = app_state
.rag_cache
.load_with(key, || async move {
Rag::init(&app_clone, "rag", &rag_path_clone, &document_paths, abort).await
Rag::init(
&app_clone,
"rag",
&rag_path_clone,
&document_paths,
abort,
true,
)
.await
})
.await?;
Some(rag)
@@ -231,6 +248,10 @@ impl Agent {
}
}
if rag.is_some() && app.function_calling_support && graph_for_rag.is_none() {
functions.append_rag_query_functions();
}
agent_config.replace_tools_placeholder(&functions);
Ok(Self {
@@ -937,6 +958,30 @@ fn resolve_document_paths(
Ok(document_paths)
}
/// How a graph rag node describes the knowledge base it wants built.
///
/// `driver` is forwarded as-is: `None` means the node did not ask for one, which
/// `RagInitConfig` resolves to yaml, so workflows written before drivers existed
/// keep their current storage.
///
/// Every field is now named explicitly, so adding one to `RagInitConfig` breaks
/// this literal. That is deliberate: the new field then gets a decision about
/// whether a rag node can drive it, instead of silently taking its default.
fn rag_init_config(rag_node: &RagNode) -> RagInitConfig {
RagInitConfig {
embedding_model: rag_node.embedding_model.clone(),
chunk_size: rag_node.chunk_size,
chunk_overlap: rag_node.chunk_overlap,
reranker_model: rag_node.reranker_model.clone(),
top_k: rag_node.top_k,
batch_size: rag_node.batch_size,
extractor_model: rag_node.extractor_model.clone(),
extractor_prompt: rag_node.extractor_prompt.clone(),
graph_hops: rag_node.graph_hops,
driver: rag_node.driver.clone(),
}
}
#[allow(clippy::too_many_arguments)]
async fn init_graph_rags(
app: &AppConfig,
@@ -964,26 +1009,28 @@ async fn init_graph_rags(
};
let rag = if rag_path.exists() {
let app_clone = app.clone();
let vault_clone = app_state.vault.clone();
let path_clone = rag_path.clone();
let name_clone = node_id.clone();
app_state
.rag_cache
.load_with(key, || async move {
Rag::load(&app_clone, &name_clone, &path_clone)
Rag::load_async(&app_clone, &vault_clone, &name_clone, &path_clone).await
})
.await?
} else {
let config = RagInitConfig {
embedding_model: rag_node.embedding_model.clone(),
chunk_size: rag_node.chunk_size,
chunk_overlap: rag_node.chunk_overlap,
reranker_model: rag_node.reranker_model.clone(),
top_k: rag_node.top_k,
batch_size: rag_node.batch_size,
extractor_model: rag_node.extractor_model.clone(),
extractor_prompt: rag_node.extractor_prompt.clone(),
graph_hops: rag_node.graph_hops,
};
// Checked before anything is built: an unknown driver would otherwise
// fall through `Rag::create`'s catch-all to a yaml store, embed every
// document, and persist the bogus driver string. The RAG would then be
// rejected on every subsequent load, leaving the agent unstartable.
// Graph validation catches this too, but it is skipped when
// `validate_before_run` is off, so this guard is the load-bearing one.
if let Some(driver) = &rag_node.driver
&& let Some(message) = graph::validator::rag_driver_error(driver)
{
bail!("rag node '{node_id}': {message}");
}
let mut config = rag_init_config(rag_node);
let fully_specified = config.embedding_model.is_some()
&& config.chunk_size.is_some()
&& config.chunk_overlap.is_some();
@@ -1009,6 +1056,10 @@ async fn init_graph_rags(
initialized. RAG initialization is required for this agent."
);
}
if config.driver.is_none() {
config.driver = Some(rag::select_rag_driver()?);
}
}
let document_paths =
@@ -1317,4 +1368,39 @@ version: "1.0"
assert_eq!(meta.description, "");
}
#[test]
fn rag_init_config_forwards_an_explicit_driver() {
let node: RagNode =
serde_yaml::from_str("documents: [\"./docs\"]\ndriver: duckdb\n").unwrap();
assert_eq!(rag_init_config(&node).driver.as_deref(), Some("duckdb"));
}
/// A node that names no driver must forward `None`, which `RagInitConfig`
/// documents as "yaml". Existing workflows therefore keep their yaml store.
#[test]
fn rag_init_config_leaves_the_driver_unset_by_default() {
let node: RagNode = serde_yaml::from_str("documents: [\"./docs\"]\n").unwrap();
assert_eq!(rag_init_config(&node).driver, None);
}
/// The driver must ride alongside the rest of the node's settings, not
/// replace them.
#[test]
fn rag_init_config_forwards_the_other_settings_too() {
let node: RagNode = serde_yaml::from_str(
"documents: [\"./docs\"]\ndriver: duckdb\nchunk_size: 512\nchunk_overlap: 64\ntop_k: 7\nembedding_model: some:model\n",
)
.unwrap();
let config = rag_init_config(&node);
assert_eq!(config.driver.as_deref(), Some("duckdb"));
assert_eq!(config.chunk_size, Some(512));
assert_eq!(config.chunk_overlap, Some(64));
assert_eq!(config.top_k, Some(7));
assert_eq!(config.embedding_model.as_deref(), Some("some:model"));
}
}
+101 -1
View File
@@ -16,7 +16,7 @@ use anyhow::{Context, Result, anyhow, bail};
use log::LevelFilter;
use std::collections::HashSet;
use std::env;
use std::fs::{read_dir, read_to_string};
use std::fs::{read_dir, read_to_string, remove_file};
use std::path::{Path, PathBuf};
pub fn config_dir() -> PathBuf {
@@ -414,6 +414,10 @@ pub fn list_rags() -> Vec<String> {
for entry in rd.flatten() {
let name = entry.file_name();
if let Some(name) = name.to_string_lossy().strip_suffix(".yaml") {
if is_rag_sidecar_name(name) {
continue;
}
names.push(name.to_string());
}
}
@@ -424,6 +428,34 @@ pub fn list_rags() -> Vec<String> {
}
}
pub(crate) fn is_rag_sidecar_name(name: &str) -> bool {
name.ends_with(".sbx-mixin")
}
pub(crate) fn remove_rag_sidecars(dir: &Path, name: &str) -> Result<()> {
let duckdb_path = dir.join(format!("{name}.duckdb"));
if duckdb_path.exists() {
let _ = remove_file(&duckdb_path);
}
let wal_path = dir.join(format!("{name}.duckdb.wal"));
if wal_path.exists() {
let _ = remove_file(&wal_path);
}
let mixin_path = dir.join(format!("{name}.sbx-mixin.yaml"));
if mixin_path.exists() {
remove_file(&mixin_path).with_context(|| {
format!(
"Failed to remove the sandbox mixin for RAG '{name}' at '{}'. \
The RAG was NOT deleted so you can retry; this host remains \
whitelisted in the sandbox until the file is removed.",
mixin_path.display()
)
})?;
}
Ok(())
}
pub fn list_macros() -> Vec<String> {
list_file_names(macros_dir(), ".yaml")
}
@@ -846,4 +878,72 @@ mod tests {
}
let _ = fs::remove_dir_all(&root);
}
fn sidecar_temp_dir(label: &str) -> PathBuf {
let unique = time::SystemTime::now()
.duration_since(time::UNIX_EPOCH)
.unwrap()
.as_nanos();
let root = env::temp_dir().join(format!("coyote-{label}-test-{unique}"));
fs::create_dir_all(&root).unwrap();
root
}
#[test]
fn is_rag_sidecar_name_accepts_dotted_rag_names() {
assert!(!is_rag_sidecar_name("v2.docs"));
assert!(!is_rag_sidecar_name("myrag"));
assert!(is_rag_sidecar_name("myrag.sbx-mixin"));
assert!(is_rag_sidecar_name("v2.docs.sbx-mixin"));
}
#[test]
fn remove_rag_sidecars_removes_duckdb_wal_and_mixin() {
let root = sidecar_temp_dir("rag-sidecars-both");
let duckdb = root.join("docs.duckdb");
let wal = root.join("docs.duckdb.wal");
let mixin = root.join("docs.sbx-mixin.yaml");
fs::write(&duckdb, "db").unwrap();
fs::write(&wal, "wal").unwrap();
fs::write(&mixin, "mixin").unwrap();
remove_rag_sidecars(&root, "docs").unwrap();
assert!(!duckdb.exists(), "the .duckdb sidecar must be removed");
assert!(!wal.exists(), "the .duckdb.wal sidecar must be removed");
assert!(
!mixin.exists(),
"the .sbx-mixin.yaml sidecar must be removed"
);
let _ = fs::remove_dir_all(&root);
}
#[test]
fn remove_rag_sidecars_is_ok_when_absent() {
let root = sidecar_temp_dir("rag-sidecars-absent");
assert!(remove_rag_sidecars(&root, "docs").is_ok());
let _ = fs::remove_dir_all(&root);
}
#[test]
fn remove_rag_sidecars_runs_before_yaml_unlink() {
let root = sidecar_temp_dir("rag-sidecars-order");
let yaml = root.join("docs.yaml");
fs::write(&yaml, "rag").unwrap();
let mixin = root.join("docs.sbx-mixin.yaml");
fs::create_dir_all(&mixin).unwrap();
fs::write(mixin.join("blocker"), "x").unwrap();
let err = remove_rag_sidecars(&root, "docs").unwrap_err();
assert!(
err.to_string()
.contains("Failed to remove the sandbox mixin"),
"got: {err}"
);
assert!(
yaml.exists(),
"the .yaml must survive a sidecar-removal failure so the delete is retryable"
);
let _ = fs::remove_dir_all(&root);
}
}
+194 -26
View File
@@ -16,8 +16,9 @@ use super::{MessageContentToolCalls, prompts};
use crate::client::{Model, ModelType, list_models};
use crate::function::{
FunctionDeclaration, Functions, ToolCallTracker, ToolResult, memory::MEMORY_FUNCTION_PREFIX,
skill::SKILL_FUNCTION_PREFIX, supervisor::SUPERVISOR_FUNCTION_PREFIX,
todo::TODO_FUNCTION_PREFIX, user_interaction::USER_FUNCTION_PREFIX,
rag_query::RAG_FUNCTION_PREFIX, skill::SKILL_FUNCTION_PREFIX,
supervisor::SUPERVISOR_FUNCTION_PREFIX, todo::TODO_FUNCTION_PREFIX,
user_interaction::USER_FUNCTION_PREFIX,
};
use crate::mcp::{
MCP_DESCRIBE_META_FUNCTION_NAME_PREFIX, MCP_INVOKE_META_FUNCTION_NAME_PREFIX,
@@ -142,6 +143,7 @@ pub struct RequestContext {
pub role: Option<Role>,
pub session: Option<Session>,
pub rag: Option<Arc<Rag>>,
pub rag_key: Option<RagKey>,
pub agent: Option<Agent>,
pub last_message: Option<LastMessage>,
@@ -176,6 +178,7 @@ impl RequestContext {
role: None,
session: None,
rag: None,
rag_key: None,
agent: None,
last_message: None,
tool_scope: ToolScope::default(),
@@ -229,6 +232,7 @@ impl RequestContext {
role: None,
session: None,
rag: None,
rag_key: None,
agent: None,
last_message: None,
tool_scope: ToolScope {
@@ -277,6 +281,7 @@ impl RequestContext {
role: self.role.clone(),
session: self.session.clone(),
rag: self.rag.clone(),
rag_key: self.rag_key.clone(),
agent: self.agent.clone(),
last_message: self.last_message.clone(),
tool_scope: self.tool_scope.clone(),
@@ -315,6 +320,7 @@ impl RequestContext {
role: None,
session: None,
rag: None,
rag_key: None,
agent: None,
last_message: None,
tool_scope: ToolScope {
@@ -710,6 +716,7 @@ impl RequestContext {
pub fn exit_rag(&mut self) -> Result<()> {
self.rag.take();
self.tool_scope.functions.remove_rag_query_functions();
Ok(())
}
@@ -1132,6 +1139,7 @@ impl RequestContext {
&& !v.name.starts_with("agent__")
&& !v.name.starts_with("memory__")
&& !v.name.starts_with("skill__")
&& !v.name.starts_with("rag__")
})
.map(|v| v.name.clone())
.collect()
@@ -1952,7 +1960,8 @@ impl RequestContext {
|| (!matches!(role.skills_enabled(), Some(false))
&& v.name.starts_with(SKILL_FUNCTION_PREFIX))
|| (self.auto_continue_config().enabled
&& v.name.starts_with(TODO_FUNCTION_PREFIX)))
&& v.name.starts_with(TODO_FUNCTION_PREFIX))
|| v.name.starts_with(RAG_FUNCTION_PREFIX))
&& !existing.contains(&v.name)
})
.cloned()
@@ -1982,6 +1991,7 @@ impl RequestContext {
|| v.name.starts_with(TODO_FUNCTION_PREFIX)
|| v.name.starts_with(SUPERVISOR_FUNCTION_PREFIX)
|| v.name.starts_with(MEMORY_FUNCTION_PREFIX)
|| v.name.starts_with(RAG_FUNCTION_PREFIX)
});
}
@@ -2554,6 +2564,14 @@ impl RequestContext {
match file_ext {
Some(file_ext) => {
if let Some(name) = name.to_string_lossy().strip_suffix(file_ext) {
// Sidecars are not independently deletable assets.
// Guarded on `kind == "rag"` because this scan is shared
// by all six kinds, and `session`/`macro` also use
// `.yaml`. The helper lives in paths.rs beside
// list_rags() so both filters cannot drift apart.
if kind == "rag" && paths::is_rag_sidecar_name(name) {
continue;
}
names.push(name.to_string());
}
}
@@ -2590,6 +2608,13 @@ impl RequestContext {
match file_ext {
Some(ext) => {
let path = dir.join(format!("{name}{ext}"));
// Sidecars FIRST. If this fails, the .yaml is still on disk, the
// RAG is still listed, and the user can retry. Unlinking the .yaml
// first would make the deletion unretryable while leaving an
// orphaned mixin whitelisting a host in every sandbox launch.
if kind == "rag" {
paths::remove_rag_sidecars(&dir, &name)?;
}
remove_file(&path).with_context(|| {
format!("Failed to delete {kind} at '{}'", path.display())
})?;
@@ -2773,7 +2798,12 @@ impl RequestContext {
}
}
"rag_top_k" => {
let value = value.parse().with_context(|| "Invalid value")?;
let value: usize = value.parse().with_context(|| "Invalid value")?;
if value == 0 {
bail!(
"rag_top_k must be >= 1; a top_k of 0 makes every query return no results."
);
}
if !self.set_rag_top_k(value)? {
self.update_app_config(|app| app.rag_top_k = value);
}
@@ -3442,6 +3472,12 @@ impl RequestContext {
if self.should_register_memory_tools() {
functions.append_memory_functions();
}
if self.rag.is_some()
&& app.function_calling_support
&& !self.agent.as_ref().is_some_and(|a| a.is_graph())
{
functions.append_rag_query_functions();
}
let tool_tracker = self.tool_scope.tool_tracker.clone();
self.tool_scope = ToolScope {
@@ -3724,6 +3760,14 @@ impl RequestContext {
.then(|| Arc::new(RwLock::new(Supervisor::new(max_concurrent, max_depth))));
self.rag = agent.rag();
// Keep `rag_key` in lockstep with `rag`. Agent RAGs are cached under
// `RagKey::Agent(<agent name>)` (see `Agent::init`), so mirror that key exactly;
// leaving the previous key in place would let `.rebuild rag` invalidate an
// unrelated RAG's cache entry, and leaving it `None` would invalidate nothing.
self.rag_key = self
.rag
.is_some()
.then(|| RagKey::Agent(agent.name().to_string()));
self.agent = Some(agent);
self.supervisor = supervisor;
self.inbox = None;
@@ -3772,6 +3816,11 @@ impl RequestContext {
self.pending_agents_guardrail_count = 0;
self.todo_list = TodoList::default();
self.rag.take();
// Cleared alongside `rag` so the pair never disagrees: an agent RAG is
// cached under `RagKey::Agent(<agent name>)`, and leaving that key behind
// would outlive the RAG it names. Latent rather than live today only
// because `rebuild_rag`/`edit_rag_docs` bail on `rag.is_none()` first.
self.rag_key = None;
self.discontinuous_last_message();
}
Ok(())
@@ -4079,10 +4128,11 @@ impl RequestContext {
}
let app = self.app.config.clone();
let vault = self.app.vault.clone();
let rag_cache = self.rag_cache();
let working_mode = self.working_mode;
let rag: Arc<Rag> = match rag {
let (rag, rag_key): (Arc<Rag>, Option<RagKey>) = match rag {
None => {
let rag_path = self.rag_file(super::TEMP_RAG_NAME);
if rag_path.exists() {
@@ -4090,15 +4140,29 @@ impl RequestContext {
format!("Failed to cleanup previous '{}' rag", super::TEMP_RAG_NAME)
})?;
}
Arc::new(Rag::init(&app, super::TEMP_RAG_NAME, &rag_path, &[], abort_signal).await?)
(
Arc::new(
Rag::init(
&app,
super::TEMP_RAG_NAME,
&rag_path,
&[],
abort_signal.clone(),
false,
)
.await?,
),
None,
)
}
Some(name) => {
let rag_path = self.rag_file(name);
let key = RagKey::Named(name.to_string());
rag_cache
.load_with(key, || {
let loaded = rag_cache
.load_with(key.clone(), || {
let app = app.clone();
let vault = vault.clone();
let rag_path = rag_path.clone();
let abort_signal = abort_signal.clone();
async move {
@@ -4106,16 +4170,41 @@ impl RequestContext {
if working_mode.is_cmd() {
bail!("Unknown RAG '{name}'");
}
Rag::init(&app, name, &rag_path, &[], abort_signal.clone()).await
Rag::init(&app, name, &rag_path, &[], abort_signal.clone(), true)
.await
} else {
Rag::load(&app, name, &rag_path)
Rag::load_async(&app, &vault, name, &rag_path).await
}
}
})
.await?
.await?;
(loaded, Some(key))
}
};
self.rag = Some(rag);
self.rag_key = rag_key;
self.refresh_tool_scope(abort_signal).await?;
Ok(())
}
pub async fn attach_rag(&mut self, name: &str, abort_signal: AbortSignal) -> Result<()> {
let rag_path = self.rag_file(name);
if rag_path.exists() {
bail!(
"RAG '{name}' already exists at '{}'. \
Use a different name, or delete the existing file first.",
rag_path.display()
);
}
let app = self.app.config.as_ref();
let vault = self.app.vault.clone();
let rag = Rag::attach(app, &vault, name, &rag_path).await?;
let rag = Arc::new(rag);
let key = RagKey::Named(name.to_string());
self.rag_cache().insert(key.clone(), &rag);
self.rag = Some(rag);
self.rag_key = Some(key);
self.refresh_tool_scope(abort_signal).await?;
Ok(())
}
@@ -4125,6 +4214,12 @@ impl RequestContext {
None => bail!("No RAG"),
};
if rag.is_attached() {
bail!(
"Cannot edit documents on an attached RAG; Coyote does not own its source documents."
);
}
let document_paths = rag.document_paths();
let temp_file = temp_file(&format!("-rag-{}", rag.name()), ".txt");
tokio::fs::write(&temp_file, &document_paths.join("\n"))
@@ -4150,15 +4245,18 @@ impl RequestContext {
bail!("No changes")
}
let key = if self.agent.is_some() {
RagKey::Agent(rag.name().to_string())
} else {
RagKey::Named(rag.name().to_string())
};
self.rag_cache().invalidate(&key);
if let Some(key) = self.rag_key.clone() {
self.rag_cache().invalidate(&key);
}
rag.refresh_document_paths(&new_document_paths, false, &self.app.config, abort_signal)
.await?;
rag.refresh_document_paths(
&new_document_paths,
false,
false,
&self.app.config,
abort_signal,
)
.await?;
self.rag = Some(Arc::new(rag));
Ok(())
}
@@ -4169,15 +4267,25 @@ impl RequestContext {
None => bail!("No RAG"),
};
let key = if self.agent.is_some() {
RagKey::Agent(rag.name().to_string())
} else {
RagKey::Named(rag.name().to_string())
};
self.rag_cache().invalidate(&key);
if rag.is_attached() {
bail!(
"Cannot rebuild an attached RAG; Coyote does not own its source documents. \
Re-index from the system that originally created '{}'.",
rag.name()
);
}
if let Some(key) = self.rag_key.clone() {
self.rag_cache().invalidate(&key);
}
let document_paths = rag.document_paths().to_vec();
rag.refresh_document_paths(&document_paths, true, &self.app.config, abort_signal)
println!(
"Rebuilding re-embeds every document ({} files). \
This will call the embedding API and may take a while.",
rag.file_count()
);
rag.refresh_document_paths(&document_paths, true, true, &self.app.config, abort_signal)
.await?;
self.rag = Some(Arc::new(rag));
Ok(())
@@ -4583,6 +4691,44 @@ mod tests {
assert!(ctx.agent.is_none());
assert!(ctx.rag.is_none());
assert_eq!(ctx.rag_key, None);
}
#[test]
#[serial]
fn use_agent_does_not_carry_stale_rag_key() {
let _guard = TestConfigDirGuard::new();
let mut ctx = create_test_ctx();
let app = ctx.app.config.clone();
let agent_name = format!(
"test_agent_{}",
SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos()
);
let agent_dir = paths::agent_data_dir(&agent_name);
create_dir_all(&agent_dir).unwrap();
write(
agent_dir.join("config.yaml"),
format!("name: {agent_name}\ninstructions: hi\n"),
)
.unwrap();
ctx.rag_key = Some(RagKey::Named("docs".to_string()));
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap()
.block_on(async {
ctx.use_agent(&app, &agent_name, None, utils::create_abort_signal())
.await
.unwrap();
});
assert!(ctx.rag.is_none());
assert_eq!(ctx.rag_key, None);
}
#[test]
@@ -5970,6 +6116,28 @@ mod tests {
assert!(paths::list_rags().is_empty());
}
#[test]
#[serial]
fn list_rags_skips_sbx_mixin_sidecars() {
let _guard = TestConfigDirGuard::new();
let rags_dir = paths::rags_dir();
create_dir_all(&rags_dir).unwrap();
write(rags_dir.join("docs.yaml"), "embedding_model: test").unwrap();
write(rags_dir.join("docs.sbx-mixin.yaml"), "kind: mixin").unwrap();
write(rags_dir.join("v2.docs.yaml"), "embedding_model: test").unwrap();
let names = paths::list_rags();
assert!(names.contains(&"docs".to_string()));
assert!(
names.contains(&"v2.docs".to_string()),
"a dotted RAG name must still be listed: {names:?}"
);
assert!(
!names.contains(&"docs.sbx-mixin".to_string()),
"the sandbox mixin sidecar must not appear as a RAG: {names:?}"
);
}
#[test]
#[serial]
fn use_agent_errors_when_already_in_session() {
+21
View File
@@ -1,4 +1,5 @@
pub(crate) mod memory;
pub(crate) mod rag_query;
pub(crate) mod skill;
pub(crate) mod supervisor;
pub(crate) mod todo;
@@ -23,6 +24,7 @@ use futures_util::future;
use indexmap::IndexMap;
use indoc::formatdoc;
use memory::MEMORY_FUNCTION_PREFIX;
use rag_query::RAG_FUNCTION_PREFIX;
use rust_embed::Embed;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
@@ -495,6 +497,16 @@ impl Functions {
.extend(user_interaction::user_interaction_function_declarations());
}
pub fn append_rag_query_functions(&mut self) {
self.declarations
.extend(rag_query::rag_query_function_declarations());
}
pub fn remove_rag_query_functions(&mut self) {
self.declarations
.retain(|f| !f.name.starts_with(RAG_FUNCTION_PREFIX));
}
pub fn append_mcp_meta_functions(&mut self, mcp_servers: Vec<String>) {
let mut invoke_function_properties = IndexMap::new();
invoke_function_properties.insert(
@@ -1252,6 +1264,15 @@ impl ToolCall {
json!({"tool_call_error": error_msg})
})
}
_ if cmd_name.starts_with(RAG_FUNCTION_PREFIX) => {
rag_query::handle_rag_tool(ctx, &cmd_name, &json_data)
.await
.unwrap_or_else(|e| {
let error_msg = format!("RAG query failed: {e}");
eprintln!("{}", muted_warning_text(&format!("⚠️ {error_msg} ⚠️")));
json!({"tool_call_error": error_msg})
})
}
_ => match run_llm_function(cmd_name, cmd_args, envs, agent_name) {
Ok(Some(contents)) => serde_json::from_str(&contents)
.ok()
+101
View File
@@ -0,0 +1,101 @@
use super::{FunctionDeclaration, JsonSchema};
use crate::config::RequestContext;
use anyhow::{Result, anyhow};
use indexmap::IndexMap;
use serde_json::{Value, json};
pub const RAG_FUNCTION_PREFIX: &str = "rag__";
pub fn rag_query_function_declarations() -> Vec<FunctionDeclaration> {
vec![FunctionDeclaration {
name: format!("{RAG_FUNCTION_PREFIX}query"),
description: "Search the RAG knowledge base attached to this session and return \
the most relevant text chunks with their source paths. The relevant \
context has already been injected into the prompt up-front; use this \
tool to pull additional context on-demand when the initial retrieval \
does not fully answer the question. Prefer specific, keyword-rich queries."
.to_string(),
parameters: JsonSchema {
type_value: Some("object".to_string()),
properties: Some(IndexMap::from([
(
"query".to_string(),
JsonSchema {
type_value: Some("string".to_string()),
description: Some(
"Natural language search query used to retrieve relevant chunks."
.into(),
),
..Default::default()
},
),
(
"top_k".to_string(),
JsonSchema {
type_value: Some("integer".to_string()),
description: Some(
"Maximum number of chunks to return. Defaults to the RAG's \
configured top_k when omitted."
.into(),
),
..Default::default()
},
),
])),
required: Some(vec!["query".to_string()]),
..Default::default()
},
agent: false,
}]
}
pub async fn handle_rag_tool(
ctx: &mut RequestContext,
cmd_name: &str,
args: &Value,
) -> Result<Value> {
let action = cmd_name
.strip_prefix(RAG_FUNCTION_PREFIX)
.unwrap_or(cmd_name);
match action {
"query" => handle_query(ctx, args).await,
_ => Err(anyhow!("Unknown RAG action: {action}")),
}
}
async fn handle_query(ctx: &RequestContext, args: &Value) -> Result<Value> {
let rag = ctx
.rag
.clone()
.ok_or_else(|| anyhow!("No RAG is attached to this session"))?;
let query = args
.get("query")
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("'query' is required"))?;
let top_k = args
.get("top_k")
.and_then(Value::as_u64)
.map(|v| v as usize)
.unwrap_or_else(|| rag.configured_top_k());
let rerank_model = rag.configured_reranker().map(|s| s.to_string());
let chunks = rag
.search_chunks(query, top_k, rerank_model.as_deref())
.await?;
let chunks_json: Vec<Value> = chunks
.into_iter()
.map(|(text, source)| json!({ "text": text, "source": source }))
.collect();
Ok(json!({
"rag_name": rag.name(),
"count": chunks_json.len(),
"chunks": chunks_json,
}))
}
+103
View File
@@ -367,6 +367,13 @@ pub struct RagNode {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub graph_hops: Option<usize>,
/// Storage driver for this node's knowledge base ("yaml", "duckdb"). `None`
/// means "yaml". Only honored when the knowledge base is first built;
/// changing it afterwards has no effect until the RAG is deleted and
/// re-initialized.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub driver: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub state_updates: Option<HashMap<String, String>>,
@@ -1152,4 +1159,100 @@ nodes:
assert!(triage.next.as_ref().unwrap().is_fan_out());
assert_eq!(triage.next.as_ref().unwrap().as_slice().len(), 2);
}
fn rag_node_of(graph: &Graph, id: &str) -> RagNode {
match &graph.get_node(id).unwrap().node_type {
NodeType::Rag(r) => r.clone(),
other => panic!("expected a rag node, got {other:?}"),
}
}
#[test]
fn rag_node_deserializes_an_explicit_driver() {
let yaml = r#"
name: kb
start: research
nodes:
research:
type: rag
documents: ["./docs"]
driver: duckdb
next: done
done:
type: end
output: ok
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
assert_eq!(
rag_node_of(&graph, "research").driver.as_deref(),
Some("duckdb")
);
}
/// Workflows written before drivers existed must keep parsing, and must keep
/// asking for nothing, so `RagInitConfig` resolves them to the yaml default.
#[test]
fn rag_node_without_a_driver_stays_unset() {
let yaml = r#"
name: kb
start: research
nodes:
research:
type: rag
documents: ["./docs"]
next: done
done:
type: end
output: ok
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
assert_eq!(rag_node_of(&graph, "research").driver, None);
}
#[test]
fn rag_node_driver_survives_a_serialize_round_trip() {
let yaml = r#"
name: kb
start: research
nodes:
research:
type: rag
documents: ["./docs"]
driver: duckdb
next: done
done:
type: end
output: ok
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
let reparsed: Graph =
serde_yaml::from_str(&serde_yaml::to_string(&graph).unwrap()).unwrap();
assert_eq!(
rag_node_of(&reparsed, "research").driver.as_deref(),
Some("duckdb")
);
}
/// `skip_serializing_if` must keep `driver:` out of graphs that never set it.
#[test]
fn rag_node_without_a_driver_omits_the_key_when_serialized() {
let yaml = r#"
name: kb
start: research
nodes:
research:
type: rag
documents: ["./docs"]
next: done
done:
type: end
output: ok
"#;
let graph: Graph = serde_yaml::from_str(yaml).unwrap();
assert!(!serde_yaml::to_string(&graph).unwrap().contains("driver"));
}
}
+101
View File
@@ -2,6 +2,7 @@ use super::state::template_root_keys;
use super::types::{Graph, Node, NodeType};
use crate::client::{Model, ModelType};
use crate::config::{Agent, AppConfig, paths};
use crate::rag::{GraphRagConfig, RagData};
use anyhow::{Result, bail};
use std::collections::{BTreeMap, HashSet, VecDeque};
use std::path::PathBuf;
@@ -96,6 +97,51 @@ pub struct GraphValidator {
skill_exists: fn(&str) -> bool,
}
/// A minimal `RagData` whose only interesting field is `driver`. The numeric
/// arguments are the smallest values that satisfy `validate()`'s unrelated
/// floors (top_k >= 1, and chunk_size >= 1 with chunk_overlap < chunk_size for
/// a non-attached RAG). `RagData::new` sets `attached: false`, which is the
/// correct shape here: a graph rag node always builds its own local knowledge
/// base from `documents` and can never be attached.
fn rag_driver_probe(driver: &str) -> RagData {
let mut data = RagData::new(
String::new(),
1,
0,
None,
1,
None,
GraphRagConfig::default(),
);
data.driver = driver.to_string();
data
}
/// `Some(message)` when `driver` is one that `RagData::validate()` would reject.
///
/// The set of valid drivers is defined in exactly one place, `RagData::validate()`,
/// so this asks that function rather than restating the list here.
///
/// Fails open on purpose: the first probe below uses the default driver, which is
/// valid by definition. If even that one is rejected, `validate()` has grown a
/// precondition the probe fixture no longer satisfies, and every verdict from here
/// would be a false positive that rejects working graphs. In that case we decline
/// to judge and leave enforcement to RAG construction. The
/// `rag_driver_probe_fixture_is_accepted` test turns that silent degradation into a
/// loud failure. Both `validate()` calls are load-bearing; neither is redundant.
pub(crate) fn rag_driver_error(driver: &str) -> Option<String> {
if rag_driver_probe(&RagData::default().driver)
.validate()
.is_err()
{
return None;
}
rag_driver_probe(driver)
.validate()
.err()
.map(|err| err.to_string())
}
impl GraphValidator {
pub fn new(base_dir: impl Into<PathBuf>) -> Self {
Self {
@@ -154,6 +200,11 @@ impl GraphValidator {
not be written to state",
));
}
if let Some(driver) = &r.driver
&& let Some(message) = rag_driver_error(driver)
{
result.error(ValidationError::with_node(node_id, message));
}
}
}
}
@@ -1031,6 +1082,7 @@ mod tests {
extractor_model: None,
extractor_prompt: None,
graph_hops: None,
driver: None,
state_updates,
timeout: None,
}),
@@ -1385,6 +1437,55 @@ mod tests {
);
}
/// Guards the fail-open branch in `rag_driver_error`. If this fails,
/// `RagData::validate()` grew a precondition the probe fixture no longer
/// satisfies and rag-node driver validation has silently switched itself off.
/// Repair the fixture in `rag_driver_probe`; do not delete this test.
#[test]
fn rag_driver_probe_fixture_is_accepted() {
let default_driver = RagData::default().driver;
assert!(
rag_driver_probe(&default_driver).validate().is_ok(),
"probe fixture rejected for the default driver '{default_driver}'"
);
}
#[test]
fn rag_driver_error_defers_to_ragdata_validate() {
assert_eq!(rag_driver_error("yaml"), None);
assert_eq!(rag_driver_error("duckdb"), None);
let message = rag_driver_error("duckdbb").expect("unknown driver must be rejected");
assert!(message.contains("duckdbb"), "got: {message}");
}
#[test]
fn rag_node_with_unknown_driver_errors_naming_the_node() {
let mut node = rag_node("kb", &["./docs"], true);
if let NodeType::Rag(ref mut r) = node.node_type {
r.driver = Some("postgres".into());
}
let graph = graph_with(vec![("kb", node), ("end", end_node("end"))], "kb");
let result = validator().validate(&graph);
assert!(!result.is_valid());
let err = result.into_result().unwrap_err().to_string();
assert!(err.contains("[kb]"), "must name the node: {err}");
assert!(err.contains("postgres"), "must name the driver: {err}");
}
#[test]
fn rag_node_with_duckdb_driver_produces_no_findings() {
let mut node = rag_node("kb", &["./docs"], true);
if let NodeType::Rag(ref mut r) = node.node_type {
r.driver = Some("duckdb".into());
}
let graph = graph_with(vec![("kb", node), ("end", end_node("end"))], "kb");
assert!(validator().validate(&graph).is_valid());
}
fn agent_node(id: &str, agent: &str, next: Option<&str>) -> Node {
Node {
id: id.into(),
+12
View File
@@ -196,6 +196,18 @@ async fn main() -> Result<()> {
return Ok(());
}
let mcp_action =
cli.mcp_list || cli.mcp_get.is_some() || cli.mcp_remove.is_some() || cli.mcp_add.is_some();
if mcp_action {
let cfg = Config::load_with_interpolation(true).await?;
let app_config = AppConfig::from_config(cfg)?;
let vault = Vault::init(&app_config)?;
mcp::manage::handle(&cli, &vault)?;
return Ok(());
}
if vault_flags {
let cfg = Config::load_with_interpolation(true).await?;
let app_config = AppConfig::from_config(cfg)?;
+443
View File
@@ -0,0 +1,443 @@
use crate::cli::{Cli, McpScopeArg, McpTransportArg};
use crate::config::{ensure_parent_exists, paths};
use crate::mcp::{JsonField, McpOAuthConfig, McpServer, McpServersConfig, McpTransportType};
use crate::vault::{SECRET_RE, Vault};
use anyhow::{Context, Result, anyhow, bail};
use indexmap::{IndexMap, IndexSet};
use inquire::Confirm;
use std::collections::HashSet;
use std::fs;
use std::path::{Path, PathBuf};
impl From<McpTransportArg> for McpTransportType {
fn from(value: McpTransportArg) -> Self {
match value {
McpTransportArg::Stdio => McpTransportType::Stdio,
McpTransportArg::Http => McpTransportType::Http,
McpTransportArg::Sse => McpTransportType::Sse,
}
}
}
pub fn handle(cli: &Cli, vault: &Vault) -> Result<()> {
if cli.mcp_list {
return handle_list(cli.scope);
}
if let Some(name) = &cli.mcp_get {
return handle_get(name, cli.scope);
}
if let Some(name) = &cli.mcp_remove {
return handle_remove(name, cli.scope, cli.mcp_force);
}
if let Some(name) = &cli.mcp_add {
return handle_add(cli, name, vault);
}
Ok(())
}
fn handle_list(scope: Option<McpScopeArg>) -> Result<()> {
let show_user = scope != Some(McpScopeArg::Workspace);
let show_workspace = scope != Some(McpScopeArg::User);
if show_user {
let user_path = paths::mcp_config_file();
let user_cfg = load_config_raw(&user_path)?;
println!("User ({})", user_path.display());
print_server_list(&user_cfg);
}
if show_workspace {
match paths::workspace_mcp_config_file() {
Some(ws_path) => {
let ws_cfg = load_config_raw(&ws_path)?;
if show_user {
println!();
}
println!("Workspace ({})", ws_path.display());
print_server_list(&ws_cfg);
}
None if scope == Some(McpScopeArg::Workspace) => {
println!("Workspace: no mcp.json found in current directory");
}
None => {}
}
}
Ok(())
}
fn print_server_list(cfg: &McpServersConfig) {
if cfg.mcp_servers.is_empty() {
println!(" (none)");
return;
}
let name_width = cfg.mcp_servers.keys().map(String::len).max().unwrap_or(0);
for (name, spec) in &cfg.mcp_servers {
let transport = match spec.transport_type {
McpTransportType::Stdio => "stdio",
McpTransportType::Http => "http",
McpTransportType::Sse => "sse",
};
let target = spec.url.clone().unwrap_or_else(|| {
let cmd = spec.command.clone().unwrap_or_default();
let args = spec.args.as_ref().map(|a| a.join(" ")).unwrap_or_default();
if args.is_empty() {
cmd
} else {
format!("{cmd} {args}")
}
});
println!(
" {name:<name_width$} {transport:<5} {target}",
name_width = name_width
);
}
}
fn handle_get(name: &str, scope: Option<McpScopeArg>) -> Result<()> {
let (path, cfg) = load_for_scope_or_search(name, scope)?;
let spec = cfg
.mcp_servers
.get(name)
.ok_or_else(|| anyhow!("MCP server '{name}' not found"))?;
let pretty =
serde_json::to_string_pretty(spec).context("failed to serialize MCP server config")?;
println!("# {}", path.display());
println!("{pretty}");
Ok(())
}
fn handle_remove(name: &str, scope: Option<McpScopeArg>, force: bool) -> Result<()> {
let (path, mut cfg) = load_for_scope_or_search(name, scope)?;
if !force {
let ok = Confirm::new(&format!(
"Remove MCP server '{name}' from {}?",
path.display()
))
.with_default(false)
.prompt()?;
if !ok {
println!("Aborted.");
return Ok(());
}
}
cfg.mcp_servers.shift_remove(name);
save_config(&path, &cfg)?;
println!("✓ Removed MCP server '{name}' from {}", path.display());
Ok(())
}
fn handle_add(cli: &Cli, name: &str, vault: &Vault) -> Result<()> {
validate_name(name)?;
let server = build_server(cli)?;
server.validate(name)?;
let scope = cli.scope.unwrap_or_default();
let path = write_path_for_scope(scope);
let mut cfg = load_config_raw(&path)?;
if cfg.mcp_servers.contains_key(name) && !cli.mcp_force {
let ok = Confirm::new(&format!(
"MCP server '{name}' already exists in {}. Overwrite?",
path.display()
))
.with_default(false)
.prompt()?;
if !ok {
println!("Aborted. Use --mcp-force to overwrite without prompting.");
return Ok(());
}
}
provision_secrets(cli, vault)?;
cfg.mcp_servers.insert(name.to_string(), server);
save_config(&path, &cfg)?;
println!("✓ Added MCP server '{name}' to {}", path.display());
Ok(())
}
fn validate_name(name: &str) -> Result<()> {
if name.is_empty() {
bail!("MCP server name cannot be empty");
}
if !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
{
bail!("Invalid MCP server name '{name}': only letters, digits, '-', and '_' are allowed");
}
Ok(())
}
fn build_server(cli: &Cli) -> Result<McpServer> {
let has_command = !cli.mcp_command.is_empty();
let has_url = cli.url.is_some();
let transport = cli
.transport
.map(McpTransportType::from)
.unwrap_or_else(|| {
if has_command {
McpTransportType::Stdio
} else {
McpTransportType::Http
}
});
match transport {
McpTransportType::Stdio => build_stdio(cli, has_url),
McpTransportType::Http | McpTransportType::Sse => build_remote(cli, transport, has_command),
}
}
fn build_stdio(cli: &Cli, has_url: bool) -> Result<McpServer> {
if cli.mcp_command.is_empty() {
bail!(
"stdio MCP server requires a command. Pass it after `--`, e.g. \
`--mcp-add NAME -- npx some-server --flag`"
);
}
if has_url {
bail!("stdio MCP server does not accept --url");
}
if !cli.header.is_empty() {
bail!("stdio MCP server does not accept --header");
}
if cli.client_id.is_some()
|| cli.client_secret.is_some()
|| cli.callback_port.is_some()
|| cli.redirect_host.is_some()
{
bail!("stdio MCP server does not accept OAuth flags");
}
let (cmd, args) = cli.mcp_command.split_first().unwrap();
let mut env: IndexMap<String, JsonField> = IndexMap::new();
for kv in &cli.env {
let (k, v) = kv
.split_once('=')
.ok_or_else(|| anyhow!("invalid --env value '{kv}': expected KEY=VALUE"))?;
if k.is_empty() {
bail!("invalid --env value '{kv}': KEY cannot be empty");
}
env.insert(k.to_string(), JsonField::Str(v.to_string()));
}
Ok(McpServer {
transport_type: McpTransportType::Stdio,
command: Some(cmd.clone()),
args: (!args.is_empty()).then(|| args.to_vec()),
env: (!env.is_empty()).then_some(env),
cwd: cli.cwd.clone(),
url: None,
headers: None,
oauth: None,
})
}
fn build_remote(cli: &Cli, transport: McpTransportType, has_command: bool) -> Result<McpServer> {
if has_command {
bail!(
"http/sse MCP server does not accept a trailing `-- <cmd>`. Use `--url` \
to specify the endpoint."
);
}
let url = cli
.url
.clone()
.ok_or_else(|| anyhow!("http/sse MCP server requires --url <URL>"))?;
if !cli.env.is_empty() {
bail!("http/sse MCP server does not accept --env; use --header instead");
}
if cli.cwd.is_some() {
bail!("http/sse MCP server does not accept --cwd");
}
let mut headers: IndexMap<String, String> = IndexMap::new();
for h in &cli.header {
let (name, value) = h
.split_once(':')
.ok_or_else(|| anyhow!("invalid --header value '{h}': expected 'Name: Value'"))?;
let name = name.trim();
let value = value.trim_start_matches(' ');
if name.is_empty() {
bail!("invalid --header value '{h}': header name cannot be empty");
}
headers.insert(name.to_string(), value.to_string());
}
let oauth = if cli.client_id.is_some()
|| cli.client_secret.is_some()
|| cli.callback_port.is_some()
|| cli.redirect_host.is_some()
{
Some(McpOAuthConfig {
client_id: cli.client_id.clone(),
client_secret: cli.client_secret.clone(),
callback_port: cli.callback_port,
redirect_host: cli.redirect_host.clone(),
})
} else {
None
};
Ok(McpServer {
transport_type: transport,
command: None,
args: None,
env: None,
cwd: None,
url: Some(url),
headers: (!headers.is_empty()).then_some(headers),
oauth,
})
}
fn provision_secrets(cli: &Cli, vault: &Vault) -> Result<()> {
let mut sources: Vec<&str> = Vec::new();
if let Some(s) = cli.url.as_deref() {
sources.push(s);
}
if let Some(s) = cli.client_secret.as_deref() {
sources.push(s);
}
if let Some(s) = cli.client_id.as_deref() {
sources.push(s);
}
if let Some(s) = cli.redirect_host.as_deref() {
sources.push(s);
}
if let Some(s) = cli.cwd.as_deref() {
sources.push(s);
}
sources.extend(cli.env.iter().map(String::as_str));
sources.extend(cli.header.iter().map(String::as_str));
let mut needed: IndexSet<String> = IndexSet::new();
for value in sources {
for caps in SECRET_RE.captures_iter(value).filter_map(Result::ok) {
if let Some(m) = caps.get(1) {
needed.insert(m.as_str().trim().to_string());
}
}
}
if needed.is_empty() {
return Ok(());
}
let existing: HashSet<String> = vault.list_secrets(false)?.into_iter().collect();
for name in needed {
if existing.contains(&name) {
continue;
}
eprintln!("Value references vault secret {{{{ {name} }}}} which is not stored yet.");
let ok = Confirm::new(&format!("Add '{name}' to the vault now?"))
.with_default(true)
.prompt()?;
if !ok {
bail!(
"Vault secret '{name}' is required by the config; aborting. \
Add it later with `coyote --add-secret {name}`."
);
}
vault.add_secret(&name)?;
}
Ok(())
}
fn load_for_scope_or_search(
name: &str,
scope: Option<McpScopeArg>,
) -> Result<(PathBuf, McpServersConfig)> {
if let Some(s) = scope {
let path = match s {
McpScopeArg::User => paths::mcp_config_file(),
McpScopeArg::Workspace => paths::workspace_mcp_config_file()
.ok_or_else(|| anyhow!("no workspace mcp.json found in the current directory"))?,
};
let cfg = load_config_raw(&path)?;
if !cfg.mcp_servers.contains_key(name) {
bail!(
"MCP server '{name}' not found in {} scope ({})",
scope_label(s),
path.display()
);
}
return Ok((path, cfg));
}
let user_path = paths::mcp_config_file();
let user_cfg = load_config_raw(&user_path)?;
if user_cfg.mcp_servers.contains_key(name) {
return Ok((user_path, user_cfg));
}
if let Some(ws_path) = paths::workspace_mcp_config_file() {
let ws_cfg = load_config_raw(&ws_path)?;
if ws_cfg.mcp_servers.contains_key(name) {
return Ok((ws_path, ws_cfg));
}
}
bail!("MCP server '{name}' not found in any scope");
}
fn write_path_for_scope(scope: McpScopeArg) -> PathBuf {
match scope {
McpScopeArg::User => paths::mcp_config_file(),
McpScopeArg::Workspace => paths::workspace_mcp_config_file()
.unwrap_or_else(|| paths::workspace_config_dir().join("mcp.json")),
}
}
fn scope_label(scope: McpScopeArg) -> &'static str {
match scope {
McpScopeArg::User => "user",
McpScopeArg::Workspace => "workspace",
}
}
fn load_config_raw(path: &Path) -> Result<McpServersConfig> {
if !path.exists() {
return Ok(McpServersConfig {
mcp_servers: IndexMap::new(),
});
}
let raw = fs::read_to_string(path)
.with_context(|| format!("failed to read MCP config at {}", path.display()))?;
if raw.trim().is_empty() {
return Ok(McpServersConfig {
mcp_servers: IndexMap::new(),
});
}
serde_json::from_str(&raw)
.with_context(|| format!("failed to parse MCP config at {}", path.display()))
}
fn save_config(path: &Path, config: &McpServersConfig) -> Result<()> {
ensure_parent_exists(path)?;
let serialized =
serde_json::to_string_pretty(config).context("failed to serialize MCP config")?;
let tmp = path.with_extension("json.tmp");
fs::write(&tmp, &serialized)
.with_context(|| format!("failed to write temporary MCP config at {}", tmp.display()))?;
fs::rename(&tmp, path)
.with_context(|| format!("failed to finalize MCP config at {}", path.display()))?;
Ok(())
}
+3
View File
@@ -1,3 +1,4 @@
pub(crate) mod manage;
pub(crate) mod oauth;
mod sse_transport;
@@ -62,6 +63,8 @@ pub(crate) struct McpServersConfig {
pub(crate) struct McpOAuthConfig {
#[serde(rename = "clientId", skip_serializing_if = "Option::is_none")]
pub client_id: Option<String>,
#[serde(rename = "clientSecret", skip_serializing_if = "Option::is_none")]
pub client_secret: Option<String>,
#[serde(rename = "callbackPort", skip_serializing_if = "Option::is_none")]
pub callback_port: Option<u16>,
#[serde(rename = "redirectHost", skip_serializing_if = "Option::is_none")]
+1550 -88
View File
File diff suppressed because it is too large Load Diff
+76
View File
@@ -0,0 +1,76 @@
use super::{DocumentId, RagData};
use anyhow::Result;
use async_trait::async_trait;
/// Abstracts where RAG vector data is stored and queried.
///
/// The Rag orchestrator owns: embeddings, chunking, BM25 keyword search, graph RAG,
/// entity extraction, RRF merging. Providers own: vector storage and content retrieval.
#[async_trait]
pub trait RagProvider: Send + Sync {
/// Vector similarity search. Returns (DocumentId, score) sorted by score desc.
/// `embedding` is a single query vector from Coyote's embedding model.
async fn vector_search(
&self,
embedding: &[f32],
top_k: usize,
min_score: f32,
) -> Result<Vec<(DocumentId, f32)>>;
/// Resolve document IDs to their page content.
///
/// **Ordering contract:** implementations MUST return results in the same
/// relative order as the input `ids` slice. `hybrid_search` passes an
/// RRF-ranked list and feeds the result straight to the LLM. A provider
/// that returns rows in storage order (e.g. Qdrant `get_points`, DuckDB
/// `WHERE id IN (...)`) would silently discard the ranking. Implementations
/// that query an unordered backend must re-sort by input position before
/// returning.
///
/// Returns only IDs that were found; callers must handle partial returns
/// (a missing ID is skipped, not an error).
/// YamlProvider: reads from an in-memory content map built from data.files.
/// DuckDbProvider: queries the documents table by id.
/// QdrantProvider: fetches payload from the remote collection.
async fn fetch_content(&self, ids: &[DocumentId]) -> Result<Vec<(DocumentId, String)>>;
/// Rebuild internal indexes from freshly updated RagData.
/// Called once at the end of every sync_documents pass.
///
/// `full_rebuild` mirrors `sync_documents`' `refresh` parameter:
/// - `true`: a full re-index (`.rebuild rag`, `--rebuild-rag`, initial build).
/// Destructive strategies (wipe-then-reindex) are permitted.
/// - `false`: an incremental change (`.edit rag-docs` adding/removing a file).
/// Implementations MUST NOT wipe existing state; upsert only.
///
/// The parameter is part of the signature from the outset so it is fixed
/// while there is exactly one implementor. Yaml/DuckDb ignore it,
/// rebuilding their local state wholesale is fast and always correct.
/// Only a remote provider is destructive enough to care.
async fn rebuild_indexes(&mut self, data: &RagData, full_rebuild: bool) -> Result<()>;
/// Keyword / full-text search. Returns (DocumentId, BM25-style score) sorted desc.
///
/// Default impl returns `Ok(vec![])`. Callers fall back to `Rag.bm25` (local in-memory
/// BM25 built from `data.files`).
///
/// Callers check `has_native_keyword_search()` before deciding which path to take:
/// - true → call this method; skip `Rag.bm25`
/// - false → call `Rag.keyword_search()` which uses `Rag.bm25` (sync, infallible)
async fn keyword_search(&self, query: &str, top_k: usize) -> Result<Vec<(DocumentId, f32)>> {
let _ = (query, top_k);
Ok(vec![])
}
/// Returns true if this provider implements a native keyword-search index.
/// When false, `Rag.hybrid_search` uses the local `Rag.bm25` field instead.
fn has_native_keyword_search(&self) -> bool {
false
}
/// Deep-clone the provider with fresh indexes derived from `data`.
/// Required because Box<dyn RagProvider> is not Clone.
/// Called by Rag's Clone impl (which clones before mutating in rebuild_rag/edit_rag_docs).
fn duplicate(&self, data: &RagData) -> Box<dyn RagProvider>;
}
File diff suppressed because it is too large Load Diff
+9
View File
@@ -0,0 +1,9 @@
mod yaml;
pub use self::yaml::YamlProvider;
mod duckdb;
pub use self::duckdb::DuckDbProvider;
pub(crate) use self::duckdb::duckdb_path_from_yaml;
mod qdrant;
pub use self::qdrant::QdrantProvider;
+828
View File
@@ -0,0 +1,828 @@
use crate::rag::provider::RagProvider;
use crate::rag::{DocumentId, RagData};
use anyhow::{Context, Result, bail};
use async_trait::async_trait;
use parking_lot::RwLock;
use reqwest::header::{HeaderMap, HeaderValue};
use reqwest::{Client, Response, StatusCode};
use serde_json::Value;
use std::collections::HashMap;
use std::sync::Arc;
use url::{Host, Url};
/// Marks a `DocumentId` that stands in for a point id Coyote cannot carry
/// directly. Qdrant accepts UUID strings as point ids, and that is what
/// LangChain writes by default.
///
/// `DocumentId` packs `(file_index, document_index)` into one `usize` with the
/// file index in the high half, so this bit is only reachable at a file index of
/// 2^31. Nothing local gets near that, and an attached RAG builds no local index
/// at all — `data.files` and `data.vectors` stay empty and every
/// `DocumentId::split` caller early-returns on `data.attached`. Along the
/// attached path the id is an opaque key carried through RRF, which is what
/// makes a synthetic one safe here and nowhere else.
const SYNTHETIC_ID_TAG: usize = 1 << (usize::BITS - 1);
/// Two-way map between a raw Qdrant point id and the `DocumentId` the retrieval
/// pipeline sees.
///
/// Only ids that cannot survive the round trip are interned. A plain `u64` that
/// fits below the tag keeps mapping to itself, so integer-keyed collections
/// behave exactly as they did before this map existed.
#[derive(Default)]
struct PointIdInterner {
handles: HashMap<String, DocumentId>,
raw: HashMap<DocumentId, Value>,
next: usize,
}
impl PointIdInterner {
/// The `DocumentId` for a raw point id, minting a handle if one is needed.
///
/// `None` only for a missing id, which is a malformed response.
fn document_id(&mut self, raw: &Value) -> Option<DocumentId> {
if raw.is_null() {
return None;
}
// The pre-existing integer path, unchanged. `try_from` rather than `as`
// so a value too wide for the target's `usize` is interned instead of
// silently truncated into a different point.
if let Some(n) = raw.as_u64()
&& let Ok(n) = usize::try_from(n)
&& n & SYNTHETIC_ID_TAG == 0
{
return Some(DocumentId(n));
}
Some(self.intern(raw))
}
fn intern(&mut self, raw: &Value) -> DocumentId {
// Keyed on the JSON rendering, so the string "1" and the integer 1 are
// not conflated into one point.
let key = raw.to_string();
if let Some(handle) = self.handles.get(&key) {
return *handle;
}
let handle = DocumentId(SYNTHETIC_ID_TAG | self.next);
self.next += 1;
self.handles.insert(key, handle);
self.raw.insert(handle, raw.clone());
handle
}
/// The original id for a handle, or `None` when the id was never interned —
/// i.e. it is a plain integer that is already its own id.
fn raw_id(&self, handle: DocumentId) -> Option<&Value> {
self.raw.get(&handle)
}
/// Builds the `ids` array for an outbound `/points` fetch. Every entry is the
/// id Qdrant issued, integer or string; a synthetic handle must never leave
/// this process.
fn outbound_ids(&self, ids: &[DocumentId]) -> Vec<Value> {
ids.iter()
.map(|id| match self.raw_id(*id) {
Some(raw) => raw.clone(),
None => Value::from(id.0 as u64),
})
.collect()
}
}
fn parse_search_hits(
interner: &mut PointIdInterner,
body: &Value,
min_score: f32,
) -> Result<Vec<(DocumentId, f32)>> {
let hits = body["result"]
.as_array()
.context("Unexpected /points/search response shape")?;
Ok(hits
.iter()
.filter_map(|pt| {
let score = pt["score"].as_f64()? as f32;
Some((interner.document_id(&pt["id"])?, score))
})
.filter(|(_, score)| min_score <= 0.0 || *score > min_score)
.collect())
}
fn parse_points(interner: &mut PointIdInterner, body: &Value) -> Result<Vec<(DocumentId, String)>> {
let points = body["result"]
.as_array()
.context("Unexpected /points response shape")?;
Ok(points
.iter()
.filter_map(|pt| {
let text = pt["payload"]["page_content"].as_str()?.to_string();
Some((interner.document_id(&pt["id"])?, text))
})
.collect())
}
/// Render Qdrant's error envelope into a human-readable message.
///
/// `body` is the raw response text. Two shapes have to be tolerated:
/// * application-level errors carry `{"status": {"error": "..."}, "time": 0.0}`,
/// while successful responses carry a bare string `{"status": "ok", ...}` — so
/// `status` is string-or-object and a struct with `status: String` fails to
/// parse every error body;
/// * routing-level 404s (a wrong HTTP verb) return an EMPTY body with no JSON at
/// all, which without the length check surfaces as "EOF while parsing a value"
/// instead of the actual 404.
fn format_error_body(status: StatusCode, body: &str) -> String {
if body.is_empty() {
return format!("HTTP {status} (empty body — check the HTTP verb and path)");
}
serde_json::from_str::<Value>(body)
.ok()
.and_then(|v| v["status"]["error"].as_str().map(str::to_string))
.unwrap_or_else(|| format!("HTTP {status}: {body}"))
}
/// Read the vector dimension out of a parsed `GET /collections/{name}` response.
fn vector_dimension_from_collection(body: &Value) -> Result<u64> {
let params = &body["result"]["config"]["params"];
params["vectors"]["size"]
.as_u64()
.or_else(|| {
params["vectors"]
.as_object()
.and_then(|m| m.values().next())
.and_then(|v| v["size"].as_u64())
})
.context("Could not determine vector dimension from collection config")
}
/// True if a parsed `GET /collections/{name}` response describes a NAMED
/// (multi-vector) collection.
///
/// `vector_search` posts an unnamed vector, which a named-vector collection
/// rejects with HTTP 400 on every query, so attaching one yields a RAG that is
/// silently 100% broken. A named collection holding a SINGLE vector is
/// structurally a map, identical in kind to the multi-named case, and rejects
/// the same way; testing for a numeric `size` directly under `vectors` catches
/// it, whereas counting keys (`len() > 1`) would wrongly accept it.
fn is_multi_vector_config(body: &Value) -> bool {
body["result"]["config"]["params"]["vectors"]["size"]
.as_u64()
.is_none()
}
/// Query-only client for an external Qdrant collection.
///
/// Attach-only: this provider never writes to the remote collection. Coyote does
/// not own the data, and `rebuild_indexes` refuses rather than pretending to.
pub struct QdrantProvider {
client: Client,
base_url: String,
collection: String,
point_ids: Arc<RwLock<PointIdInterner>>,
}
impl QdrantProvider {
fn skips_proxy(base_url: &str) -> bool {
let Ok(url) = Url::parse(base_url) else {
return false;
};
match url.host() {
Some(Host::Domain(name)) => {
name == "localhost" || name.ends_with(".localhost") || name.ends_with(".local")
}
Some(Host::Ipv4(ip)) => ip.is_loopback() || ip.is_private() || ip.is_link_local(),
// No stable is_unique_local, so fc00::/7 is matched directly.
Some(Host::Ipv6(ip)) => ip.is_loopback() || ip.segments()[0] & 0xfe00 == 0xfc00,
None => false,
}
}
fn make_client(base_url: &str, api_key: Option<&str>) -> Result<Client> {
let mut headers = HeaderMap::new();
if let Some(key) = api_key {
let mut value =
HeaderValue::from_str(key).context("api-key header value is not valid ASCII")?;
value.set_sensitive(true);
headers.insert("api-key", value);
}
let mut builder = Client::builder().default_headers(headers);
if Self::skips_proxy(base_url) {
builder = builder.no_proxy();
}
builder.build().context("Failed to build reqwest client")
}
pub(crate) fn normalize_base_url(host: &str) -> String {
if host.starts_with("http://") || host.starts_with("https://") {
host.to_string()
} else {
format!("http://{host}")
}
}
async fn error_message(resp: Response) -> String {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
format_error_body(status, &body)
}
/// Shared `GET /collections/{name}` fetch. Both the dimension and the
/// multi-vector probe discriminate on this same response.
async fn fetch_collection(
host: &str,
collection: &str,
api_key: Option<&str>,
) -> Result<Value> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(&base_url, api_key)?;
let resp = client
.get(format!("{base_url}/collections/{collection}"))
.send()
.await
.with_context(|| format!("Failed to connect to {host}"))?;
if !resp.status().is_success() {
bail!(
"Failed to read collection '{collection}': {}",
Self::error_message(resp).await
);
}
Ok(resp.json().await?)
}
pub async fn new(host: &str, collection: &str, api_key: Option<&str>) -> Result<Self> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(&base_url, api_key)?;
let resp = client
.get(format!("{base_url}/collections/{collection}"))
.send()
.await
.with_context(|| format!("Failed to connect to {host}"))?;
if !resp.status().is_success() {
bail!(
"Collection '{collection}' not accessible at {host}: {}",
Self::error_message(resp).await
);
}
Ok(Self {
client,
base_url,
collection: collection.to_string(),
point_ids: Arc::default(),
})
}
pub async fn list_collections(host: &str, api_key: Option<&str>) -> Result<Vec<String>> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(&base_url, api_key)?;
let resp = client
.get(format!("{base_url}/collections"))
.send()
.await
.with_context(|| format!("Failed to connect to {host}"))?;
if !resp.status().is_success() {
bail!(
"Failed to list collections: {}",
Self::error_message(resp).await
);
}
let body: Value = resp.json().await?;
let names = body["result"]["collections"]
.as_array()
.context("Unexpected /collections response shape")?
.iter()
.filter_map(|v| v["name"].as_str().map(str::to_string))
.collect();
Ok(names)
}
pub async fn get_vector_dimension(
host: &str,
collection: &str,
api_key: Option<&str>,
) -> Result<u64> {
let body = Self::fetch_collection(host, collection, api_key).await?;
vector_dimension_from_collection(&body)
}
pub async fn is_multi_vector(
host: &str,
collection: &str,
api_key: Option<&str>,
) -> Result<bool> {
let body = Self::fetch_collection(host, collection, api_key).await?;
Ok(is_multi_vector_config(&body))
}
pub async fn sample_point_id(
host: &str,
collection: &str,
api_key: Option<&str>,
) -> Result<Option<String>> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(&base_url, api_key)?;
let url = format!("{base_url}/collections/{collection}/points/scroll");
let body = serde_json::json!({ "limit": 1, "with_payload": false });
let resp = client
.post(&url)
.json(&body)
.send()
.await
.with_context(|| format!("Failed to connect to {host}"))?;
if !resp.status().is_success() {
bail!(
"Failed to sample a point from '{collection}': {}",
Self::error_message(resp).await
);
}
let data: Value = resp.json().await?;
let id_val = data["result"]["points"]
.as_array()
.and_then(|pts| pts.first())
.map(|pt| pt["id"].to_string());
Ok(id_val)
}
}
#[async_trait]
impl RagProvider for QdrantProvider {
async fn vector_search(
&self,
embedding: &[f32],
top_k: usize,
min_score: f32,
) -> Result<Vec<(DocumentId, f32)>> {
let url = format!(
"{}/collections/{}/points/search",
self.base_url, self.collection
);
// `score_threshold` is deliberately NOT sent. It is metric-aware: on Cosine
// collections 0.0 means "no floor" as expected, but Euclid collections score
// by negative distance, where 0.0 filters everything out. The attach wizard
// does not pin the distance metric, so filter locally instead; i.e. where a
// 0.0 floor is correctly treated as "no floor" (see `parse_search_hits`).
let body = serde_json::json!({
"vector": embedding,
"limit": top_k,
"with_payload": false,
});
let resp = self.client.post(&url).json(&body).send().await?;
if !resp.status().is_success() {
bail!(
"Qdrant search on '{}' failed: {}",
self.collection,
Self::error_message(resp).await
);
}
let data: Value = resp.json().await?;
// The interner is what lets a UUID-keyed collection work: a string id gets
// a synthetic handle here and the original is replayed by `fetch_content`.
let mut interner = self.point_ids.write();
parse_search_hits(&mut interner, &data, min_score)
}
async fn fetch_content(&self, ids: &[DocumentId]) -> Result<Vec<(DocumentId, String)>> {
if ids.is_empty() {
return Ok(vec![]);
}
let url = format!("{}/collections/{}/points", self.base_url, self.collection);
// Qdrant is asked for the ids it issued, never for a synthetic handle.
let id_list = self.point_ids.read().outbound_ids(ids);
let body = serde_json::json!({
"ids": id_list,
"with_payload": true,
});
let resp = self.client.post(&url).json(&body).send().await?;
if !resp.status().is_success() {
bail!(
"Qdrant point fetch on '{}' failed: {}",
self.collection,
Self::error_message(resp).await
);
}
let data: Value = resp.json().await?;
let mut rows = {
let mut interner = self.point_ids.write();
parse_points(&mut interner, &data)?
};
// `/points` does not guarantee response order matches request order, and the
// caller's RRF ranking is carried by that order. Restore it.
let position: HashMap<DocumentId, usize> =
ids.iter().enumerate().map(|(i, id)| (*id, i)).collect();
rows.sort_by_key(|(id, _)| position.get(id).copied().unwrap_or(usize::MAX));
Ok(rows)
}
async fn rebuild_indexes(&mut self, data: &RagData, _full_rebuild: bool) -> Result<()> {
// Both arms refuse. A silent `Ok(())` would make `.rebuild rag` and
// `.edit rag-docs` look like they worked while writing nothing to the
// remote, leaving the user believing the collection was updated.
if data.attached {
bail!(
"This RAG is attached to an external Qdrant collection. Coyote does not own \
its documents and cannot rebuild it. Manage the collection directly, or \
create a Coyote-owned RAG with `.rag <name>`."
);
}
bail!("Writing to Qdrant is not supported yet (attach-only).");
}
fn duplicate(&self, _data: &RagData) -> Box<dyn RagProvider> {
// Cloning the client shares the connection pool and the injected api-key
// header. Sharing is correct: both handles address the same remote
// collection, and neither of them writes to it.
//
// The point-id map is shared for the same reason, and because it MUST be:
// `Rag::clone()` hands the clone `DocumentId`s that the original minted,
// so a fresh map would resolve them to nothing and `fetch_content` would
// ask Qdrant for a synthetic handle — zero results, no error. Resetting it
// would also re-mint handles for ids the original still holds.
Box::new(Self {
client: self.client.clone(),
base_url: self.base_url.clone(),
collection: self.collection.clone(),
point_ids: Arc::clone(&self.point_ids),
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn error_message_reads_the_object_status_envelope() {
let body =
r#"{"status": {"error": "Wrong input: Not existing vector name error:"}, "time": 0.0}"#;
let msg = format_error_body(StatusCode::BAD_REQUEST, body);
assert!(msg.contains("Not existing vector name"), "got: {msg}");
assert!(
!msg.contains("EOF"),
"must not fall through to a parse error"
);
}
#[test]
fn error_message_survives_the_string_status_and_the_empty_body() {
let ok = format_error_body(StatusCode::OK, r#"{"status": "ok", "time": 0.0}"#);
assert!(
ok.contains("200"),
"no `status.error` present → fall back to status+body: {ok}"
);
let empty = format_error_body(StatusCode::NOT_FOUND, "");
assert!(empty.contains("empty body"), "got: {empty}");
assert!(
empty.contains("verb"),
"the message must point at the likely cause: {empty}"
);
}
#[test]
fn vector_dimension_handles_both_collection_shapes() {
let unnamed = serde_json::json!({
"result": {"config": {"params": {"vectors": {"size": 1536, "distance": "Cosine"}}}}
});
assert_eq!(vector_dimension_from_collection(&unnamed).unwrap(), 1536);
let named = serde_json::json!({
"result": {"config": {"params": {"vectors": {"text": {"size": 768, "distance": "Cosine"}}}}}
});
assert_eq!(vector_dimension_from_collection(&named).unwrap(), 768);
let junk = serde_json::json!({"result": {"config": {"params": {}}}});
assert!(vector_dimension_from_collection(&junk).is_err());
}
#[test]
fn is_multi_vector_rejects_the_named_single_collection() {
let unnamed = serde_json::json!({
"result": {"config": {"params": {"vectors": {"size": 1536, "distance": "Cosine"}}}}
});
assert!(!is_multi_vector_config(&unnamed));
let named_single = serde_json::json!({
"result": {"config": {"params": {"vectors": {"text": {"size": 1536}}}}}
});
assert!(
is_multi_vector_config(&named_single),
"named-single must be rejected too"
);
let named_multi = serde_json::json!({
"result": {"config": {"params": {"vectors": {"text": {"size": 1536}, "image": {"size": 512}}}}}
});
assert!(is_multi_vector_config(&named_multi));
}
#[test]
fn normalize_base_url_only_adds_a_scheme_when_missing() {
assert_eq!(
QdrantProvider::normalize_base_url("qdrant.example.com:6333"),
"http://qdrant.example.com:6333"
);
assert_eq!(
QdrantProvider::normalize_base_url("https://xyz.cloud.qdrant.io"),
"https://xyz.cloud.qdrant.io"
);
assert_eq!(
QdrantProvider::normalize_base_url("http://localhost:6333"),
"http://localhost:6333"
);
}
#[tokio::test]
async fn rebuild_indexes_refuses_for_attached_and_unattached_alike() {
let mut provider = QdrantProvider {
client: Client::new(),
base_url: "http://localhost:6333".to_string(),
collection: "c".to_string(),
point_ids: Arc::default(),
};
let attached = RagData {
driver: "qdrant".to_string(),
attached: true,
..Default::default()
};
let err = provider
.rebuild_indexes(&attached, true)
.await
.expect_err("an attached qdrant RAG must never report a successful rebuild");
assert!(err.to_string().contains("cannot rebuild"), "got: {err}");
// `attached: false` is reserved for the (unimplemented) write path. It must
// also refuse: silently succeeding would run a full paid embedding pass and
// then discard every vector.
let owned = RagData {
driver: "qdrant".to_string(),
attached: false,
..Default::default()
};
let err = provider
.rebuild_indexes(&owned, true)
.await
.expect_err("writing to qdrant is unimplemented and must fail loudly");
assert!(err.to_string().contains("not supported yet"), "got: {err}");
}
#[tokio::test]
async fn fetch_content_short_circuits_on_an_empty_id_list() {
let provider = QdrantProvider {
client: Client::new(),
base_url: "http://127.0.0.1:1".to_string(),
collection: "c".to_string(),
point_ids: Arc::default(),
};
assert!(provider.fetch_content(&[]).await.unwrap().is_empty());
}
#[test]
fn local_and_private_hosts_skip_the_proxy() {
for host in [
"http://localhost:6333",
"http://127.0.0.1:6333",
"http://192.168.0.56:6333",
"http://10.1.2.3:6333",
"http://172.16.4.5:6333",
"http://qdrant.local:6333",
"http://[::1]:6333",
] {
assert!(
QdrantProvider::skips_proxy(host),
"{host} should not be proxied"
);
}
}
#[test]
fn public_hosts_still_honour_the_environment() {
for host in [
"https://qdrant.example.com",
"http://8.8.8.8:6333",
"https://xyz.eu-central.aws.cloud.qdrant.io:6333",
"http://172.32.0.1:6333",
] {
assert!(
!QdrantProvider::skips_proxy(host),
"{host} must keep the environment's proxy"
);
}
}
/// Euclid collections score by NEGATIVE distance, so the 0.0 the caller
/// passes must mean "no floor". Filtering on it drops every hit — the exact
/// bug that keeps Qdrant's own `score_threshold` off the wire.
#[test]
fn a_zero_floor_keeps_negative_euclid_scores() {
let mut interner = PointIdInterner::default();
let search = serde_json::json!({
"result": [
{"id": 1, "score": -0.12},
{"id": 2, "score": -8.5},
]
});
let hits = parse_search_hits(&mut interner, &search, 0.0).unwrap();
assert_eq!(hits.len(), 2, "a 0.0 floor must not drop negative scores");
}
#[test]
fn a_positive_floor_still_filters() {
let mut interner = PointIdInterner::default();
let search = serde_json::json!({
"result": [
{"id": 1, "score": 0.9},
{"id": 2, "score": 0.2},
]
});
let hits = parse_search_hits(&mut interner, &search, 0.5).unwrap();
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].0, DocumentId(1));
}
/// A UUID-keyed collection has to survive the whole `vector_search` →
/// `fetch_content` round trip, and the fetch must ask Qdrant for the ORIGINAL
/// string id. Parsing ids with `as_u64()` used to drop these hits inside a
/// `filter_map`, i.e. zero results and no error.
#[test]
fn uuid_point_ids_round_trip_and_are_requested_verbatim() {
let mut interner = PointIdInterner::default();
let first_uuid = "3f1b0c2e-1111-4000-8000-000000000001";
let second_uuid = "3f1b0c2e-2222-4000-8000-000000000002";
let search = serde_json::json!({
"result": [
{"id": first_uuid, "score": 0.91},
{"id": second_uuid, "score": 0.42},
]
});
let hits = parse_search_hits(&mut interner, &search, 0.0).unwrap();
assert_eq!(hits.len(), 2, "string ids must not be silently dropped");
let ids: Vec<DocumentId> = hits.iter().map(|(id, _)| *id).collect();
assert_eq!(
interner.outbound_ids(&ids),
vec![Value::from(first_uuid), Value::from(second_uuid)],
"the fetch must send the ids Qdrant issued, not the handles"
);
// Qdrant may answer /points in any order; the handles still map back and
// the caller's RRF ranking is recoverable.
let points = serde_json::json!({
"result": [
{"id": second_uuid, "payload": {"page_content": "second"}},
{"id": first_uuid, "payload": {"page_content": "first"}},
]
});
let mut rows = parse_points(&mut interner, &points).unwrap();
let position: HashMap<DocumentId, usize> =
ids.iter().enumerate().map(|(i, id)| (*id, i)).collect();
rows.sort_by_key(|(id, _)| position.get(id).copied().unwrap_or(usize::MAX));
assert_eq!(
rows,
vec![
(ids[0], "first".to_string()),
(ids[1], "second".to_string())
]
);
}
/// Integer-keyed collections must be untouched by the interner: the id maps to
/// itself on the way in and goes back out as the same integer.
#[test]
fn integer_point_ids_are_passed_through_untouched() {
let mut interner = PointIdInterner::default();
let search = serde_json::json!({
"result": [{"id": 7, "score": 0.9}, {"id": 0, "score": 0.5}]
});
let hits = parse_search_hits(&mut interner, &search, 0.0).unwrap();
assert_eq!(
hits,
vec![(DocumentId(7), 0.9_f32), (DocumentId(0), 0.5_f32)]
);
let ids: Vec<DocumentId> = hits.iter().map(|(id, _)| *id).collect();
assert_eq!(
interner.outbound_ids(&ids),
vec![Value::from(7_u64), Value::from(0_u64)],
"integer ids must not be regressed into synthetic handles"
);
assert!(
interner.raw_id(DocumentId(7)).is_none(),
"a plain integer id is its own id and needs no map entry"
);
}
/// Synthetic handles are stable per point id and live in a range no packed
/// `DocumentId` can reach.
#[test]
fn synthetic_handles_are_stable_and_never_collide_with_packed_ids() {
let mut interner = PointIdInterner::default();
let uuid = Value::from("9d2f0a11-3333-4000-8000-00000000000a");
let handle = interner.document_id(&uuid).unwrap();
assert_eq!(
interner.document_id(&uuid).unwrap(),
handle,
"the same point id must keep the same handle across queries"
);
assert_ne!(
interner.document_id(&Value::from("other")).unwrap(),
handle,
"distinct point ids must not share a handle"
);
assert_ne!(handle.0 & SYNTHETIC_ID_TAG, 0, "a handle carries the tag");
// A packed (file_index, document_index) never sets the tag bit: it is the
// top bit of the file index, which would take 2^31 indexed files.
for (file_index, document_index) in [(0, 0), (1, 0), (0, 4242), (1_000_000, 999)] {
assert_eq!(
DocumentId::new(file_index, document_index).0 & SYNTHETIC_ID_TAG,
0,
"packed ({file_index}, {document_index}) must stay out of the handle range"
);
}
// The one integer id that WOULD land on the tag is interned instead of
// being handed back as itself, so it cannot alias a handle.
let collides = Value::from(SYNTHETIC_ID_TAG as u64);
let interned = interner.document_id(&collides).unwrap();
assert_eq!(interner.raw_id(interned), Some(&collides));
assert_eq!(
interner.outbound_ids(&[interned]),
vec![collides],
"the original integer must still be what Qdrant is asked for"
);
}
/// `duplicate()` shares the map rather than resetting it: `Rag::clone()` hands
/// the clone `DocumentId`s the original minted, and a fresh map would turn
/// those into requests for a synthetic handle — zero results, no error.
#[test]
fn duplicate_shares_the_point_id_map() {
let provider = QdrantProvider {
client: Client::new(),
base_url: "http://127.0.0.1:1".to_string(),
collection: "c".to_string(),
point_ids: Arc::default(),
};
let uuid = Value::from("c0ffee00-4444-4000-8000-000000000007");
let handle = provider.point_ids.write().document_id(&uuid).unwrap();
let dup = provider.duplicate(&RagData {
driver: "qdrant".to_string(),
attached: true,
..Default::default()
});
// Downcasting is not available through `dyn RagProvider`, so go via the
// shared Arc: the clone must observe the original's interning.
assert_eq!(Arc::strong_count(&provider.point_ids), 2);
assert_eq!(provider.point_ids.read().raw_id(handle), Some(&uuid));
drop(dup);
}
#[tokio::test]
#[ignore]
async fn qdrant_list_collections_requires_running_instance() {
let collections = QdrantProvider::list_collections("http://localhost:6333", None)
.await
.unwrap();
assert!(!collections.is_empty());
}
#[tokio::test]
#[ignore]
async fn qdrant_vector_search_returns_results() {
let provider = QdrantProvider::new("http://localhost:6333", "test-collection", None)
.await
.unwrap();
let embedding = vec![0.0f32; 1536];
let results = provider.vector_search(&embedding, 5, 0.0).await.unwrap();
assert!(results.len() <= 5);
}
}
+219
View File
@@ -0,0 +1,219 @@
use crate::rag::provider::RagProvider;
use crate::rag::{DocumentId, RagData};
use anyhow::Result;
use async_trait::async_trait;
use hnsw_rs::prelude::*;
use indexmap::IndexMap;
pub struct YamlProvider {
hnsw: Hnsw<'static, f32, DistCosine>,
content_map: IndexMap<DocumentId, String>,
}
impl YamlProvider {
pub fn from_data(data: &RagData) -> Self {
Self {
hnsw: data.build_hnsw(),
content_map: Self::build_content_map(data),
}
}
fn build_content_map(data: &RagData) -> IndexMap<DocumentId, String> {
data.iter_documents()
.map(|(id, doc)| (id, doc.page_content.clone()))
.collect()
}
}
#[async_trait]
impl RagProvider for YamlProvider {
async fn vector_search(
&self,
embedding: &[f32],
top_k: usize,
min_score: f32,
) -> Result<Vec<(DocumentId, f32)>> {
let results = self
.hnsw
.parallel_search(&[embedding.to_vec()], top_k, 30)
.into_iter()
.flat_map(|list| {
list.into_iter().filter_map(|v| {
let score = 1.0 - v.distance;
if score > min_score {
Some((DocumentId(v.d_id), score))
} else {
None
}
})
})
.collect();
Ok(results)
}
async fn fetch_content(&self, ids: &[DocumentId]) -> Result<Vec<(DocumentId, String)>> {
Ok(ids
.iter()
.filter_map(|id| self.content_map.get(id).map(|text| (*id, text.clone())))
.collect())
}
async fn rebuild_indexes(&mut self, data: &RagData, _full_rebuild: bool) -> Result<()> {
self.hnsw = data.build_hnsw();
self.content_map = Self::build_content_map(data);
Ok(())
}
fn duplicate(&self, data: &RagData) -> Box<dyn RagProvider> {
Box::new(YamlProvider::from_data(data))
}
}
#[cfg(test)]
mod provider_tests {
use super::*;
use crate::rag::{RagDocument, RagFile};
fn minimal_rag_data() -> RagData {
RagData {
embedding_model: "text-embedding-3-small".to_string(),
chunk_size: 1024,
chunk_overlap: 50,
top_k: 5,
driver: "yaml".to_string(),
attached: false,
..Default::default()
}
}
/// Two files, one chunk each, with vectors, the minimum needed to exercise
/// `build_content_map` and the `fetch_content` ordering contract.
/// `DocumentId::new(f, d)` packs (file_index, document_index); `RagData::add`
/// is the real insertion path but a direct literal is sufficient and avoids
/// the embedding pipeline.
fn populated_rag_data() -> RagData {
let mut data = minimal_rag_data();
// `files` must be populated: build_content_map iterates data.iter_documents(),
// which enumerates `files`. Populating `vectors` alone would produce an EMPTY
// content map, and every assertion below would vacuously pass on a broken impl.
// The vectors inserted at the end are for the HNSW side only.
data.files.insert(
0,
RagFile {
hash: "h0".to_string(),
path: "/tmp/a.md".to_string(),
documents: vec![RagDocument {
page_content: "alpha".to_string(),
metadata: Default::default(),
}],
},
);
data.files.insert(
1,
RagFile {
hash: "h1".to_string(),
path: "/tmp/b.md".to_string(),
documents: vec![RagDocument {
page_content: "beta".to_string(),
metadata: Default::default(),
}],
},
);
data.vectors
.insert(DocumentId::new(0, 0), vec![1.0, 0.0, 0.0]);
data.vectors
.insert(DocumentId::new(1, 0), vec![0.0, 1.0, 0.0]);
data
}
#[tokio::test]
async fn yaml_provider_empty_data_returns_nothing() {
let data = minimal_rag_data();
let provider = YamlProvider::from_data(&data);
let results = provider.fetch_content(&[]).await.unwrap();
assert!(results.is_empty());
}
#[tokio::test]
async fn yaml_provider_fetch_content_preserves_input_order() {
let data = populated_rag_data();
let provider = YamlProvider::from_data(&data);
let a = DocumentId::new(0, 0);
let b = DocumentId::new(1, 0);
let forward = provider.fetch_content(&[a, b]).await.unwrap();
assert_eq!(forward.len(), 2, "both documents must resolve");
assert_eq!(forward[0].1, "alpha");
assert_eq!(forward[1].1, "beta");
let reversed = provider.fetch_content(&[b, a]).await.unwrap();
assert_eq!(
reversed[0].1, "beta",
"fetch_content must honor input order"
);
assert_eq!(reversed[1].1, "alpha");
}
#[tokio::test]
async fn yaml_provider_fetch_content_skips_missing_ids() {
let data = populated_rag_data();
let provider = YamlProvider::from_data(&data);
let a = DocumentId::new(0, 0);
let missing = DocumentId::new(99, 0);
let b = DocumentId::new(1, 0);
let out = provider.fetch_content(&[a, missing, b]).await.unwrap();
assert_eq!(out.len(), 2, "missing id is skipped, not an error");
assert_eq!(out[0].1, "alpha");
assert_eq!(out[1].1, "beta");
}
#[tokio::test]
async fn yaml_provider_duplicate_returns_equivalent_content() {
let data = populated_rag_data();
let provider = YamlProvider::from_data(&data);
let dup = provider.duplicate(&data);
let ids = [DocumentId::new(0, 0), DocumentId::new(1, 0)];
let r1 = provider.fetch_content(&ids).await.unwrap();
let r2 = dup.fetch_content(&ids).await.unwrap();
assert_eq!(r1.len(), 2, "fixture must resolve both documents");
assert_eq!(
r1, r2,
"duplicate must resolve the same content as the original"
);
}
#[tokio::test]
async fn yaml_provider_content_is_keyed_on_files_not_vectors() {
let mut data = populated_rag_data();
let orphan = DocumentId::new(9, 0);
data.vectors.insert(orphan, vec![0.0, 0.0, 1.0]);
let provider = YamlProvider::from_data(&data);
let out = provider.fetch_content(&[orphan]).await.unwrap();
assert!(
out.is_empty(),
"an id present only in `vectors` must not resolve to content"
);
let real = provider
.fetch_content(&[DocumentId::new(0, 0), DocumentId::new(1, 0)])
.await
.unwrap();
assert_eq!(real.len(), 2, "file-backed documents must still resolve");
assert_eq!(real[0].1, "alpha");
assert_eq!(real[1].1, "beta");
}
}
+80
View File
@@ -1749,6 +1749,86 @@ std::error::Error>> {
);
}
fn strip_ansi(text: &str) -> String {
let mut out = String::with_capacity(text.len());
let mut chars = text.chars();
while let Some(c) = chars.next() {
if c != '\u{1b}' {
out.push(c);
continue;
}
if chars.next() == Some('[') {
for c in chars.by_ref() {
if ('\u{40}'..='\u{7e}').contains(&c) {
break;
}
}
}
}
out
}
#[test]
fn strip_ansi_removes_sgr_and_keeps_text() {
assert_eq!(strip_ansi("\x1b[1mbold\x1b[0m"), "bold");
assert_eq!(strip_ansi("\x1b[38;5;120mx\x1b[39m"), "x");
assert_eq!(strip_ansi("plain"), "plain");
}
#[test]
fn render_table_pads_columns_by_display_width() {
use unicode_width::UnicodeWidthStr;
const WRAP_WIDTH: u16 = 80;
let options = RenderOptions::default();
let mut render = MarkdownRender::init(options).unwrap();
render.wrap_width = Some(WRAP_WIDTH);
let header = vec![
"**Setting**".into(),
"*Default*".into(),
"`Description`".into(),
];
let alignments = vec![
CellAlignment::Left,
CellAlignment::Right,
CellAlignment::Center,
];
let rows = vec![
vec![
"**temperature**".into(),
"`0.7`".into(),
"Controls how *random* the sampled reply is allowed to be".into(),
],
vec![
"**top_p**".into(),
"`1.0`".into(),
"Nucleus sampling cutoff, applied **after** temperature".into(),
],
];
let output = render.render_table(header, alignments, rows);
assert!(
output.contains('\u{1b}'),
"fixture must actually contain ANSI escapes: {output:?}",
);
let widths: Vec<usize> = output
.lines()
.map(|line| strip_ansi(line).width())
.collect();
assert!(!widths.is_empty(), "table rendered no lines");
for (index, width) in widths.iter().enumerate() {
assert_eq!(
*width, WRAP_WIDTH as usize,
"line {index} display width; all widths were {widths:?} in output:\n{output}",
);
}
}
#[test]
fn state_machine_renders_full_table_and_flushes_on_paragraph() {
let options = RenderOptions::default();
+19 -6
View File
@@ -53,7 +53,7 @@ pub const DEFAULT_CONTINUATION_PROMPT: &str = indoc! {"
4. Continue with the next pending item now. Call tools immediately."
};
static REPL_COMMANDS: LazyLock<[ReplCommand; 59]> = LazyLock::new(|| {
static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| {
[
ReplCommand::new(".help", "Show this help guide", AssertState::pass()),
ReplCommand::new(".info", "Show system info", AssertState::pass()),
@@ -217,6 +217,11 @@ static REPL_COMMANDS: LazyLock<[ReplCommand; 59]> = LazyLock::new(|| {
"Initialize or access RAG",
AssertState::False(StateFlags::AGENT),
),
ReplCommand::new(
".rag attach",
"Attach to a pre-existing external RAG",
AssertState::False(StateFlags::AGENT),
),
ReplCommand::new(
".edit rag-docs",
"Add or remove documents from an existing RAG",
@@ -884,9 +889,17 @@ pub async fn run_repl_command(
let version = args.map(|s| s.trim().to_string());
task::spawn_blocking(move || config::run_self_update(version, false)).await??;
}
".rag" => {
ctx.use_rag(args, abort_signal.clone()).await?;
}
".rag" => match split_first_arg(args) {
Some(("attach", rest)) => match rest {
Some(name) if !name.trim().is_empty() => {
ctx.attach_rag(name.trim(), abort_signal.clone()).await?;
}
_ => println!("Usage: .rag attach <name>"),
},
_ => {
ctx.use_rag(args, abort_signal.clone()).await?;
}
},
".agent" => match split_first_arg(args) {
Some((agent_name, args)) => {
let (new_args, _) = split_args_text(args.unwrap_or_default(), cfg!(windows));
@@ -1711,8 +1724,8 @@ mod tests {
}
#[test]
fn repl_commands_has_59_entries() {
assert_eq!(REPL_COMMANDS.len(), 59);
fn repl_commands_has_60_entries() {
assert_eq!(REPL_COMMANDS.len(), 60);
}
#[test]
+54 -38
View File
@@ -387,7 +387,7 @@ pub(crate) fn collect_server_allow_entries(
out.into_iter().collect()
}
fn allow_entry_for_url(raw: &str) -> Option<String> {
pub(crate) fn allow_entry_for_url(raw: &str) -> Option<String> {
let url = Url::parse(raw).ok()?;
let scheme = url.scheme();
if scheme != "https" && scheme != "http" {
@@ -426,8 +426,8 @@ fn placeholders(text: &str) -> Result<Vec<PlaceholderMatch>> {
struct CredentialsMixin {
schema_version: &'static str,
kind: &'static str,
name: &'static str,
description: &'static str,
name: String,
description: String,
#[serde(skip_serializing_if = "Vec::is_empty")]
credentials: Vec<CredentialEntry>,
#[serde(skip_serializing_if = "Option::is_none")]
@@ -435,20 +435,20 @@ struct CredentialsMixin {
}
#[derive(Serialize)]
struct CredentialEntry {
service: String,
description: String,
pub(crate) struct CredentialEntry {
pub service: String,
pub description: String,
#[serde(rename = "apiKey")]
api_key: ApiKey,
pub api_key: ApiKey,
}
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct ApiKey {
name: String,
proxy_managed: bool,
pub(crate) struct ApiKey {
pub name: String,
pub proxy_managed: bool,
#[serde(skip_serializing_if = "Vec::is_empty")]
inject: Vec<InjectRule>,
pub inject: Vec<InjectRule>,
}
#[derive(Serialize)]
@@ -461,40 +461,24 @@ struct Network {
allow: Vec<String>,
}
pub(crate) fn render_mixin_yaml(
credentials: &[CredentialSpec],
server_allow_entries: &[String],
pub(crate) fn render_mixin_document(
name: &str,
description: &str,
credentials: Vec<CredentialEntry>,
extra_allow_entries: &[String],
) -> Result<String> {
let mut allow: BTreeSet<String> = credentials
.iter()
.flat_map(|c| c.inject.iter().map(|r| r.domain.clone()))
.flat_map(|c| c.api_key.inject.iter().map(|r| r.domain.clone()))
.collect();
allow.extend(server_allow_entries.iter().cloned());
allow.extend(extra_allow_entries.iter().cloned());
let mixin = CredentialsMixin {
schema_version: "2",
kind: "mixin",
name: MCP_MIXIN_NAME,
description: "Auto-generated by Coyote at launch: allows network egress to the user's \
remote MCP servers and declares their credentials so Docker Sandboxes \
binds them (bindings are approved on first interactive run). Values are \
pre-seeded from Coyote's vault via `sbx secret set`.",
credentials: credentials
.iter()
.map(|c| CredentialEntry {
service: c.service_id.clone(),
description: format!(
"Coyote vault secret '{}', used by MCP server(s) {}",
c.secret_name,
quoted_list(&c.servers)
),
api_key: ApiKey {
name: c.env_var.clone(),
proxy_managed: c.proxy_managed,
inject: c.inject.clone(),
},
})
.collect(),
name: name.to_string(),
description: description.to_string(),
credentials,
permissions: (!allow.is_empty()).then(|| Permissions {
network: Network {
allow: allow.into_iter().collect(),
@@ -502,7 +486,39 @@ pub(crate) fn render_mixin_yaml(
}),
};
serde_yaml::to_string(&mixin).context("Failed to serialize generated MCP credentials mixin")
serde_yaml::to_string(&mixin).context("Failed to serialize generated sandbox mixin")
}
pub(crate) fn render_mixin_yaml(
credentials: &[CredentialSpec],
server_allow_entries: &[String],
) -> Result<String> {
let entries = credentials
.iter()
.map(|c| CredentialEntry {
service: c.service_id.clone(),
description: format!(
"Coyote vault secret '{}', used by MCP server(s) {}",
c.secret_name,
quoted_list(&c.servers)
),
api_key: ApiKey {
name: c.env_var.clone(),
proxy_managed: c.proxy_managed,
inject: c.inject.clone(),
},
})
.collect();
render_mixin_document(
MCP_MIXIN_NAME,
"Auto-generated by Coyote at launch: allows network egress to the user's remote MCP \
servers and declares their credentials so Docker Sandboxes binds them (bindings are \
approved on first interactive run). Values are pre-seeded from Coyote's vault via \
`sbx secret set`.",
entries,
server_allow_entries,
)
}
#[cfg(test)]
+528 -21
View File
@@ -1,16 +1,19 @@
use std::env;
use std::fs;
use std::fs::{read_dir, read_to_string};
use std::io;
use std::path::{Path, PathBuf};
use anyhow::{Context, Result};
use anyhow::{Context, Result, anyhow, bail};
use serde_yaml::Value;
use sha2::{Digest, Sha256};
use crate::config::paths;
const SBX_MIXIN_FILE_NAME: &str = "sbx-mixin.yaml";
const SBX_MIXIN_FILE_SUFFIX: &str = ".sbx-mixin.yaml";
const KIT_SPEC_FILE_NAME: &str = "spec.yaml";
const MIXIN_FILES_DIR_NAME: &str = "files";
#[derive(Debug, Clone)]
pub struct DiscoveredMixin {
@@ -33,43 +36,168 @@ impl DiscoveredMixin {
pub fn wrap_mixin_as_kit(mixin_path: &Path) -> Result<PathBuf> {
let bytes = fs::read(mixin_path)
.with_context(|| format!("Failed to read sbx mixin {}", mixin_path.display()))?;
wrap_mixin_bytes_as_kit(&bytes, &mixin_path.display().to_string())
let label = mixin_path.display().to_string();
let files = mixin_path
.parent()
.map(|p| p.join(MIXIN_FILES_DIR_NAME))
.filter(|p| p.is_dir())
.map(|dir| collect_staged_files(&dir))
.transpose()?
.unwrap_or_default();
stage_kit(&bytes, &files, &label)
}
pub fn wrap_mixin_bytes_as_kit(bytes: &[u8], label: &str) -> Result<PathBuf> {
stage_kit(bytes, &[], label)
}
struct StagedFile {
relpath: PathBuf,
mode: u32,
bytes: Vec<u8>,
}
fn stage_kit(spec_bytes: &[u8], files: &[StagedFile], label: &str) -> Result<PathBuf> {
let mut hasher = Sha256::new();
hasher.update(bytes);
hasher.update(spec_bytes);
for f in files {
let rel_str = f.relpath.to_str().ok_or_else(|| {
anyhow!(
"Non-UTF-8 path inside mixin {MIXIN_FILES_DIR_NAME}/: {}",
f.relpath.display()
)
})?;
hasher.update(b"\0COYOTE_MIXIN_FILE\0");
hasher.update((rel_str.len() as u64).to_le_bytes());
hasher.update(rel_str.as_bytes());
hasher.update(f.mode.to_le_bytes());
hasher.update((f.bytes.len() as u64).to_le_bytes());
hasher.update(&f.bytes);
}
let hash = format!("{:x}", hasher.finalize());
let kit_dir = paths::sbx_mixin_kits_dir().join(&hash);
let spec_path = kit_dir.join(KIT_SPEC_FILE_NAME);
let files_dst = kit_dir.join(MIXIN_FILES_DIR_NAME);
if let Ok(existing) = fs::read(&spec_path)
&& existing == bytes
{
let spec_matches = fs::read(&spec_path).is_ok_and(|existing| existing == spec_bytes);
let files_ready = files.is_empty() || files_dst.is_dir();
if spec_matches && files_ready {
return Ok(kit_dir);
}
fs::create_dir_all(&kit_dir)
.with_context(|| format!("Failed to create mixin kit dir {}", kit_dir.display()))?;
fs::write(&spec_path, bytes)
fs::write(&spec_path, spec_bytes)
.with_context(|| format!("Failed to write {}", spec_path.display()))?;
if !files.is_empty() {
if files_dst.exists() {
fs::remove_dir_all(&files_dst).with_context(|| {
format!(
"Failed to clear stale mixin files at {}",
files_dst.display()
)
})?;
}
for f in files {
let dst = files_dst.join(&f.relpath);
if let Some(parent) = dst.parent() {
fs::create_dir_all(parent)
.with_context(|| format!("Failed to create dir {}", parent.display()))?;
}
fs::write(&dst, &f.bytes)
.with_context(|| format!("Failed to write staged mixin file {}", dst.display()))?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&dst, fs::Permissions::from_mode(f.mode))
.with_context(|| format!("Failed to set mode on {}", dst.display()))?;
}
}
}
debug!("Wrapped mixin {label} as kit at {}", kit_dir.display());
Ok(kit_dir)
}
fn collect_staged_files(root: &Path) -> Result<Vec<StagedFile>> {
let mut out = Vec::new();
walk_staged_files(root, Path::new(""), &mut out)?;
Ok(out)
}
fn walk_staged_files(abs_dir: &Path, rel_dir: &Path, out: &mut Vec<StagedFile>) -> Result<()> {
let rd = fs::read_dir(abs_dir)
.with_context(|| format!("Failed to read mixin files dir {}", abs_dir.display()))?;
let mut entries: Vec<_> = rd
.collect::<io::Result<Vec<_>>>()
.with_context(|| format!("Failed to iterate mixin files dir {}", abs_dir.display()))?;
entries.sort_by_key(|e| e.file_name());
for entry in entries {
let file_type = entry
.file_type()
.with_context(|| format!("Failed to stat {}", entry.path().display()))?;
let abs = entry.path();
let rel = rel_dir.join(entry.file_name());
if file_type.is_symlink() {
bail!(
"Symlinks are not allowed inside a mixin {MIXIN_FILES_DIR_NAME}/ tree: {}",
abs.display()
);
}
if file_type.is_dir() {
walk_staged_files(&abs, &rel, out)?;
} else if file_type.is_file() {
let bytes = fs::read(&abs)
.with_context(|| format!("Failed to read staged mixin file {}", abs.display()))?;
let mode = staged_file_mode(&entry)?;
out.push(StagedFile {
relpath: rel,
mode,
bytes,
});
}
}
Ok(())
}
#[cfg(unix)]
fn staged_file_mode(entry: &fs::DirEntry) -> Result<u32> {
use std::os::unix::fs::PermissionsExt;
let meta = entry
.metadata()
.with_context(|| format!("Failed to stat {}", entry.path().display()))?;
Ok(meta.permissions().mode() & 0o777)
}
#[cfg(not(unix))]
fn staged_file_mode(_entry: &fs::DirEntry) -> Result<u32> {
Ok(0o644)
}
pub fn discover() -> Result<Vec<DiscoveredMixin>> {
let mut out = Vec::new();
push_if_exists(&mut out, paths::sbx_mixin_file())?;
push_if_exists(&mut out, paths::global_tools_sbx_mixin_file())?;
for path in collect_subdir_mixins(&paths::functions_dir()) {
for path in collect_mixins(&paths::functions_dir(), &[ScanMode::SubdirNamed]) {
out.push(read_mixin(path)?);
}
for path in collect_subdir_mixins(&paths::agents_data_dir()) {
for path in collect_mixins(
&paths::agents_data_dir(),
&[ScanMode::SubdirNamed, ScanMode::SubdirFlat],
) {
out.push(read_mixin(path)?);
}
for path in collect_mixins(&paths::rags_dir(), &[ScanMode::Flat]) {
out.push(read_mixin(path)?);
}
@@ -156,7 +284,73 @@ fn read_mixin(path: PathBuf) -> Result<DiscoveredMixin> {
})
}
fn collect_subdir_mixins(dir: &Path) -> Vec<PathBuf> {
/// One on-disk layout a mixin scan can look for. A scan takes a set of these,
/// and each mode contributes only the shape it names.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ScanMode {
/// `<dir>/*.sbx-mixin.yaml`
Flat,
/// `<dir>/*/sbx-mixin.yaml`
SubdirNamed,
/// `<dir>/*/*.sbx-mixin.yaml`
SubdirFlat,
}
/// Collects mixin paths under `dir` for every requested layout. Missing or
/// unreadable directories yield nothing rather than an error — these paths are
/// all optional on disk.
///
/// Order is deterministic: flat matches first (sorted by file name), then each
/// subdirectory in sorted order, contributing its named mixin before its
/// suffixed ones.
fn collect_mixins(dir: &Path, modes: &[ScanMode]) -> Vec<PathBuf> {
let mut result = Vec::new();
if modes.contains(&ScanMode::Flat) {
result.extend(suffixed_mixins_in(dir));
}
let named = modes.contains(&ScanMode::SubdirNamed);
let subdir_flat = modes.contains(&ScanMode::SubdirFlat);
if !named && !subdir_flat {
return result;
}
for subdir in subdirs_of(dir) {
if named {
let candidate = subdir.join(SBX_MIXIN_FILE_NAME);
if candidate.exists() {
result.push(candidate);
}
}
if subdir_flat {
result.extend(suffixed_mixins_in(&subdir));
}
}
result
}
fn suffixed_mixins_in(dir: &Path) -> Vec<PathBuf> {
let mut result = Vec::new();
let Ok(rd) = read_dir(dir) else { return result };
let mut entries: Vec<_> = rd
.flatten()
.filter(|e| e.file_type().map(|t| t.is_file()).unwrap_or(false))
.filter(|e| {
e.file_name()
.to_str()
.is_some_and(|n| n.ends_with(SBX_MIXIN_FILE_SUFFIX))
})
.collect();
entries.sort_by_key(|e| e.file_name());
result.extend(entries.into_iter().map(|e| e.path()));
result
}
fn subdirs_of(dir: &Path) -> Vec<PathBuf> {
let mut result = Vec::new();
let Ok(rd) = read_dir(dir) else { return result };
@@ -166,13 +360,7 @@ fn collect_subdir_mixins(dir: &Path) -> Vec<PathBuf> {
.collect();
entries.sort_by_key(|e| e.file_name());
for entry in entries {
let candidate = entry.path().join(SBX_MIXIN_FILE_NAME);
if candidate.exists() {
result.push(candidate);
}
}
result.extend(entries.into_iter().map(|e| e.path()));
result
}
@@ -192,6 +380,13 @@ mod tests {
root
}
fn file_names(paths: &[PathBuf]) -> Vec<&str> {
paths
.iter()
.map(|p| p.file_name().unwrap().to_str().unwrap())
.collect()
}
#[test]
fn summarize_counts_installs_and_domains() {
let root = unique_root("sbx-mixin-counts");
@@ -275,7 +470,7 @@ network:
}
#[test]
fn collect_subdir_mixins_sorts_and_skips_missing() {
fn subdir_named_scan_sorts_and_skips_missing() {
let root = unique_root("sbx-mixin-subdirs");
for name in ["zebra", "apple", "no-mixin", "mango"] {
let dir = root.join(name);
@@ -285,7 +480,7 @@ network:
}
}
let found = collect_subdir_mixins(&root);
let found = collect_mixins(&root, &[ScanMode::SubdirNamed]);
let names: Vec<String> = found
.iter()
.map(|p| {
@@ -303,9 +498,9 @@ network:
}
#[test]
fn collect_subdir_mixins_returns_empty_for_missing_dir() {
fn subdir_named_scan_returns_empty_for_missing_dir() {
let absent = env::temp_dir().join("coyote-definitely-not-here-xyz");
let found = collect_subdir_mixins(&absent);
let found = collect_mixins(&absent, &[ScanMode::SubdirNamed]);
assert!(found.is_empty());
}
@@ -482,5 +677,317 @@ network:
"kit_path should not return the original file path"
);
}
fn write_staged_file(mixin: &Path, rel: &str, content: &[u8]) {
let dst = mixin.parent().unwrap().join(MIXIN_FILES_DIR_NAME).join(rel);
fs::create_dir_all(dst.parent().unwrap()).unwrap();
fs::write(&dst, content).unwrap();
}
#[test]
#[serial]
fn wrap_mixin_as_kit_copies_sibling_files_tree_into_kit() {
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-copy", "kind: mixin\nname: probe\n");
write_staged_file(&mixin, "home/hello.md", b"# hello\n");
write_staged_file(&mixin, "home/nested/deep.txt", b"deep\n");
let kit_dir = wrap_mixin_as_kit(&mixin).unwrap();
assert!(kit_dir.join("spec.yaml").exists());
let files_root = kit_dir.join(MIXIN_FILES_DIR_NAME);
assert!(files_root.is_dir(), "kit dir must contain a files/ tree");
assert_eq!(
fs::read(files_root.join("home/hello.md")).unwrap(),
b"# hello\n"
);
assert_eq!(
fs::read(files_root.join("home/nested/deep.txt")).unwrap(),
b"deep\n"
);
}
#[test]
#[serial]
fn wrap_mixin_as_kit_hash_changes_when_a_staged_file_is_edited() {
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-hash-content", "kind: mixin\nname: probe\n");
write_staged_file(&mixin, "home/note.md", b"before\n");
let kit_before = wrap_mixin_as_kit(&mixin).unwrap();
write_staged_file(&mixin, "home/note.md", b"after\n");
let kit_after = wrap_mixin_as_kit(&mixin).unwrap();
assert_ne!(
kit_before, kit_after,
"editing a staged file must invalidate the kit hash"
);
assert_eq!(
fs::read(kit_after.join("files/home/note.md")).unwrap(),
b"after\n"
);
}
#[test]
#[serial]
fn wrap_mixin_as_kit_hash_changes_when_a_staged_file_is_added() {
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-hash-added", "kind: mixin\nname: probe\n");
write_staged_file(&mixin, "home/one.md", b"one\n");
let kit_before = wrap_mixin_as_kit(&mixin).unwrap();
write_staged_file(&mixin, "home/two.md", b"two\n");
let kit_after = wrap_mixin_as_kit(&mixin).unwrap();
assert_ne!(
kit_before, kit_after,
"adding a staged file must invalidate the kit hash"
);
}
#[test]
#[serial]
fn wrap_mixin_as_kit_hash_unchanged_when_no_files_dir() {
let _guard = TestCacheDirGuard::new();
let content = "kind: mixin\nname: legacy\n";
let mixin = write_mixin("legacy-no-files", content);
let with_helper = wrap_mixin_as_kit(&mixin).unwrap();
let bytes_only = wrap_mixin_bytes_as_kit(content.as_bytes(), "legacy").unwrap();
assert_eq!(
with_helper, bytes_only,
"mixins without a sibling files/ must keep the legacy bytes-only hash to reuse existing cache dirs"
);
}
#[test]
#[serial]
fn wrap_mixin_as_kit_ignores_sibling_files_that_is_not_a_directory() {
let _guard = TestCacheDirGuard::new();
let content = "kind: mixin\nname: probe\n";
let mixin = write_mixin("files-not-a-dir", content);
fs::write(mixin.parent().unwrap().join(MIXIN_FILES_DIR_NAME), b"decoy").unwrap();
let wrapped = wrap_mixin_as_kit(&mixin).unwrap();
let bytes_only = wrap_mixin_bytes_as_kit(content.as_bytes(), "probe").unwrap();
assert_eq!(
wrapped, bytes_only,
"a regular file named files must be ignored, not staged"
);
assert!(!wrapped.join(MIXIN_FILES_DIR_NAME).exists());
}
#[test]
#[serial]
fn wrap_mixin_as_kit_rebuilds_files_when_cache_dir_missing_files_tree() {
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-rebuild", "kind: mixin\nname: probe\n");
write_staged_file(&mixin, "home/hello.md", b"hi\n");
let kit_dir = wrap_mixin_as_kit(&mixin).unwrap();
let files_dst = kit_dir.join(MIXIN_FILES_DIR_NAME);
fs::remove_dir_all(&files_dst).unwrap();
assert!(!files_dst.exists());
let kit_again = wrap_mixin_as_kit(&mixin).unwrap();
assert_eq!(kit_again, kit_dir, "kit path is content-addressed");
assert!(
files_dst.is_dir(),
"a partial cache (spec present, files/ missing) must be rebuilt"
);
assert_eq!(fs::read(files_dst.join("home/hello.md")).unwrap(), b"hi\n");
}
#[test]
#[serial]
fn wrap_mixin_as_kit_deterministic_with_staged_files() {
let _guard = TestCacheDirGuard::new();
let content = "kind: mixin\nname: probe\n";
let mixin_one = write_mixin("determ-1", content);
write_staged_file(&mixin_one, "home/note.md", b"same\n");
let mixin_two = write_mixin("determ-2", content);
write_staged_file(&mixin_two, "home/note.md", b"same\n");
let kit_a = wrap_mixin_as_kit(&mixin_one).unwrap();
let kit_b = wrap_mixin_as_kit(&mixin_two).unwrap();
assert_eq!(
kit_a, kit_b,
"identical spec+files must produce the same content-addressed kit dir"
);
}
#[cfg(unix)]
#[test]
#[serial]
fn wrap_mixin_as_kit_rejects_symlinks_inside_files_tree() {
use std::os::unix::fs::symlink;
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-symlink", "kind: mixin\nname: probe\n");
let files_dir = mixin.parent().unwrap().join(MIXIN_FILES_DIR_NAME);
fs::create_dir_all(&files_dir).unwrap();
let target = files_dir.join("target.txt");
fs::write(&target, b"real").unwrap();
symlink(&target, files_dir.join("link.txt")).unwrap();
let err = wrap_mixin_as_kit(&mixin).unwrap_err();
let msg = format!("{err:#}");
assert!(
msg.contains("Symlinks are not allowed"),
"expected symlink rejection, got: {msg}"
);
}
#[cfg(unix)]
#[test]
#[serial]
fn wrap_mixin_as_kit_preserves_executable_bit() {
use std::os::unix::fs::PermissionsExt;
let _guard = TestCacheDirGuard::new();
let mixin = write_mixin("files-exec", "kind: mixin\nname: probe\n");
write_staged_file(&mixin, "bin/run.sh", b"#!/bin/sh\necho hi\n");
let src = mixin
.parent()
.unwrap()
.join(MIXIN_FILES_DIR_NAME)
.join("bin/run.sh");
fs::set_permissions(&src, fs::Permissions::from_mode(0o755)).unwrap();
let kit_dir = wrap_mixin_as_kit(&mixin).unwrap();
let dst = kit_dir.join("files/bin/run.sh");
let mode = fs::metadata(&dst).unwrap().permissions().mode() & 0o777;
assert_eq!(
mode, 0o755,
"executable bit must survive the copy into the kit dir"
);
}
}
#[test]
fn flat_scan_matches_rag_sidecars_by_suffix() {
let root = unique_root("flat-mixins");
fs::write(root.join("company-docs.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(root.join("alpha.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(root.join("company-docs.yaml"), "driver: qdrant\n").unwrap();
fs::write(root.join("notes.yaml"), "driver: yaml\n").unwrap();
fs::create_dir_all(root.join("decoy.sbx-mixin.yaml")).unwrap();
let found = collect_mixins(&root, &[ScanMode::Flat]);
assert_eq!(
file_names(&found),
vec!["alpha.sbx-mixin.yaml", "company-docs.sbx-mixin.yaml"]
);
let _ = fs::remove_dir_all(&root);
}
/// Every scan site in `discover()` picks its modes assuming each mode owns
/// exactly one layout and nothing else. `agents_data_dir()` requests two
/// modes at once, so an overlap would collect the same file twice and
/// `create_sandbox` would pass it as two `--kit` flags.
#[test]
fn each_scan_mode_owns_exactly_one_layout() {
let root = unique_root("scan-mode-ownership");
let agent = root.join("researcher");
fs::create_dir_all(&agent).unwrap();
let flat = root.join("company-docs.sbx-mixin.yaml");
let subdir_named = agent.join("sbx-mixin.yaml");
let subdir_flat = agent.join("handbook.sbx-mixin.yaml");
for path in [&flat, &subdir_named, &subdir_flat] {
fs::write(path, "kind: mixin\n").unwrap();
}
assert_eq!(collect_mixins(&root, &[ScanMode::Flat]), vec![flat.clone()]);
assert_eq!(
collect_mixins(&root, &[ScanMode::SubdirNamed]),
vec![subdir_named.clone()]
);
assert_eq!(
collect_mixins(&root, &[ScanMode::SubdirFlat]),
vec![subdir_flat.clone()]
);
let all = collect_mixins(
&root,
&[ScanMode::Flat, ScanMode::SubdirNamed, ScanMode::SubdirFlat],
);
assert_eq!(all, vec![flat, subdir_named, subdir_flat]);
let mut deduped = all.clone();
deduped.sort();
deduped.dedup();
assert_eq!(
deduped.len(),
all.len(),
"no mixin may be collected twice: {all:?}"
);
let _ = fs::remove_dir_all(&root);
}
#[test]
fn flat_scan_tolerates_a_missing_directory() {
let root = unique_root("flat-missing");
let absent = root.join("nope");
assert!(collect_mixins(&absent, &[ScanMode::Flat]).is_empty());
let _ = fs::remove_dir_all(&root);
}
/// `generate_rag_sbx_mixin` writes an agent-scoped RAG sidecar next to the
/// rag yaml, at `<agents>/<agent>/<rag>.sbx-mixin.yaml`. Before `SubdirFlat`
/// existed, nothing scanned that shape and attaching a Qdrant RAG from
/// inside an agent produced no network allow rule and no credential.
#[test]
fn agent_scoped_rag_sidecar_is_discovered() {
let root = unique_root("agent-scoped-rag");
let agent = root.join("researcher");
fs::create_dir_all(&agent).unwrap();
fs::write(agent.join("company-docs.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(agent.join("company-docs.yaml"), "driver: qdrant\n").unwrap();
let found = collect_mixins(&root, &[ScanMode::SubdirNamed, ScanMode::SubdirFlat]);
assert_eq!(found, vec![agent.join("company-docs.sbx-mixin.yaml")]);
let _ = fs::remove_dir_all(&root);
}
#[test]
fn agent_level_mixin_and_rag_sidecars_are_both_discovered() {
let root = unique_root("agent-both-shapes");
let agent = root.join("researcher");
fs::create_dir_all(&agent).unwrap();
fs::write(agent.join("sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(agent.join("zebra.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(agent.join("alpha.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
let found = collect_mixins(&root, &[ScanMode::SubdirNamed, ScanMode::SubdirFlat]);
assert_eq!(
file_names(&found),
vec![
"sbx-mixin.yaml",
"alpha.sbx-mixin.yaml",
"zebra.sbx-mixin.yaml"
]
);
let _ = fs::remove_dir_all(&root);
}
#[test]
fn subdir_flat_scan_ignores_a_directory_named_like_a_mixin() {
let root = unique_root("subdir-flat-decoy");
let agent = root.join("researcher");
fs::create_dir_all(agent.join("decoy.sbx-mixin.yaml")).unwrap();
assert!(collect_mixins(&root, &[ScanMode::SubdirFlat]).is_empty());
let _ = fs::remove_dir_all(&root);
}
}
+145 -1
View File
@@ -10,7 +10,7 @@ use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use which::which;
mod mcp_credentials;
pub(crate) mod mcp_credentials;
mod mixins;
pub(crate) use mcp_credentials::sandbox_secret_env_var;
@@ -19,6 +19,7 @@ use crate::config::AppConfig;
use crate::config::Config;
use crate::config::VAULT_DATA_FILE_NAME;
use crate::config::paths;
use crate::rag::RagData;
use crate::sandbox::mcp_credentials::MCP_MIXIN_NAME;
use crate::sandbox::mixins::DiscoveredMixin;
use crate::utils::run_command_with_output;
@@ -53,6 +54,10 @@ pub fn launch(name: Option<String>, fresh: bool) -> Result<()> {
let vault = Vault::init(&bootstrap)?;
let registered = sbx_registered_services()?;
inject_llm_secret(&config_content, &vault, &registered)?;
if !fresh {
inject_rag_secrets(&vault, &registered)?;
}
let credentials_mixin = if fresh {
None
} else {
@@ -309,6 +314,87 @@ fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Opt
)?))
}
fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()> {
let rags_dir = paths::rags_dir();
if !rags_dir.exists() {
return Ok(());
}
for entry in fs::read_dir(&rags_dir)?.flatten() {
let path = entry.path();
if path.extension().and_then(|e| e.to_str()) != Some("yaml") {
continue;
}
let stem = match path.file_stem().and_then(|s| s.to_str()) {
Some(s) if !paths::is_rag_sidecar_name(s) => s.to_string(),
_ => continue,
};
let Ok(raw) = fs::read_to_string(&path) else {
continue;
};
let Ok(data) = serde_yaml::from_str::<RagData>(&raw) else {
continue;
};
if !data.attached {
continue;
}
let secret_names = driver_config_secret_names(&data);
let Some((primary, extra)) = secret_names.split_first() else {
continue;
};
let service_id = mcp_credentials::secret_service_id(&stem);
if !service_id.is_empty() && !registered.contains(&service_id) {
bind_rag_secret(vault, &service_id, primary, &stem)?;
}
for name in extra {
let id = mcp_credentials::secret_service_id(name);
if !id.is_empty() && !registered.contains(&id) {
bind_rag_secret(vault, &id, name, &stem)?;
}
}
}
Ok(())
}
fn driver_config_secret_names(data: &RagData) -> Vec<String> {
let mut names: Vec<String> = Vec::new();
for value in data.driver_config.values() {
let trimmed = value.trim();
let Ok(Some(caps)) = SECRET_RE.captures(trimmed) else {
continue;
};
if caps.get(0).map(|m| m.as_str()) != Some(trimmed) {
continue;
}
let Some(name) = caps.get(1).map(|m| m.as_str().trim()) else {
continue;
};
if !name.is_empty() && !names.iter().any(|n| n == name) {
names.push(name.to_string());
}
}
names
}
fn bind_rag_secret(vault: &Vault, service_id: &str, secret_name: &str, stem: &str) -> Result<()> {
match vault.get_secret(secret_name, false) {
Ok(secret_value) => {
sbx_secret_set(service_id, &secret_value)
.context("Failed to register RAG secret with sbx")?;
}
Err(e) => {
eprintln!(
"Warning: could not load secret '{secret_name}' for RAG '{stem}': {e}. \
Queries to this RAG will fail inside the sandbox. \
Run `coyote --add-secret {secret_name}` to fix."
);
}
}
Ok(())
}
fn provider_to_sbx_service(provider_type: &str, client_name: Option<&str>) -> String {
match provider_type {
"claude" => "anthropic".to_string(),
@@ -588,6 +674,64 @@ fn chown_agent_recursive(sandbox: &str, path: &str) -> Result<()> {
#[cfg(test)]
mod tests {
use super::*;
fn rag_with(driver_config: &[(&str, &str)]) -> RagData {
let mut data = RagData::new("m".into(), 1024, 50, None, 5, None, Default::default());
data.driver = "qdrant".to_string();
data.attached = true;
for (k, v) in driver_config {
data.driver_config.insert(k.to_string(), v.to_string());
}
data
}
#[test]
fn secret_names_are_found_whatever_the_field_is_called() {
let data = rag_with(&[
("host", "qdrant.example.com:6333"),
("collection", "docs"),
("token", "{{SOME_TOKEN}}"),
]);
assert_eq!(driver_config_secret_names(&data), vec!["SOME_TOKEN"]);
}
#[test]
fn a_literal_credential_is_not_treated_as_a_secret_name() {
let data = rag_with(&[("api_key", "sk-a-real-looking-key")]);
assert!(driver_config_secret_names(&data).is_empty());
}
#[test]
fn plain_values_are_never_mistaken_for_secrets() {
let data = rag_with(&[("host", "localhost:6333"), ("collection", "docs")]);
assert!(driver_config_secret_names(&data).is_empty());
}
#[test]
fn a_partial_placeholder_is_not_a_credential() {
let data = rag_with(&[("api_key", "Bearer {{KEY}}")]);
assert!(driver_config_secret_names(&data).is_empty());
}
#[test]
fn several_secrets_are_all_found_and_deduped() {
let data = rag_with(&[
("api_key", "{{QDRANT_KEY}}"),
("host", "localhost:6333"),
("token", "{{ OTHER_TOKEN }}"),
("fallback_key", "{{QDRANT_KEY}}"),
]);
assert_eq!(
driver_config_secret_names(&data),
vec!["QDRANT_KEY", "OTHER_TOKEN"],
"order follows driver_config, and a repeat is not registered twice"
);
}
use std::time::{SystemTime, UNIX_EPOCH};
#[test]