Agent startup and graph rag nodes both run an interactive wizard when their
knowledge base has not been built, but neither offered the driver choice that
interactive named-RAG creation has, so both silently produced a yaml store.
A plain agent was the worse of the two: AgentConfig carries only documents, so
there was no way to get a duckdb RAG for one, interactively or declaratively. A
graph node could at least declare driver: in the workflow.
Agent startup now passes prompt_for_driver, and a rag node whose wizard runs is
asked too. The prompt is skipped when the node already declares a driver, and
sits inside the not-fully-specified branch after the non-interactive bail, so
declarative workflows and headless runs are unchanged. Temp RAGs still pass
false: they are deleted on the next run, so a persistent store would only leave
a sidecar behind.
The prompt moves to select_rag_driver rather than being duplicated.
Deleting a RAG removed its .duckdb file but left the sibling .duckdb.wal
behind. DuckDB only removes that log on a clean close, so any RAG whose
process was killed left one on disk, and creating a new RAG under the same
name let it inherit a write-ahead log describing someone else's data.
The test helper already cleaned the log up after itself, which is why no
test noticed the production path did not.
`RagNode` gains an optional `driver`, forwarded into `RagInitConfig` so a
graph node can build its knowledge base on duckdb instead of yaml. Nodes
that name no driver forward `None`, which still resolves to yaml, so
existing workflows are unaffected.
An unknown driver is rejected up front rather than at construction time.
`Rag::create` dispatches unknown drivers to its yaml catch-all, so a typo
would otherwise embed every document and persist the bogus string, after
which every subsequent load fails validation and the agent cannot start.
The check asks `RagData::validate()` through a probe value instead of
restating the list of valid drivers, so the two cannot drift.
The RAG attach sidecar was written against the sbx kit v1 spec and still emitted schemaVersion "1" with network.allowedDomains, network.serviceDomains, network.serviceAuth, credentials.sources.<n>.env and environment.proxyManaged. Every one of those keys was removed in kit v2. Coyote does not validate mixins, it copies them byte-for-byte into spec.yaml, so the invalid document surfaced only as an opaque sbx failure with no indication of which mixin caused it.
generate_rag_sbx_mixin now builds the document from the shared serializer structs instead of a format! string, which is how the envelope drifted unnoticed in the first place. render_mixin_yaml and the RAG sidecar both go through a new render_mixin_document, giving one definition of the envelope and one enforcement point for the rule that every inject domain must also appear in permissions.network.allow.
Fix an auth bug the port exposed: inject_rag_secrets bound the API key with sbx secret set, but nothing ever emitted a matching credentials entry, so the proxy held a value with no inject rule and never rewrote the auth header. An attached RAG credential silently did not work inside the sandbox. The sidecar now declares that credential; a RAG with no API key declares none while still receiving egress.
Fix the service id: the bind passed the raw file stem instead of routing it through secret_service_id, so a RAG named My_Docs produced an illegal id. The bind and the generated credentials service now share that derivation and cannot disagree.
Retire sbx_domain_forms in favour of allow_entry_for_url, now pub(crate). It emitted both a bare host and host:port because v1 serviceDomains needed a bare key; v2 has no such need, so the extra entry is simply wrong. It also defaulted a schemeless host to port 6333 while normalize_base_url resolves it to http and port 80, meaning the allow entry named a port the client never dialled.
Adds QdrantProvider as a read-only driver for pre-existing remote Qdrant
collections, an interactive '.rag attach' wizard, and the sandbox credential
and domain-whitelisting wiring that lets an attached RAG work inside a sandbox.
Attach-only by design: rebuild_indexes bails for both the attached and the
unattached case rather than silently succeeding. Coyote never writes to Qdrant
in this change.
Vectors are never hydrated back from Qdrant. Cosine collections L2-normalize
stored vectors on write, so reading them back returns unit-length copies of the
originals; the YAML vector copy is authoritative and the serialization guard
stays scoped to the duckdb driver alone.
Collections keyed by string or UUID point IDs are rejected at attach time. The
read path parses point ids as u64 inside a filter_map, so such a collection
would otherwise yield zero results with no error.
The three response-shape parsers are pure functions over an already-parsed JSON
body, unit-tested against captured fixtures, with the async wrappers delegating
to them rather than duplicating the logic.
'.rag' now splits its first argument, so '.rag attach <name>' no longer tries to
load a RAG literally named 'attach <name>'.
Phase 3 of the RAG driver abstraction. Adds a `DuckDbProvider` that keeps
vectors and document content in a `.duckdb` sidecar next to the existing
YAML metadata, selected by the `driver: duckdb` field.
- `src/rag/providers/duckdb.rs` (new): vector search via the vss extension
and keyword search via fts, an all-or-nothing hydration path (a partial
read is an error, never a shorter map), and an anti-wipe guard that
refuses the destructive `CREATE OR REPLACE TABLE` when `data.vectors` is
empty while `data.files` is not and the store still holds rows.
- `src/rag/mod.rs`: `sync_documents` now refreshes `bm25`/`node_to_docs`
BEFORE the fallible `provider.rebuild_indexes`. `self.data` is already
mutated by that point, so propagating a provider error afterwards would
leave the derived in-memory state describing the previous corpus while
`data` describes the new one. Both rebuilds are pure functions of
`self.data` and cannot fail, so running them first is always safe.
- `src/config/paths.rs`: sidecar path helpers.
- `src/rag/providers/mod.rs`, `src/config/agent.rs`: driver dispatch and
RAG cache keying.
Also keeps `RequestContext::rag_key` in lockstep with `rag` at the two
sites that were still missing it, so that a cache insert and its matching
invalidate are structurally incapable of disagreeing:
- `use_agent` assigned `self.rag` from the agent but never set `rag_key`.
This one was live. Agent RAGs are inserted under `RagKey::Agent(<name>)`,
so with `rag_key == None` the invalidation guards in `rebuild_rag` and
`edit_rag_docs` matched nothing and `.rebuild rag` left the stale cache
entry in place. Worse, a preceding `.rag <name>` left a stale
`Named(<name>)` key attached to the agent's RAG, pointing the
invalidation at an unrelated RAG's cache entry. Now mirrors the insert
key exactly, yielding `None` when the agent has no RAG.
- `exit_agent` cleared `self.rag` but left `rag_key` behind. Latent rather
than live, since `rebuild_rag`/`edit_rag_docs` both bail on
`rag.is_none()` before reaching the invalidate guards, but the guards
that make it unobservable are not the kind of thing to depend on.
Covered by `use_agent_does_not_carry_stale_rag_key`, and by a new
assertion in `exit_agent_clears_all_agent_state`.
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).
Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
`attached: bool` as the first two fields of RagData, so driver metadata sits
at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
It is deliberately not derived: a derived impl yields `driver: ""`, which is
not a valid driver string.
Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
It enforces the (driver, attached) matrix and, critically, numeric floors
that the new defaults would otherwise mask: `top_k >= 1` unconditionally
(a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
`chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
this, the new load-time floor turns one keystroke into an unloadable RAG:
the setter saves immediately and no dot-command can reach the file again.
Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
nothing, because the content-hash skip fired regardless of the refresh flag.
Extract that decision into a module-level `find_hash_skip()` free function and
thread a `force_reingest` flag through `sync_documents()` and
`refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
now prints a one-line file-count warning first (no prompt: the path is
reachable from a non-interactive CLI flag).
Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
driver/attached through `Rag::export()` so `.info rag` shows them.
Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
- Hang fix: 120s timeout on compression LLM call + raw_stream
channel-close handling (None => break instead of spinning)
- Supervisor effective active count: stop counting finished
JoinHandles as occupying capacity slots
- Universal tool result size cap: truncate_if_needed() on
ToolResult, applied in eval_tool_calls() after escalation block;
configurable via max_tool_result_chars in AppConfig/AgentConfig
- Windowed compression: compression_keep_last config param keeps
the N most recent messages visible after compression
- Fix pre-existing flaky test: add #[serial] to
handle_list_available_unrestricted_when_no_whitelist so it does
not race with TestConfigDirGuard-based tests that temporarily
populate the agents data dir
Dark-Alex-17