Phase 3 of the RAG driver abstraction. Adds a `DuckDbProvider` that keeps
vectors and document content in a `.duckdb` sidecar next to the existing
YAML metadata, selected by the `driver: duckdb` field.
- `src/rag/providers/duckdb.rs` (new): vector search via the vss extension
and keyword search via fts, an all-or-nothing hydration path (a partial
read is an error, never a shorter map), and an anti-wipe guard that
refuses the destructive `CREATE OR REPLACE TABLE` when `data.vectors` is
empty while `data.files` is not and the store still holds rows.
- `src/rag/mod.rs`: `sync_documents` now refreshes `bm25`/`node_to_docs`
BEFORE the fallible `provider.rebuild_indexes`. `self.data` is already
mutated by that point, so propagating a provider error afterwards would
leave the derived in-memory state describing the previous corpus while
`data` describes the new one. Both rebuilds are pure functions of
`self.data` and cannot fail, so running them first is always safe.
- `src/config/paths.rs`: sidecar path helpers.
- `src/rag/providers/mod.rs`, `src/config/agent.rs`: driver dispatch and
RAG cache keying.
Also keeps `RequestContext::rag_key` in lockstep with `rag` at the two
sites that were still missing it, so that a cache insert and its matching
invalidate are structurally incapable of disagreeing:
- `use_agent` assigned `self.rag` from the agent but never set `rag_key`.
This one was live. Agent RAGs are inserted under `RagKey::Agent(<name>)`,
so with `rag_key == None` the invalidation guards in `rebuild_rag` and
`edit_rag_docs` matched nothing and `.rebuild rag` left the stale cache
entry in place. Worse, a preceding `.rag <name>` left a stale
`Named(<name>)` key attached to the agent's RAG, pointing the
invalidation at an unrelated RAG's cache entry. Now mirrors the insert
key exactly, yielding `None` when the agent has no RAG.
- `exit_agent` cleared `self.rag` but left `rag_key` behind. Latent rather
than live, since `rebuild_rag`/`edit_rag_docs` both bail on
`rag.is_none()` before reaching the invalidate guards, but the guards
that make it unobservable are not the kind of thing to depend on.
Covered by `use_agent_does_not_carry_stale_rag_key`, and by a new
assertion in `exit_agent_clears_all_agent_state`.
Adds the duckdb crate with the bundled feature ahead of any provider
code, so the dependency and build surface can be proven on every CI
target on its own.
duckdb constrains comfy-table to ~7.1, so comfy-table moves from 7.2.2
to 7.1.4 while keeping custom_styling. That feature swaps
measure_text_width for an ANSI-stripping implementation that
render_table's pre-styled cells depend on; dropping it still compiles
and still passes every other test, and only corrupts column widths. A
regression test now renders a styled table at a fixed wrap width and
asserts every line has an equal ANSI-stripped display width.
comfy-table 7.1.4 pins crossterm 0.28 while coyote pins 0.29, so both
now build side by side. comfy_table::Color, Attribute and Cell are
consequently crossterm 0.28 types and must not be used; table styling
stays ANSI-string based.
Caches the DuckDB extension directory in CI, keyed on the runner OS and
the DuckDB version, so vss and fts survive an upstream outage.
Also switches merge_vector_results to f32::total_cmp. The previous
partial_cmp().unwrap_or(Equal) comparator is not total under NaN, which
sort_by is permitted to answer with a panic in release builds.
Introduce a narrow `RagProvider` trait covering vector search and content
retrieval, and make `Rag` delegate to a boxed provider instead of owning an
HNSW index directly. `YamlProvider` is the sole implementation for now.
The trait deliberately stays narrow: embeddings, chunking, BM25 keyword
search, graph RAG, entity extraction, RRF merging and persistence all remain
on `Rag`/`RagData`, so a new storage backend does not have to reimplement
Coyote's indexing logic.
Notable points:
- `fetch_content`'s ordering contract is part of the trait, not an accident.
Implementations must return results in input-`ids` order; `hybrid_search`
passes an RRF-ranked list straight to the prompt builder, so a provider
returning storage order would silently discard the ranking.
- The content store is keyed on `data.files`, never `data.vectors`. Both the
content map and BM25 now route through the new `RagData::iter_documents()`
so the two key spaces match by construction. `RagData::add` zips document
ids with embeddings and truncates silently, so ids in `files \ vectors` are
genuinely reachable.
- A provider keyword-search failure degrades to an empty ranker with a
warning rather than failing the whole query; it is one of three RRF inputs.
It deliberately does not fall back to the local BM25, which would be a
silent ranking-algorithm swap once a provider with native FTS exists.
- The rerank path builds its text and id vectors from a single `fetch_content`
result in one pass, so the reranker's positional indices cannot desync.
This is not a bit-for-bit no-op. `vector_search` now dedups by best score and
sorts globally instead of concatenating per-chunk hit lists. Single-chunk
queries (the common case) are unaffected. Multi-chunk queries get corrected
rank assignment and no longer let a document that matched several query
chunks accumulate multiple RRF contributions. There is no overall cap on the
merged pool — truncation remains `reciprocal_rank_fusion`'s job.
`RagData::get()` is removed: its only two callers were the content lookups
replaced here, and an unused private-module method fails the build under
`--deny warnings`. Its three tests were rewritten against `iter_documents()`,
one of which now guards the files-vs-vectors keying directly.
Implements Phase 2 of the RAG driver abstraction design (§6).
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).
Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
`attached: bool` as the first two fields of RagData, so driver metadata sits
at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
It is deliberately not derived: a derived impl yields `driver: ""`, which is
not a valid driver string.
Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
It enforces the (driver, attached) matrix and, critically, numeric floors
that the new defaults would otherwise mask: `top_k >= 1` unconditionally
(a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
`chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
this, the new load-time floor turns one keystroke into an unloadable RAG:
the setter saves immediately and no dot-command can reach the file again.
Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
nothing, because the content-hash skip fired regardless of the refresh flag.
Extract that decision into a module-level `find_hash_skip()` free function and
thread a `force_reingest` flag through `sync_documents()` and
`refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
now prints a one-line file-count warning first (no prompt: the path is
reachable from a non-interactive CLI flag).
Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
driver/attached through `Rag::export()` so `.info rag` shows them.
Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
run_prompt_turn previously performed a single completion and returned,
leaving tool calls unexecuted. It now mirrors start_directive's loop:
call -> execute tools -> merge results -> continue until no tool results,
then check the pending-agents guardrail before returning.
The pending-agents guardrail injects a reminder prompt when sub-agents
are pending, matching the same loop termination semantics as --headless.
The session is NOT exited between turns (unlike start_directive) so
multi-turn ACP conversations retain history across session/prompt calls.
Manual gate (tool-probe agent, fs_write + fs_read tools):
id 3 result: {"output":"DONE:probe.txt","stopReason":"end_turn"}
probe.txt exists: YES, content: hello
DEFECT 1: session/prompt now accepts the spec-shaped params as primary:
{"prompt": [{"type": "text", "text": "..."}]}
Text blocks are joined with newlines. Non-text block types are silently
ignored. The legacy params.text alias is preserved as a fallback. -32602
is returned only when neither a non-empty prompt array with text blocks
nor a non-empty text field is present.
DEFECT 2: initialize result now emits protocolVersion as integer 1
instead of the string "1", matching the ACP spec's InitializeResponse.
Tests: 4 new unit tests pin the spec-shaped prompt path, the non-text
block ignore behavior, the missing-both -32602 path, and the numeric
protocolVersion type.
All CLI flag processing (--agent, --role, --rag, --model, --no-memory,
--no-stream, --no-workspace-instructions, etc.) now runs through run()
before the REPL/cmd split. The ACP server is dispatched right before
match is_repl, after apply_prelude and skills loading, so it benefits
from the complete context setup with no duplication or drift risk.
These CLI flags were previously ignored because the ACP branch returned
before run() could apply them. Now the context is configured with the
requested agent, role, RAG index, and model before the server starts.
Session management remains protocol-driven via session/new and session/load.
RenderMode::Silent was incorrectly applied to --headless in addition to
--acp-server. Standalone headless should still display the LLM response;
only ACP mode requires stdout purity for JSON-RPC. The acp server's
run_prompt_turn already sets Silent before each prompt call.
Dark-Alex-17
github-actions[bot]