Adds the duckdb crate with the bundled feature ahead of any provider
code, so the dependency and build surface can be proven on every CI
target on its own.
duckdb constrains comfy-table to ~7.1, so comfy-table moves from 7.2.2
to 7.1.4 while keeping custom_styling. That feature swaps
measure_text_width for an ANSI-stripping implementation that
render_table's pre-styled cells depend on; dropping it still compiles
and still passes every other test, and only corrupts column widths. A
regression test now renders a styled table at a fixed wrap width and
asserts every line has an equal ANSI-stripped display width.
comfy-table 7.1.4 pins crossterm 0.28 while coyote pins 0.29, so both
now build side by side. comfy_table::Color, Attribute and Cell are
consequently crossterm 0.28 types and must not be used; table styling
stays ANSI-string based.
Caches the DuckDB extension directory in CI, keyed on the runner OS and
the DuckDB version, so vss and fts survive an upstream outage.
Also switches merge_vector_results to f32::total_cmp. The previous
partial_cmp().unwrap_or(Equal) comparator is not total under NaN, which
sort_by is permitted to answer with a panic in release builds.
Introduce a narrow `RagProvider` trait covering vector search and content
retrieval, and make `Rag` delegate to a boxed provider instead of owning an
HNSW index directly. `YamlProvider` is the sole implementation for now.
The trait deliberately stays narrow: embeddings, chunking, BM25 keyword
search, graph RAG, entity extraction, RRF merging and persistence all remain
on `Rag`/`RagData`, so a new storage backend does not have to reimplement
Coyote's indexing logic.
Notable points:
- `fetch_content`'s ordering contract is part of the trait, not an accident.
Implementations must return results in input-`ids` order; `hybrid_search`
passes an RRF-ranked list straight to the prompt builder, so a provider
returning storage order would silently discard the ranking.
- The content store is keyed on `data.files`, never `data.vectors`. Both the
content map and BM25 now route through the new `RagData::iter_documents()`
so the two key spaces match by construction. `RagData::add` zips document
ids with embeddings and truncates silently, so ids in `files \ vectors` are
genuinely reachable.
- A provider keyword-search failure degrades to an empty ranker with a
warning rather than failing the whole query; it is one of three RRF inputs.
It deliberately does not fall back to the local BM25, which would be a
silent ranking-algorithm swap once a provider with native FTS exists.
- The rerank path builds its text and id vectors from a single `fetch_content`
result in one pass, so the reranker's positional indices cannot desync.
This is not a bit-for-bit no-op. `vector_search` now dedups by best score and
sorts globally instead of concatenating per-chunk hit lists. Single-chunk
queries (the common case) are unaffected. Multi-chunk queries get corrected
rank assignment and no longer let a document that matched several query
chunks accumulate multiple RRF contributions. There is no overall cap on the
merged pool — truncation remains `reciprocal_rank_fusion`'s job.
`RagData::get()` is removed: its only two callers were the content lookups
replaced here, and an unused private-module method fails the build under
`--deny warnings`. Its three tests were rewritten against `iter_documents()`,
one of which now guards the files-vs-vectors keying directly.
Implements Phase 2 of the RAG driver abstraction design (§6).
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).
Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
`attached: bool` as the first two fields of RagData, so driver metadata sits
at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
It is deliberately not derived: a derived impl yields `driver: ""`, which is
not a valid driver string.
Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
It enforces the (driver, attached) matrix and, critically, numeric floors
that the new defaults would otherwise mask: `top_k >= 1` unconditionally
(a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
`chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
this, the new load-time floor turns one keystroke into an unloadable RAG:
the setter saves immediately and no dot-command can reach the file again.
Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
nothing, because the content-hash skip fired regardless of the refresh flag.
Extract that decision into a module-level `find_hash_skip()` free function and
thread a `force_reingest` flag through `sync_documents()` and
`refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
now prints a one-line file-count warning first (no prompt: the path is
reachable from a non-interactive CLI flag).
Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
driver/attached through `Rag::export()` so `.info rag` shows them.
Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
Dark-Alex-17