Introduce a narrow `RagProvider` trait covering vector search and content
retrieval, and make `Rag` delegate to a boxed provider instead of owning an
HNSW index directly. `YamlProvider` is the sole implementation for now.
The trait deliberately stays narrow: embeddings, chunking, BM25 keyword
search, graph RAG, entity extraction, RRF merging and persistence all remain
on `Rag`/`RagData`, so a new storage backend does not have to reimplement
Coyote's indexing logic.
Notable points:
- `fetch_content`'s ordering contract is part of the trait, not an accident.
Implementations must return results in input-`ids` order; `hybrid_search`
passes an RRF-ranked list straight to the prompt builder, so a provider
returning storage order would silently discard the ranking.
- The content store is keyed on `data.files`, never `data.vectors`. Both the
content map and BM25 now route through the new `RagData::iter_documents()`
so the two key spaces match by construction. `RagData::add` zips document
ids with embeddings and truncates silently, so ids in `files \ vectors` are
genuinely reachable.
- A provider keyword-search failure degrades to an empty ranker with a
warning rather than failing the whole query; it is one of three RRF inputs.
It deliberately does not fall back to the local BM25, which would be a
silent ranking-algorithm swap once a provider with native FTS exists.
- The rerank path builds its text and id vectors from a single `fetch_content`
result in one pass, so the reranker's positional indices cannot desync.
This is not a bit-for-bit no-op. `vector_search` now dedups by best score and
sorts globally instead of concatenating per-chunk hit lists. Single-chunk
queries (the common case) are unaffected. Multi-chunk queries get corrected
rank assignment and no longer let a document that matched several query
chunks accumulate multiple RRF contributions. There is no overall cap on the
merged pool — truncation remains `reciprocal_rank_fusion`'s job.
`RagData::get()` is removed: its only two callers were the content lookups
replaced here, and an unused private-module method fails the build under
`--deny warnings`. Its three tests were rewritten against `iter_documents()`,
one of which now guards the files-vs-vectors keying directly.
Implements Phase 2 of the RAG driver abstraction design (§6).
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).
Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
`attached: bool` as the first two fields of RagData, so driver metadata sits
at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
It is deliberately not derived: a derived impl yields `driver: ""`, which is
not a valid driver string.
Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
It enforces the (driver, attached) matrix and, critically, numeric floors
that the new defaults would otherwise mask: `top_k >= 1` unconditionally
(a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
`chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
this, the new load-time floor turns one keystroke into an unloadable RAG:
the setter saves immediately and no dot-command can reach the file again.
Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
nothing, because the content-hash skip fired regardless of the refresh flag.
Extract that decision into a module-level `find_hash_skip()` free function and
thread a `force_reingest` flag through `sync_documents()` and
`refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
now prints a one-line file-count warning first (no prompt: the path is
reachable from a non-interactive CLI flag).
Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
driver/attached through `Rag::export()` so `.info rag` shows them.
Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
run_prompt_turn previously performed a single completion and returned,
leaving tool calls unexecuted. It now mirrors start_directive's loop:
call -> execute tools -> merge results -> continue until no tool results,
then check the pending-agents guardrail before returning.
The pending-agents guardrail injects a reminder prompt when sub-agents
are pending, matching the same loop termination semantics as --headless.
The session is NOT exited between turns (unlike start_directive) so
multi-turn ACP conversations retain history across session/prompt calls.
Manual gate (tool-probe agent, fs_write + fs_read tools):
id 3 result: {"output":"DONE:probe.txt","stopReason":"end_turn"}
probe.txt exists: YES, content: hello
DEFECT 1: session/prompt now accepts the spec-shaped params as primary:
{"prompt": [{"type": "text", "text": "..."}]}
Text blocks are joined with newlines. Non-text block types are silently
ignored. The legacy params.text alias is preserved as a fallback. -32602
is returned only when neither a non-empty prompt array with text blocks
nor a non-empty text field is present.
DEFECT 2: initialize result now emits protocolVersion as integer 1
instead of the string "1", matching the ACP spec's InitializeResponse.
Tests: 4 new unit tests pin the spec-shaped prompt path, the non-text
block ignore behavior, the missing-both -32602 path, and the numeric
protocolVersion type.
All CLI flag processing (--agent, --role, --rag, --model, --no-memory,
--no-stream, --no-workspace-instructions, etc.) now runs through run()
before the REPL/cmd split. The ACP server is dispatched right before
match is_repl, after apply_prelude and skills loading, so it benefits
from the complete context setup with no duplication or drift risk.
These CLI flags were previously ignored because the ACP branch returned
before run() could apply them. Now the context is configured with the
requested agent, role, RAG index, and model before the server starts.
Session management remains protocol-driven via session/new and session/load.
RenderMode::Silent was incorrectly applied to --headless in addition to
--acp-server. Standalone headless should still display the LLM response;
only ACP mode requires stdout purity for JSON-RPC. The acp server's
run_prompt_turn already sets Silent before each prompt call.
Dark-Alex-17
github-actions[bot]