feat(rag): add driver/attached fields, validation floors and force-reingest
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a). Data model: - Add `driver: String` (serde default "yaml" via RagData::default_driver) and `attached: bool` as the first two fields of RagData, so driver metadata sits at the top of each RAG YAML. Old files without them load unchanged. - Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty map renders no `vectors:` key. - Add a hand-written `impl Default for RagData` delegating to `RagData::new()`. It is deliberately not derived: a derived impl yields `driver: ""`, which is not a valid driver string. Validation (the price of the new serde defaults): - Add `RagData::validate()`, called from `Rag::load()` after deserialization. It enforces the (driver, attached) matrix and, critically, numeric floors that the new defaults would otherwise mask: `top_k >= 1` unconditionally (a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus `chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real divide-by-zero panic while sizing embedding batches). - Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without this, the new load-time floor turns one keystroke into an unloadable RAG: the setter saves immediately and no dot-command can reach the file again. Rebuild actually re-embeds now: - `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded nothing, because the content-hash skip fired regardless of the refresh flag. Extract that decision into a module-level `find_hash_skip()` free function and thread a `force_reingest` flag through `sync_documents()` and `refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs` stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()` now prints a one-line file-count warning first (no prompt: the path is reachable from a non-interactive CLI flag). Attached-RAG guards: - Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs, which Coyote did not index and whose source documents it does not own. - Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface driver/attached through `Rag::export()` so `.info rag` shows them. Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a forced re-ingest does not hash-skip while an ordinary refresh still does.
This commit is contained in:
@@ -2773,7 +2773,12 @@ impl RequestContext {
|
||||
}
|
||||
}
|
||||
"rag_top_k" => {
|
||||
let value = value.parse().with_context(|| "Invalid value")?;
|
||||
let value: usize = value.parse().with_context(|| "Invalid value")?;
|
||||
if value == 0 {
|
||||
bail!(
|
||||
"rag_top_k must be >= 1; a top_k of 0 makes every query return no results."
|
||||
);
|
||||
}
|
||||
if !self.set_rag_top_k(value)? {
|
||||
self.update_app_config(|app| app.rag_top_k = value);
|
||||
}
|
||||
@@ -4125,6 +4130,12 @@ impl RequestContext {
|
||||
None => bail!("No RAG"),
|
||||
};
|
||||
|
||||
if rag.is_attached() {
|
||||
bail!(
|
||||
"Cannot edit documents on an attached RAG — Coyote does not own its source documents."
|
||||
);
|
||||
}
|
||||
|
||||
let document_paths = rag.document_paths();
|
||||
let temp_file = temp_file(&format!("-rag-{}", rag.name()), ".txt");
|
||||
tokio::fs::write(&temp_file, &document_paths.join("\n"))
|
||||
@@ -4157,8 +4168,14 @@ impl RequestContext {
|
||||
};
|
||||
self.rag_cache().invalidate(&key);
|
||||
|
||||
rag.refresh_document_paths(&new_document_paths, false, &self.app.config, abort_signal)
|
||||
.await?;
|
||||
rag.refresh_document_paths(
|
||||
&new_document_paths,
|
||||
false,
|
||||
false,
|
||||
&self.app.config,
|
||||
abort_signal,
|
||||
)
|
||||
.await?;
|
||||
self.rag = Some(Arc::new(rag));
|
||||
Ok(())
|
||||
}
|
||||
@@ -4169,6 +4186,14 @@ impl RequestContext {
|
||||
None => bail!("No RAG"),
|
||||
};
|
||||
|
||||
if rag.is_attached() {
|
||||
bail!(
|
||||
"Cannot rebuild an attached RAG — Coyote does not own its source documents. \
|
||||
Re-index from the system that originally created '{}'.",
|
||||
rag.name()
|
||||
);
|
||||
}
|
||||
|
||||
let key = if self.agent.is_some() {
|
||||
RagKey::Agent(rag.name().to_string())
|
||||
} else {
|
||||
@@ -4177,7 +4202,12 @@ impl RequestContext {
|
||||
self.rag_cache().invalidate(&key);
|
||||
|
||||
let document_paths = rag.document_paths().to_vec();
|
||||
rag.refresh_document_paths(&document_paths, true, &self.app.config, abort_signal)
|
||||
println!(
|
||||
"Rebuilding re-embeds every document ({} files). \
|
||||
This will call the embedding API and may take a while.",
|
||||
rag.file_count()
|
||||
);
|
||||
rag.refresh_document_paths(&document_paths, true, true, &self.app.config, abort_signal)
|
||||
.await?;
|
||||
self.rag = Some(Arc::new(rag));
|
||||
Ok(())
|
||||
|
||||
Reference in New Issue
Block a user