feat(rag): add driver/attached fields, validation floors and force-reingest

Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).

Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
  `attached: bool` as the first two fields of RagData, so driver metadata sits
  at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
  YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
  map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
  It is deliberately not derived: a derived impl yields `driver: ""`, which is
  not a valid driver string.

Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
  It enforces the (driver, attached) matrix and, critically, numeric floors
  that the new defaults would otherwise mask: `top_k >= 1` unconditionally
  (a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
  `chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
  divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
  this, the new load-time floor turns one keystroke into an unloadable RAG:
  the setter saves immediately and no dot-command can reach the file again.

Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
  nothing, because the content-hash skip fired regardless of the refresh flag.
  Extract that decision into a module-level `find_hash_skip()` free function and
  thread a `force_reingest` flag through `sync_documents()` and
  `refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
  stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
  now prints a one-line file-count warning first (no prompt: the path is
  reachable from a non-interactive CLI flag).

Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
  which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
  driver/attached through `Rag::export()` so `.info rag` shows them.

Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
This commit is contained in:
2026-08-10 11:04:51 -06:00
parent f404acdbca
commit a968c3228d
2 changed files with 357 additions and 13 deletions
+34 -4
View File
@@ -2773,7 +2773,12 @@ impl RequestContext {
}
}
"rag_top_k" => {
let value = value.parse().with_context(|| "Invalid value")?;
let value: usize = value.parse().with_context(|| "Invalid value")?;
if value == 0 {
bail!(
"rag_top_k must be >= 1; a top_k of 0 makes every query return no results."
);
}
if !self.set_rag_top_k(value)? {
self.update_app_config(|app| app.rag_top_k = value);
}
@@ -4125,6 +4130,12 @@ impl RequestContext {
None => bail!("No RAG"),
};
if rag.is_attached() {
bail!(
"Cannot edit documents on an attached RAG — Coyote does not own its source documents."
);
}
let document_paths = rag.document_paths();
let temp_file = temp_file(&format!("-rag-{}", rag.name()), ".txt");
tokio::fs::write(&temp_file, &document_paths.join("\n"))
@@ -4157,8 +4168,14 @@ impl RequestContext {
};
self.rag_cache().invalidate(&key);
rag.refresh_document_paths(&new_document_paths, false, &self.app.config, abort_signal)
.await?;
rag.refresh_document_paths(
&new_document_paths,
false,
false,
&self.app.config,
abort_signal,
)
.await?;
self.rag = Some(Arc::new(rag));
Ok(())
}
@@ -4169,6 +4186,14 @@ impl RequestContext {
None => bail!("No RAG"),
};
if rag.is_attached() {
bail!(
"Cannot rebuild an attached RAG — Coyote does not own its source documents. \
Re-index from the system that originally created '{}'.",
rag.name()
);
}
let key = if self.agent.is_some() {
RagKey::Agent(rag.name().to_string())
} else {
@@ -4177,7 +4202,12 @@ impl RequestContext {
self.rag_cache().invalidate(&key);
let document_paths = rag.document_paths().to_vec();
rag.refresh_document_paths(&document_paths, true, &self.app.config, abort_signal)
println!(
"Rebuilding re-embeds every document ({} files). \
This will call the embedding API and may take a while.",
rag.file_count()
);
rag.refresh_document_paths(&document_paths, true, true, &self.app.config, abort_signal)
.await?;
self.rag = Some(Arc::new(rag));
Ok(())