diff --git a/Agents.md b/Agents.md index 691ca0e..6023d9d 100644 --- a/Agents.md +++ b/Agents.md @@ -310,7 +310,7 @@ of documents that Coyote supports and how to use custom document loaders, refer Anytime your agent starts up, it will automatically be using the RAG you've defined here. The first time an agent starts with documents attached, Coyote offers to build the knowledge base and asks which -[storage driver](RAG#storage-drivers) to store it with — `yaml` (the default, and what Coyote has always done) or +[storage driver](RAG#storage-drivers) to store it with. `yaml` (the default, and what Coyote has always done) or `duckdb`, which keeps the vectors and text on disk so a large corpus doesn't have to be read back into memory on every start. The choice is fixed once the knowledge base is built: changing it means removing the agent's built knowledge base and letting it rebuild, which re-embeds every document. diff --git a/Graph-Agents.md b/Graph-Agents.md index 1b1a561..f0e9753 100644 --- a/Graph-Agents.md +++ b/Graph-Agents.md @@ -585,12 +585,12 @@ base is first built): (default: `1`). `0` returns only documents directly linked to matched entities with no traversal. Falls back to `rag_graph_hops`. Higher values surface more loosely related documents; keep at `1` for most corpora. - **`batch_size`:** Embedding-request batch size. -- **`driver`:** [Storage driver](RAG#storage-drivers) for this node's knowledge base — `yaml` +- **`driver`:** [Storage driver](RAG#storage-drivers) for this node's knowledge base. Can be `yaml` (default) or `duckdb`. Use `duckdb` when the node's corpus is large enough that rebuilding the in-memory index on every agent start is noticeable; it keeps vectors and text on disk in a `.duckdb` file beside the generated knowledge base. An invalid driver is rejected when the graph is validated, naming the offending node. Like every field in this - section it only applies when the knowledge base is *first* built — changing it later has no + section, it only applies when the knowledge base is *first* built. Changing it later has no effect until the knowledge base is deleted and rebuilt. When the node omits it and the knowledge base is built interactively, Coyote asks which driver to use; setting it here answers that question up front. The `qdrant` driver is not available here: attaching a remote collection is an diff --git a/RAG.md b/RAG.md index 654004c..c47dfa5 100644 --- a/RAG.md +++ b/RAG.md @@ -69,14 +69,14 @@ When you define RAG, Coyote will first "build" the RAG. This means that Coyote w generate [embeddings](https://huggingface.co/spaces/hesamation/primer-llm-embedding) for that text. This essentially just means that Coyote translates the document into a language the LLM can understand. -These embeddings are stored in a vector database — in memory, on disk, or in a remote Qdrant collection, depending on -the RAG's [storage driver](#storage-drivers). Coyote also indexes every document chunk in a full-text +These embeddings are stored in a vector database. It is either in memory, on disk, or in a remote Qdrant collection, +depending on the RAG's [storage driver](#storage-drivers). Coyote also indexes every document chunk in a full-text search index (BM25) for keyword-based retrieval. Building costs real time and real money: every chunk is sent to an embedding model, and every chunk is sent again to a chat model if graph RAG is enabled. Adding or editing documents with `.edit rag-docs` only processes what actually -changed, but `.rebuild rag` deliberately re-embeds the entire corpus from scratch — reach for it when a document's -contents changed underneath Coyote or the knowledge base looks wrong, not as routine maintenance. +changed, but `.rebuild rag` deliberately re-embeds the entire corpus from scratch. You should only reach for it when a +document's contents changed underneath Coyote or the knowledge base looks wrong, not as routine maintenance. If an [extractor model is configured](#graph-based-rag), Coyote additionally runs an LLM-based entity and relationship extraction pass over each document chunk and builds a **knowledge graph** of named entities and their connections. This @@ -124,13 +124,13 @@ giving the model query-specific context to answer your question. Every RAG picks a **storage driver** when it's created. The driver decides where your vectors and document text actually live: inside the RAG's own file, in a local database beside it, or in a Qdrant server you already run. -| Driver | Where the vectors live | Reach for it when | -|--------|------------------------|-------------------| +| Driver | Where the vectors live | Reach for it when | +|--------------------|-------------------------------------------------------|-----------------------------------------------------------------------------------------------| | `yaml` *(default)* | In the RAG's own YAML file, loaded into memory on use | You want the simplest thing that works, or several Coyote processes need the same RAG at once | -| `duckdb` | A `.duckdb` file beside the RAG | The corpus is large enough that re-loading it into memory on every start is annoying | -| `qdrant` | A collection on a Qdrant server you already run | Something else already built and maintains the collection | +| `duckdb` | A `.duckdb` file beside the RAG | The corpus is large enough that re-loading it into memory on every start is annoying | +| `qdrant` | A collection on a Qdrant server you already run | Something else already built and maintains the collection | -Coyote asks which one you want whenever it builds a knowledge base interactively — creating a RAG with `.rag`, an +Coyote asks which one you want whenever it builds a knowledge base interactively, creating a RAG with `.rag`, an [agent](Agents#3-initializing-rag) starting up for the first time, or a [graph agent's](Graph-Agents) `rag` node: ```text @@ -141,7 +141,7 @@ Coyote asks which one you want whenever it builds a knowledge base interactively If you don't care, take the default. `yaml` is what Coyote has always done. -> **A RAG's driver is fixed when it's created.** There is no conversion step — switching means deleting the RAG and +> **A RAG's driver is fixed when it's created.** There is no conversion step. Switching means deleting the RAG and > building it again, which re-embeds every document. Worth a moment's thought before you pick, but not worth agonizing > over: for small corpora the difference is not something you'll notice. @@ -150,7 +150,7 @@ The default. Vectors and document text are stored in the RAG's YAML file and rea where the HNSW index is built on the fly. Its useful property is that nothing holds a lock: as many Coyote processes as you like can use the same RAG -simultaneously. Its cost is startup — a large corpus means a large file to parse every time the RAG is loaded. +simultaneously. Its cost is startup: a large corpus means a large file to parse every time the RAG is loaded. ## duckdb Vectors, document text and the search indexes live in a `.duckdb` file next to the RAG, and stay there between @@ -164,22 +164,22 @@ Two things are worth knowing before you choose it: * **First use needs network access.** DuckDB's vector (`vss`) and full-text (`fts`) extensions are downloaded the first time a duckdb RAG is created, then cached and reused. On a machine that's offline from the start, create one duckdb RAG while connected and every later one works offline. Pre-installing them from a DuckDB shell is fiddlier - than it sounds — the cache is keyed by DuckDB version, and Coyote bundles its own — so connecting once is much the + than it sounds (the cache is keyed by DuckDB version, and Coyote bundles its own), so connecting once is much the easier route. Deleting the RAG removes the `.duckdb` file and its write-ahead log along with it. A duckdb RAG is **two files**: the `.yaml` and the `.duckdb` beside it. Copy, move or back up both together. The YAML deliberately holds no vectors, so a RAG that arrives without its store loads without complaint, -still lists every indexed document, and answers every query with nothing. Coyote warns when it sees that combination — -indexed files, empty store — naming the file it expected. If the store is genuinely gone, `.rebuild rag` re-embeds the -corpus from scratch; `.edit rag-docs` will *not* refill it, because unchanged documents are skipped by hash. +still lists every indexed document, and answers every query with nothing. Coyote warns when it sees that combination +(i.e. indexed files, empty store) naming the file it expected. If the store is genuinely gone, `.rebuild rag` re-embeds +the corpus from scratch; `.edit rag-docs` will *not* refill it, because unchanged documents are skipped by hash. -## qdrant — attaching an existing collection +## qdrant - attaching an existing collection The `qdrant` driver connects a RAG to a collection on a Qdrant server you already run. It is **read-only**: Coyote queries the collection and never writes to it, so whatever populated it stays the sole owner of its contents. This is -the driver to use when another system — a LangChain pipeline, an ingestion job, a colleague's tooling — already -maintains a collection you'd like to ask questions about. +the driver to use when another system, be that a LangChain pipeline, an ingestion job, a colleague's tooling, etc., and +it already maintains a collection you'd like to ask questions about. Because the collection already exists, you don't create this kind of RAG with `.rag`; you *attach* it: @@ -189,13 +189,13 @@ Because the collection already exists, you don't create this kind of RAG with `. Coyote then walks you through it: -1. **Host** — for example `qdrant.company.com:6333`. -2. **API key** — Coyote asks whether the instance needs one, then which [Vault](Vault) secret holds it (defaulting to +1. **Host:** For example `qdrant.company.com:6333`. +2. **API key:** Coyote asks whether the instance needs one, then which [Vault](Vault) secret holds it (defaulting to `QDRANT_API_KEY`). If that secret doesn't exist yet you can create it right there, without leaving the wizard. The key itself is never written into the RAG file; only the secret's name is. -3. **Collection** — Coyote connects, lists what's available, and lets you pick. +3. **Collection:** Coyote connects, lists what's available, and lets you pick. -It then checks the collection is actually usable and tells you what it found — the vector dimension, and a warning if +It then checks the collection is actually usable and tells you what it found: the vector dimension, and a warning if the collection is empty (you can attach anyway, but every query will return nothing until something writes to it). A collection has to meet two requirements: @@ -208,11 +208,11 @@ A collection has to meet two requirements: Point IDs may be integers or UUIDs; both work. > **A Qdrant on your own machine or network is contacted directly.** If `HTTP_PROXY` or `HTTPS_PROXY` are set in your -> environment, Coyote still reaches a loopback, private-range or `.local` host without going through the proxy — a +> environment, Coyote still reaches a loopback, private-range or `.local` host without going through the proxy. A > proxy that has never heard of your local server would otherwise refuse the connection outright. A remote Qdrant > continues to honour whatever your environment configures. -> **Use the same embedding model that built the collection.** Coyote can't detect a mismatch — the dimensions may line +> **Use the same embedding model that built the collection.** Coyote can't detect a mismatch. The dimensions may line > up perfectly while the vectors mean entirely different things. The symptom is retrieval that returns confident > nonsense, so it's worth checking rather than discovering later. @@ -224,15 +224,15 @@ sbx proxy automatically, so queries work inside the sandbox exactly as they do o # Supported Document Sources Coyote supports a number of document sources that can be used for RAG: -| Source | Example | Comments | -|--------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| -| Files | `/tmp/dir1/file1;/tmp/dir1/file2` | | -| Directory | `/tmp/dir` | Picks up all files in a directory and all its subdirectories | -| Directory (extensions} | `/tmp/dir2/**/*.{md,txt}` | Finds all files in all subdirectories with the specified extensions | -| Recursive Filename | `/tmp/*/COYOTE.md` | The following files will be picked up:
  • `/tmp/dir1/COYOTE.md`
  • `/tmp/dir2/subdir1/COYOTE.md`
  • `/tmp/dir2/subdir2/COYOTE.md`
| -| URL | `https://www.ohdsi.org/data-standardization/` | Downloads and loads the specified webpage into the
knowledge base | -| Recursive URL (Websites) | `https://github.com/OHDSI/Vocabulary-v5.0/wiki/**` | Crawls all pages under the given URL and loads them
into the knowledge base | -| Document Loader (custom) | `jina:https://cloud.google.com/bigquery/docs/reference/standard-sql/` | Use a custom document loader to parse the given document | +| Source | Example | Comments | +|--------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Files | `/tmp/dir1/file1;/tmp/dir1/file2` | | +| Directory | `/tmp/dir` | Picks up all files in a directory and all its subdirectories | +| Directory (extensions} | `/tmp/dir2/**/*.{md,txt}` | Finds all files in all subdirectories with the specified extensions | +| Recursive Filename | `/tmp/*/COYOTE.md` | The following files will be picked up:
  • `/tmp/dir1/COYOTE.md`
  • `/tmp/dir2/subdir1/COYOTE.md`
  • `/tmp/dir2/subdir2/COYOTE.md`
| +| URL | `https://www.ohdsi.org/data-standardization/` | Downloads and loads the specified webpage into the
knowledge base | +| Recursive URL (Websites) | `https://github.com/OHDSI/Vocabulary-v5.0/wiki/**` | Crawls all pages under the given URL and loads them
into the knowledge base | +| Document Loader (custom) | `jina:https://cloud.google.com/bigquery/docs/reference/standard-sql/` | Use a custom document loader to parse the given document | # Document Loaders Coyote only has built-in support for loading text files. But that functionality can be extended to read all kinds of files diff --git a/REPL.md b/REPL.md index 030b9e5..8188c22 100644 --- a/REPL.md +++ b/REPL.md @@ -145,15 +145,15 @@ For more information, see the [Todo System documentation](TODO-System). RAG (Retrieval Augmented Generation) enables you to load documents into the LLM so you can ask questions about it or complete tasks using the documents as additional context. -| Command | Description | -|------------------|------------------------------------------------------------------------------| -| `.rag` | Initialize or access a RAG | -| `.rag attach ` | Attach a RAG to an existing remote [Qdrant](RAG#qdrant--attaching-an-existing-collection) collection (read-only) | -| `.edit rag-docs` | Add or remove documents from the active RAG using your preferred text editor | -| `.rebuild rag` | Rebuild the active RAG from scratch, re-embedding every document | -| `.sources rag` | Show a works-cited of the sources used in the last query | -| `.info rag` | Display information about the active RAG | -| `.exit rag` | Exit the active RAG | +| Command | Description | +|----------------------|-------------------------------------------------------------------------------------------------------------------| +| `.rag` | Initialize or access a RAG | +| `.rag attach ` | Attach a RAG to an existing remote [Qdrant](RAG#qdrant---attaching-an-existing-collection) collection (read-only) | +| `.edit rag-docs` | Add or remove documents from the active RAG using your preferred text editor | +| `.rebuild rag` | Rebuild the active RAG from scratch, re-embedding every document | +| `.sources rag` | Show a works-cited of the sources used in the last query | +| `.info rag` | Display information about the active RAG | +| `.exit rag` | Exit the active RAG | ![rag](./images/rag/persistent-rag.gif) diff --git a/Sandboxes.md b/Sandboxes.md index d803bf3..18a4f38 100644 --- a/Sandboxes.md +++ b/Sandboxes.md @@ -189,7 +189,7 @@ network policy, and the API key has to reach the server without ever entering th When you attach a RAG, Coyote writes a small mixin next to it declaring the host as an allowed domain and the API key as a proxy-managed credential. On launch that mixin is discovered alongside the generated `coyote-mcp` one, the key is registered with the sbx secret store on your host, and the proxy writes it into the outgoing request header at the -network edge — the same treatment LLM provider keys get. Queries then behave identically inside and outside the +network edge (the same treatment LLM provider keys get). Queries then behave identically inside and outside the sandbox. This is also why a RAG's `api_key` must be a `{{SECRET_NAME}}` placeholder rather than the key itself: Coyote reads