`ensure_extension` fell back to `INSTALL` whenever `LOAD` failed. With a cold
extension cache every thread's `LOAD` fails at once, so every thread ran
`INSTALL` concurrently for the same extension. DuckDB installs by downloading
to a temp file and then MOVING it into `~/.duckdb/extensions/...`; POSIX allows
replacing a file other handles hold open, so Linux and macOS survived, but
Windows rejects that move with "Access is denied" and the losing threads failed.
Guard the install step with a process-global mutex and re-check `LOAD` after
acquiring it. The re-check is what bounds the work to a single install: without
it every thread queued behind the winner would still run a redundant `INSTALL`
and repeat the same move over a file that is now open.
`LOAD` is per-connection, so it still runs on every connection; only `INSTALL`
is serialized. An already-installed extension takes the pre-lock fast path and
costs neither a lock nor network. The lock is never held across the connection
mutex, so it cannot invert lock order.
Traced with strace on a cold cache under default test parallelism: before, 17
threads moved files into the store (13 racing on vss alone); after, exactly one
rename per extension.
The DuckDB schema init loaded the vss and fts extensions but nothing ever
installed them, so any machine without them already present failed with
'IO Error: Extension "vss.duckdb_extension" not found'. This surfaced as 13
failing tests in CI while passing locally, because local runs had the
extensions installed already.
Loading is attempted first so an extension that is already present costs
nothing and never touches the network; INSTALL is reached only once, on a
machine seeing the extension for the first time, and reports an actionable
message if it cannot download.
CI cached the extension directory but nothing populated it, so the cache
saved an empty directory forever. The cache key now derives from Cargo.lock
rather than a hardcoded DuckDB version, and a step on cache miss installs the
extensions so the post-job save has something to store.
Phase 3 of the RAG driver abstraction. Adds a `DuckDbProvider` that keeps
vectors and document content in a `.duckdb` sidecar next to the existing
YAML metadata, selected by the `driver: duckdb` field.
- `src/rag/providers/duckdb.rs` (new): vector search via the vss extension
and keyword search via fts, an all-or-nothing hydration path (a partial
read is an error, never a shorter map), and an anti-wipe guard that
refuses the destructive `CREATE OR REPLACE TABLE` when `data.vectors` is
empty while `data.files` is not and the store still holds rows.
- `src/rag/mod.rs`: `sync_documents` now refreshes `bm25`/`node_to_docs`
BEFORE the fallible `provider.rebuild_indexes`. `self.data` is already
mutated by that point, so propagating a provider error afterwards would
leave the derived in-memory state describing the previous corpus while
`data` describes the new one. Both rebuilds are pure functions of
`self.data` and cannot fail, so running them first is always safe.
- `src/config/paths.rs`: sidecar path helpers.
- `src/rag/providers/mod.rs`, `src/config/agent.rs`: driver dispatch and
RAG cache keying.
Also keeps `RequestContext::rag_key` in lockstep with `rag` at the two
sites that were still missing it, so that a cache insert and its matching
invalidate are structurally incapable of disagreeing:
- `use_agent` assigned `self.rag` from the agent but never set `rag_key`.
This one was live. Agent RAGs are inserted under `RagKey::Agent(<name>)`,
so with `rag_key == None` the invalidation guards in `rebuild_rag` and
`edit_rag_docs` matched nothing and `.rebuild rag` left the stale cache
entry in place. Worse, a preceding `.rag <name>` left a stale
`Named(<name>)` key attached to the agent's RAG, pointing the
invalidation at an unrelated RAG's cache entry. Now mirrors the insert
key exactly, yielding `None` when the agent has no RAG.
- `exit_agent` cleared `self.rag` but left `rag_key` behind. Latent rather
than live, since `rebuild_rag`/`edit_rag_docs` both bail on
`rag.is_none()` before reaching the invalidate guards, but the guards
that make it unobservable are not the kind of thing to depend on.
Covered by `use_agent_does_not_carry_stale_rag_key`, and by a new
assertion in `exit_agent_clears_all_agent_state`.
Dark-Alex-17