Commit Graph
100 Commits
Author SHA1 Message Date
Dark-Alex-17 0f35e03a85 fix: properly handle OAuth refreshes
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-13 16:00:42 -06:00
Dark-Alex-17 c2b0c120d7 chore: Added grok4.6 to models.yaml
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-13 14:20:35 -06:00
Dark-Alex-17 65c9be36b2 fix: correct newline removal from fs_write and fs_patch 2026-08-13 14:18:45 -06:00
Dark-Alex-17 2658ca776e feat: Installed duckdb into the coyote image
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 19:29:26 -06:00
Dark-Alex-17 f8682102a0 docs: Added duckdb prerequisite
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 19:07:31 -06:00
Dark-Alex-17 3fa0f5c428 feat: Support managing MCP servers from the CLI directly
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 17:55:33 -06:00
Dark-Alex-17 68135b97d1 feat: append new built-in rag__query function to RAG contexts to allow further querying by LLMs
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 16:25:50 -06:00
Dark-Alex-17 b87a3460c4 fix: detect duplicate tool call IDs client-side before sending to Claude
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 13:33:24 -06:00
Dark-Alex-17 cb23da6490 feat: support static file bundling with sbx-mixins
CI / All (ubuntu-latest) (push) Failing after 29s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-12 13:18:38 -06:00
Dark-Alex-17 ebba976a27 fmt: applied formatting 2026-08-12 12:07:38 -06:00
Dark-Alex-17 c84f9522e9 feat(rag): offer the storage driver when an agent initializes its RAG
Agent startup and graph rag nodes both run an interactive wizard when their
knowledge base has not been built, but neither offered the driver choice that
interactive named-RAG creation has, so both silently produced a yaml store.

A plain agent was the worse of the two: AgentConfig carries only documents, so
there was no way to get a duckdb RAG for one, interactively or declaratively. A
graph node could at least declare driver: in the workflow.

Agent startup now passes prompt_for_driver, and a rag node whose wizard runs is
asked too. The prompt is skipped when the node already declares a driver, and
sits inside the not-fully-specified branch after the non-interactive bail, so
declarative workflows and headless runs are unchanged. Temp RAGs still pass
false: they are deleted on the next run, so a persistent store would only leave
a sidecar behind.

The prompt moves to select_rag_driver rather than being duplicated.
2026-08-12 12:02:15 -06:00
Dark-Alex-17 81ed769f8a fix(rag): warn when a duckdb store is empty but files are indexed
A duckdb RAG is two files. The .yaml deliberately carries no vectors, and
open() runs CREATE TABLE IF NOT EXISTS, so a .yaml copied or synced without its
.duckdb sidecar produces a fresh empty store, hydrates to nothing, and answers
every query with nothing while .info rag still lists every indexed file.

Neither existing guard catches it: the anti-wipe check in rebuild_indexes needs
existing > 0, and the mandatory ? on hydration needs a genuine error, while an
absent store is the same Ok(empty) as a RAG with nothing indexed yet.

Warn rather than bail, so a store deleted on purpose still loads and can be
rebuilt.
2026-08-12 11:26:16 -06:00
Dark-Alex-17 6f7defe25f style: removed redundant comment 2026-08-12 10:56:34 -06:00
Dark-Alex-17 b837f82d7e fix(rag): keep a local Qdrant off an ambient proxy
Reverts the global proxy rework in 54685be and narrows it to the provider.

That commit took over proxy detection for every client in order to exempt
loopback and private ranges. Too broad: reqwest's detection also reads macOS
System Settings and the Windows registry behind its system-proxy feature, which
sits in its default set. Coyote disables default features today, so hand-rolling
the environment lookup happened to match — but re-enabling defaults later would
silently restore that support for main and not for the hand-rolled version. It
also made an explicitly configured proxy skip local hosts, which nobody asked
for: a proxy named for a LAN endpoint should be used.

build_client and utils are byte-identical to main again. The bypass now lives in
QdrantProvider::make_client, which is the only place that knows the target host,
and applies solely when that host is loopback, link-local, private or .local. A
public or cloud-hosted store keeps whatever the environment configures.

Also drops apply_proxy: with build_client reverted there was one caller left, and
set_proxy already covers it.

Both #[ignore]d live tests still pass against a Qdrant on loopback while an
ambient proxy that rejects it is in force.
2026-08-12 10:52:31 -06:00
Dark-Alex-17 54685be9a2 fix: keep loopback and LAN traffic off an ambient proxy
Pre-existing on main, not introduced by the driver work, but it makes a local
RAG backend unusable so it belongs with this change.

build_client only called set_proxy when a client had configured one of its own.
With nothing configured, reqwest's own detection applied, which sends every
request through a *_PROXY variable including ones bound for 127.0.0.1 or a LAN
address. A proxy cannot usefully forward those, and anything that intercepts
proxied traffic answers on behalf of a service that is running perfectly well,
so the error names the proxy rather than the store and reads as a Coyote fault.

Concretely, an installed Socket Firewall exports HTTP_PROXY to the processes it
wraps and rejects hosts outside its allow list. That turned a healthy Ollama on
the LAN into 'error decoding response body: expected value at line 2 column 1' —
its HTML refusal page parsed as JSON — and a loopback Qdrant into an HTTP 405.

Proxy handling is now always applied and always exempts loopback and private
ranges, with NO_PROXY merged in since replacing reqwest's detection also
replaces its handling of that variable. HTTP_PROXY and HTTPS_PROXY are kept
separate because they are allowed to differ. An explicitly configured proxy
still wins, and '-' still means none.

This also supersedes the unconditional no_proxy() added to the Qdrant client in
af9622d: that made it the only client to ignore a proxy outright, on a
justification I got wrong. It now shares this path, so a remote store behind a
real proxy keeps working.
2026-08-12 10:33:03 -06:00
Dark-Alex-17 4dd6e794b2 docs: removed redundant comment 2026-08-11 22:19:14 -06:00
Dark-Alex-17 af9622d31c fix(rag): stop routing Qdrant requests through an ambient proxy
make_client used a bare reqwest builder, which honours whatever proxy the
environment advertises. That made it the only HTTP client in Coyote to do so:
utils::set_proxy discards ambient settings and applies only Coyote's configured
proxy, and every other client goes through it.

The symptom is that a perfectly healthy Qdrant is unreachable and the error
belongs to the interposing proxy, not the store, so it reads as a Coyote or
Qdrant fault. Locally an installed Socket Firewall answered `.rag attach`
against 127.0.0.1:6333 with an HTML 'Connection Required' page and HTTP 405.

Both #[ignore]d live tests now pass against a real Qdrant; they failed with that
same 405 before this change, which is the first time either has run green.

A remote store that genuinely needs Coyote's configured proxy is a follow-up:
that means threading the proxy config into the provider.
2026-08-11 22:15:04 -06:00
Dark-Alex-17 6f586bd535 style: cleanup 2026-08-11 22:07:35 -06:00
Dark-Alex-17 78740db170 chore: ignore the .coyote workspace directory
It holds generated workspace state and should never be committed.
2026-08-11 22:03:45 -06:00
Dark-Alex-17 64d594f4ee refactor(rag): discover driver_config secrets by grammar, not field name
Sandbox provisioning only ever looked at driver_config["api_key"], so a driver
whose credential is called anything else would have been silently unprovisioned
inside a sandbox. It now scans every driver_config value and treats any that is
a secret placeholder as a credential, which is the same rule resolve_driver_config
already used at point of use.

The first one binds to the RAG's own service id, which is what the generated
mixin declares; any others register under their own names, as MCP secrets do.
The mixin still carries a single credential entry, so a driver needing two bound
secrets remains a follow-up.

Also drops the placeholder parser added in 74bc613. crate::vault::SECRET_RE is
already the canonical definition and was already imported here, so that was a
third implementation of the same grammar. Requiring the whole value to match is
what keeps a literal key from being read as a secret name and printed.

The api_key check is gone from RagData::validate: a generic config validator
should not know a provider's field names.
2026-08-11 22:03:45 -06:00
Dark-Alex-17 1322d73c7b style: further cleanup 2026-08-11 21:50:46 -06:00
Dark-Alex-17 de91ffa517 fix(rag): treat a zero min_score as no floor on Qdrant searches
parse_search_hits filtered on score > min_score, and the only caller passes
0.0. Qdrant Euclid collections score by negative distance, so every hit was
dropped and an attached Euclid collection returned nothing at all, silently.

This is the same trap the surrounding code already documents: score_threshold
is deliberately not sent because it is metric-aware and a 0.0 floor filters
everything out on Euclid. The local filter then reproduced it exactly. Only a
positive floor is now treated as a floor.
2026-08-11 21:07:20 -06:00
Dark-Alex-17 74bc613d94 fix(rag): address Copilot review findings on the driver abstraction
Five review comments, all real:

- hybrid_search ran its vector and keyword legs sequentially after the
  provider refactor; main ran them under tokio::join!. Restores the
  concurrency while keeping the degrade-on-error keyword behaviour, so a
  remote provider no longer pays two serial round trips per query.

- inject_rag_secrets derived a vault secret name by trimming braces, which
  leaves a literal key untouched. A RAG holding a plaintext api_key therefore
  looked the secret up by its own value and printed it to stderr on failure.
  Parsing is now strict and a non-placeholder is skipped with a warning that
  names no credential.

- validate() now refuses a driver_config.api_key that is not a {{NAME}}
  placeholder, so a plaintext key cannot reach the RAG YAML at all.

- Rag::create's catch-all arm treated any unrecognised driver as yaml. A typo
  built a yaml store, paid to embed the corpus, persisted the bad driver and
  only failed on the next run. Unknown drivers now fail immediately.

- The qdrant arm's error was written for a developer; it now tells the user
  that only attached collections are readable and points at .rag attach.
2026-08-11 21:04:21 -06:00
Dark-Alex-17 6d0a5550fe style: Removed some redundant comments 2026-08-11 20:56:51 -06:00
Dark-Alex-17 7b1c0342b4 fix(rag): delete the DuckDB write-ahead log alongside the store
Deleting a RAG removed its .duckdb file but left the sibling .duckdb.wal
behind. DuckDB only removes that log on a clean close, so any RAG whose
process was killed left one on disk, and creating a new RAG under the same
name let it inherit a write-ahead log describing someone else's data.

The test helper already cleaned the log up after itself, which is why no
test noticed the production path did not.
2026-08-11 16:55:19 -06:00
Dark-Alex-17 d6c114fe58 feat: simplified the duckdb selection prompt 2026-08-11 16:50:50 -06:00
Dark-Alex-17 912e00a627 docs(rag): correct the duckdb concurrency note in the driver prompt
The driver picker still told users a duckdb RAG can only be open in one Coyote
process at a time. That stopped being true once the store began opening
read-only for queries: any number of processes can now query it concurrently.

The restriction that remains is narrower and only bites while writing, so the
prompt now states that instead — several processes can query at once, but an
ingest or rebuild locks the others out until it finishes.
2026-08-11 16:38:55 -06:00
Dark-Alex-17 e006e29ff1 feat(rag): let several Coyote processes query one duckdb RAG at once
The DuckDB store was always opened read-write, which takes an exclusive file
lock, so a second Coyote process could not even read the RAG. Querying does not
write, and DuckDB permits many concurrent readers as long as no writer is
attached, so the store is now opened read-only whenever it already carries a
complete schema.

Creating or initializing the store still writes, as does rebuilding, so those
paths take the exclusive handle. The rebuild path upgrades a read-only
connection in place, which every clone of the handle observes because the mode
lives behind the shared mutex rather than beside it. Extension loads and the
HNSW persistence setting are per-connection and are re-established on the
upgraded connection.

An upgrade that loses the race for the write lock reports that another process
holds the RAG and that nothing was written, then reopens read-only so the
session can keep querying. A read-only handle also refuses writes outright, so a
missed upgrade cannot silently discard an ingest.
2026-08-11 14:58:18 -06:00
Dark-Alex-17 c0067d387c refactor(rag): drop the hardcoded embedding model hint from attach
The attach wizard mapped a collection's vector dimension to a hardcoded list of
model names and printed them as likely candidates. The list was never checked
against the models the user actually has configured, so it could recommend a
model they cannot select, and one entry was a parenthetical note rather than a
model id and so could never match anything. Any list like this rots as models
are released.

The dimension itself comes from the server and is worth stating, so it is still
printed, as is the warning that a mismatched embedding model returns bad
results. Deriving real candidates would need a dimension recorded against each
configured model, which the model config does not carry today.
2026-08-11 14:06:57 -06:00
Dark-Alex-17 118c346345 feat(rag): support string and UUID Qdrant point IDs
Point ids were read with as_u64() inside a filter_map, so a string id was
silently dropped and a UUID-keyed collection returned zero hits with no error.
The attach wizard therefore refused such collections and told the user to
rebuild with integer ids, which defeats the purpose of attaching to a
collection someone else already built. LangChain, a common way to populate
Qdrant, uses UUIDs by default.

The integer id was never load-bearing for this driver. DocumentId is a packed
(file, chunk) pair used positionally by the local drivers, but an attached RAG
holds no local files or vectors and every positional consumer already returns
early on it, so the id only has to survive the round trip from search back to
the content fetch. Ids that cannot make that trip as a u64 are interned behind
a synthetic handle and restored when the fetch is issued, leaving collections
that already use integer ids on exactly the path they used before.
2026-08-11 14:05:38 -06:00
Dark-Alex-17 dc677a2529 fix(sandbox): discover agent-scoped RAG mixin sidecars
An agent-scoped RAG writes its config to <data>/agents/<agent>/<rag>.yaml, so
its sbx mixin sidecar lands beside it as <rag>.sbx-mixin.yaml. Discovery scanned
the agents directory only for a file named exactly sbx-mixin.yaml, and scanned
for suffixed sidecars only in the top-level rags directory, so a RAG attached
while an agent was active contributed no network allow rule and no credential to
the sandbox. The failure was silent: the sandbox launched and the RAG was simply
unreachable from inside it.

The two collectors differed only in the filename shape they matched, so they are
now one scan that takes the set of layouts to look for. The agents directory
asks for both its own sbx-mixin.yaml and the suffixed sidecars one level in,
which is the shape that was missing. Discovery order is unchanged, and it is
load-bearing: each mixin becomes a --kit in list order and later ones layer over
earlier ones, so the workspace mixin must stay last.
2026-08-11 14:05:31 -06:00
Dark-Alex-17 5e2b9c98ad fix(rag): stop the attach wizard from silently accepting an empty collection
`sample_point_id` returns `None` for a collection with no points, so the
UUID guard's `if let Some(..)` fell straight through and the wizard attached
happily. The result is a RAG that answers every query with zero hits and
never says why.

Sample once, then check for emptiness explicitly. This warns and asks rather
than hard-failing: an empty collection is not necessarily a mistake, since
another tool may be about to populate it, and none of the wizard's remaining
probes can distinguish that from a misconfiguration. The confirmation
defaults to "no" so it cannot be walked past by accident, and `attach`
already refuses to run non-interactively, so no unattended path reaches it.
2026-08-11 13:55:59 -06:00
Dark-Alex-17 c458ca93a9 feat(rag): create the API key secret inline in the attach wizard
The wizard hard-errored with "Secret 'X' not found in vault. Run
`coyote --add-secret X` first.", throwing away every answer the user had
already given it. Offer to create the secret in place instead, deferring to
`Vault::add_secret` for the masked prompt, the provider write and the
confirmation line, then read it back.

Only a genuine `SecretError::NotFound` triggers the offer. An auth failure,
a provider outage, or the vault being disabled inside a sandbox all
propagate with their own message, because prompting for a value that cannot
be stored would fail one step later and bury the real cause. Declining the
offer fails with both ways out spelled: add the secret up front, or answer
"no" to the API-key question.
2026-08-11 13:50:33 -06:00
Dark-Alex-17 860566bf50 feat: let workflow rag nodes select a RAG driver
`RagNode` gains an optional `driver`, forwarded into `RagInitConfig` so a
graph node can build its knowledge base on duckdb instead of yaml. Nodes
that name no driver forward `None`, which still resolves to yaml, so
existing workflows are unaffected.

An unknown driver is rejected up front rather than at construction time.
`Rag::create` dispatches unknown drivers to its yaml catch-all, so a typo
would otherwise embed every document and persist the bogus string, after
which every subsequent load fails validation and the agent cannot start.
The check asks `RagData::validate()` through a probe value instead of
restating the list of valid drivers, so the two cannot drift.
2026-08-11 13:46:59 -06:00
Dark-Alex-17 7f90710427 refactor(rag): interpolate every driver_config value, not just api_key
Only `driver_config["api_key"]` was interpolated, so any credential-bearing
driver field added later would have shipped its raw `{{PLACEHOLDER}}` to the
server. Resolve every value instead, via `resolve_driver_config`.

Resolution still happens into a function-local copy and never touches
`RagData`: `save()` serializes `self.data` and is called by `.set rag_top_k`
and friends, so a resolved credential parked there would be written to the
RAG's YAML in plaintext. The literal `{{NAME}}` also has to survive on disk
because sandbox credential provisioning parses it back out to learn which
vault secret to bind. Scope stays `driver_config` deliberately: the rest of
a RAG file is ingested document text, where `{{...}}` is ordinary content.

Covered by a test that saves after a load and asserts the placeholder, not
the secret, is what reaches the file.
2026-08-11 13:46:32 -06:00
Dark-Alex-17 93a934439b fix(rag): fail loudly when a RAG's vault secret is missing
`interpolate_secrets` does not error on a secret the vault cannot resolve:
it substitutes the empty string and returns the name in its second tuple
element. `load_async` discarded that vec, so a typo'd or deleted vault
secret produced `api_key = ""` and an unexplained 401 from Qdrant.

Bail instead, naming the RAG and the missing secrets, matching what global
config loading already does.
2026-08-11 13:44:00 -06:00
Dark-Alex-17 ecda258d3a style: Cleaned up some minor styling issues 2026-08-11 13:04:27 -06:00
Dark-Alex-17 3e598065f8 fix(rag): serialize DuckDB extension installs to stop a Windows race
`ensure_extension` fell back to `INSTALL` whenever `LOAD` failed. With a cold
extension cache every thread's `LOAD` fails at once, so every thread ran
`INSTALL` concurrently for the same extension. DuckDB installs by downloading
to a temp file and then MOVING it into `~/.duckdb/extensions/...`; POSIX allows
replacing a file other handles hold open, so Linux and macOS survived, but
Windows rejects that move with "Access is denied" and the losing threads failed.

Guard the install step with a process-global mutex and re-check `LOAD` after
acquiring it. The re-check is what bounds the work to a single install: without
it every thread queued behind the winner would still run a redundant `INSTALL`
and repeat the same move over a file that is now open.

`LOAD` is per-connection, so it still runs on every connection; only `INSTALL`
is serialized. An already-installed extension takes the pre-lock fast path and
costs neither a lock nor network. The lock is never held across the connection
mutex, so it cannot invert lock order.

Traced with strace on a cold cache under default test parallelism: before, 17
threads moved files into the store (13 racing on vss alone); after, exactly one
rename per extension.
2026-08-10 16:13:36 -06:00
Dark-Alex-17 3abc30d633 fix(rag): emit an sbx kit v2 mixin and declare RAG credentials to the proxy
The RAG attach sidecar was written against the sbx kit v1 spec and still emitted schemaVersion "1" with network.allowedDomains, network.serviceDomains, network.serviceAuth, credentials.sources.<n>.env and environment.proxyManaged. Every one of those keys was removed in kit v2. Coyote does not validate mixins, it copies them byte-for-byte into spec.yaml, so the invalid document surfaced only as an opaque sbx failure with no indication of which mixin caused it.

generate_rag_sbx_mixin now builds the document from the shared serializer structs instead of a format! string, which is how the envelope drifted unnoticed in the first place. render_mixin_yaml and the RAG sidecar both go through a new render_mixin_document, giving one definition of the envelope and one enforcement point for the rule that every inject domain must also appear in permissions.network.allow.

Fix an auth bug the port exposed: inject_rag_secrets bound the API key with sbx secret set, but nothing ever emitted a matching credentials entry, so the proxy held a value with no inject rule and never rewrote the auth header. An attached RAG credential silently did not work inside the sandbox. The sidecar now declares that credential; a RAG with no API key declares none while still receiving egress.

Fix the service id: the bind passed the raw file stem instead of routing it through secret_service_id, so a RAG named My_Docs produced an illegal id. The bind and the generated credentials service now share that derivation and cannot disagree.

Retire sbx_domain_forms in favour of allow_entry_for_url, now pub(crate). It emitted both a bare host and host:port because v1 serviceDomains needed a bare key; v2 has no such need, so the extra entry is simply wrong. It also defaulted a schemeless host to port 6333 while normalize_base_url resolves it to http and port 80, meaning the allow entry named a port the client never dialled.
2026-08-10 15:58:40 -06:00
Dark-Alex-17 f68937611e fix(rag): install DuckDB vss and fts extensions when they are missing
The DuckDB schema init loaded the vss and fts extensions but nothing ever
installed them, so any machine without them already present failed with
'IO Error: Extension "vss.duckdb_extension" not found'. This surfaced as 13
failing tests in CI while passing locally, because local runs had the
extensions installed already.

Loading is attempted first so an extension that is already present costs
nothing and never touches the network; INSTALL is reached only once, on a
machine seeing the extension for the first time, and reports an actionable
message if it cannot download.

CI cached the extension directory but nothing populated it, so the cache
saved an empty directory forever. The cache key now derives from Cargo.lock
rather than a hardcoded DuckDB version, and a step on cache miss installs the
extensions so the post-job save has something to store.
2026-08-10 15:47:15 -06:00
Dark-Alex-17 a12cf84eb6 Merge remote-tracking branch 'refs/remotes/origin/main' 2026-08-10 15:41:50 -06:00
Dark-Alex-17 2b45e3a9b8 feat: improved wording and heuristic detection for sisyphus suite of agents
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-10 15:37:53 -06:00
Dark-Alex-17 91dbaf5533 feat: upgraded to sbx kit v2 spec for improved integration
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-10 15:26:16 -06:00
Dark-Alex-17 7a732436aa feat(rag): add attach-only Qdrant provider, attach wizard and sandbox wiring
Adds QdrantProvider as a read-only driver for pre-existing remote Qdrant
collections, an interactive '.rag attach' wizard, and the sandbox credential
and domain-whitelisting wiring that lets an attached RAG work inside a sandbox.

Attach-only by design: rebuild_indexes bails for both the attached and the
unattached case rather than silently succeeding. Coyote never writes to Qdrant
in this change.

Vectors are never hydrated back from Qdrant. Cosine collections L2-normalize
stored vectors on write, so reading them back returns unit-length copies of the
originals; the YAML vector copy is authoritative and the serialization guard
stays scoped to the duckdb driver alone.

Collections keyed by string or UUID point IDs are rejected at attach time. The
read path parses point ids as u64 inside a filter_map, so such a collection
would otherwise yield zero results with no error.

The three response-shape parsers are pure functions over an already-parsed JSON
body, unit-tested against captured fixtures, with the async wrappers delegating
to them rather than duplicating the logic.

'.rag' now splits its first argument, so '.rag attach <name>' no longer tries to
load a RAG literally named 'attach <name>'.
2026-08-10 13:40:22 -06:00
Dark-Alex-17 98d3ba4a83 feat(rag): add DuckDB provider behind the RAG driver abstraction
Phase 3 of the RAG driver abstraction. Adds a `DuckDbProvider` that keeps
vectors and document content in a `.duckdb` sidecar next to the existing
YAML metadata, selected by the `driver: duckdb` field.

- `src/rag/providers/duckdb.rs` (new): vector search via the vss extension
  and keyword search via fts, an all-or-nothing hydration path (a partial
  read is an error, never a shorter map), and an anti-wipe guard that
  refuses the destructive `CREATE OR REPLACE TABLE` when `data.vectors` is
  empty while `data.files` is not and the store still holds rows.
- `src/rag/mod.rs`: `sync_documents` now refreshes `bm25`/`node_to_docs`
  BEFORE the fallible `provider.rebuild_indexes`. `self.data` is already
  mutated by that point, so propagating a provider error afterwards would
  leave the derived in-memory state describing the previous corpus while
  `data` describes the new one. Both rebuilds are pure functions of
  `self.data` and cannot fail, so running them first is always safe.
- `src/config/paths.rs`: sidecar path helpers.
- `src/rag/providers/mod.rs`, `src/config/agent.rs`: driver dispatch and
  RAG cache keying.

Also keeps `RequestContext::rag_key` in lockstep with `rag` at the two
sites that were still missing it, so that a cache insert and its matching
invalidate are structurally incapable of disagreeing:

- `use_agent` assigned `self.rag` from the agent but never set `rag_key`.
  This one was live. Agent RAGs are inserted under `RagKey::Agent(<name>)`,
  so with `rag_key == None` the invalidation guards in `rebuild_rag` and
  `edit_rag_docs` matched nothing and `.rebuild rag` left the stale cache
  entry in place. Worse, a preceding `.rag <name>` left a stale
  `Named(<name>)` key attached to the agent's RAG, pointing the
  invalidation at an unrelated RAG's cache entry. Now mirrors the insert
  key exactly, yielding `None` when the agent has no RAG.
- `exit_agent` cleared `self.rag` but left `rag_key` behind. Latent rather
  than live, since `rebuild_rag`/`edit_rag_docs` both bail on
  `rag.is_none()` before reaching the invalidate guards, but the guards
  that make it unobservable are not the kind of thing to depend on.

Covered by `use_agent_does_not_carry_stale_rag_key`, and by a new
assertion in `exit_agent_clears_all_agent_state`.
2026-08-10 12:51:56 -06:00
Dark-Alex-17 d734276927 build(rag): add duckdb dependency and pin comfy-table to 7.1.4
Adds the duckdb crate with the bundled feature ahead of any provider
code, so the dependency and build surface can be proven on every CI
target on its own.

duckdb constrains comfy-table to ~7.1, so comfy-table moves from 7.2.2
to 7.1.4 while keeping custom_styling. That feature swaps
measure_text_width for an ANSI-stripping implementation that
render_table's pre-styled cells depend on; dropping it still compiles
and still passes every other test, and only corrupts column widths. A
regression test now renders a styled table at a fixed wrap width and
asserts every line has an equal ANSI-stripped display width.

comfy-table 7.1.4 pins crossterm 0.28 while coyote pins 0.29, so both
now build side by side. comfy_table::Color, Attribute and Cell are
consequently crossterm 0.28 types and must not be used; table styling
stays ANSI-string based.

Caches the DuckDB extension directory in CI, keyed on the runner OS and
the DuckDB version, so vss and fts survive an upstream outage.

Also switches merge_vector_results to f32::total_cmp. The previous
partial_cmp().unwrap_or(Equal) comparator is not total under NaN, which
sort_by is permitted to answer with a panic in release builds.
2026-08-10 11:59:57 -06:00
Dark-Alex-17 5049143fcc refactor(rag): extract RagProvider trait and add YamlProvider
Introduce a narrow `RagProvider` trait covering vector search and content
retrieval, and make `Rag` delegate to a boxed provider instead of owning an
HNSW index directly. `YamlProvider` is the sole implementation for now.

The trait deliberately stays narrow: embeddings, chunking, BM25 keyword
search, graph RAG, entity extraction, RRF merging and persistence all remain
on `Rag`/`RagData`, so a new storage backend does not have to reimplement
Coyote's indexing logic.

Notable points:

- `fetch_content`'s ordering contract is part of the trait, not an accident.
  Implementations must return results in input-`ids` order; `hybrid_search`
  passes an RRF-ranked list straight to the prompt builder, so a provider
  returning storage order would silently discard the ranking.
- The content store is keyed on `data.files`, never `data.vectors`. Both the
  content map and BM25 now route through the new `RagData::iter_documents()`
  so the two key spaces match by construction. `RagData::add` zips document
  ids with embeddings and truncates silently, so ids in `files \ vectors` are
  genuinely reachable.
- A provider keyword-search failure degrades to an empty ranker with a
  warning rather than failing the whole query; it is one of three RRF inputs.
  It deliberately does not fall back to the local BM25, which would be a
  silent ranking-algorithm swap once a provider with native FTS exists.
- The rerank path builds its text and id vectors from a single `fetch_content`
  result in one pass, so the reranker's positional indices cannot desync.

This is not a bit-for-bit no-op. `vector_search` now dedups by best score and
sorts globally instead of concatenating per-chunk hit lists. Single-chunk
queries (the common case) are unaffected. Multi-chunk queries get corrected
rank assignment and no longer let a document that matched several query
chunks accumulate multiple RRF contributions. There is no overall cap on the
merged pool — truncation remains `reciprocal_rank_fusion`'s job.

`RagData::get()` is removed: its only two callers were the content lookups
replaced here, and an unused private-module method fails the build under
`--deny warnings`. Its three tests were rewritten against `iter_documents()`,
one of which now guards the files-vs-vectors keying directly.

Implements Phase 2 of the RAG driver abstraction design (§6).
2026-08-10 11:41:37 -06:00
Dark-Alex-17 a968c3228d feat(rag): add driver/attached fields, validation floors and force-reingest
Phase 1 of the RAG driver abstraction (design doc sections 5.1-5.5a).

Data model:
- Add `driver: String` (serde default "yaml" via RagData::default_driver) and
  `attached: bool` as the first two fields of RagData, so driver metadata sits
  at the top of each RAG YAML. Old files without them load unchanged.
- Add `#[serde(default)]` to the non-Option fields so a minimal attached-RAG
  YAML deserializes, and add `skip_serializing_if` to `vectors` so an empty
  map renders no `vectors:` key.
- Add a hand-written `impl Default for RagData` delegating to `RagData::new()`.
  It is deliberately not derived: a derived impl yields `driver: ""`, which is
  not a valid driver string.

Validation (the price of the new serde defaults):
- Add `RagData::validate()`, called from `Rag::load()` after deserialization.
  It enforces the (driver, attached) matrix and, critically, numeric floors
  that the new defaults would otherwise mask: `top_k >= 1` unconditionally
  (a 0 makes every query return nothing, silently), and `chunk_size >= 1` plus
  `chunk_overlap < chunk_size` when not attached (a 0 chunk_size is a real
  divide-by-zero panic while sizing embedding batches).
- Reject `.set rag_top_k 0` at the setter, before the set/update fork. Without
  this, the new load-time floor turns one keystroke into an unloadable RAG:
  the setter saves immediately and no dot-command can reach the file again.

Rebuild actually re-embeds now:
- `.rebuild rag` and `--rebuild-rag` previously re-scanned paths and re-embedded
  nothing, because the content-hash skip fired regardless of the refresh flag.
  Extract that decision into a module-level `find_hash_skip()` free function and
  thread a `force_reingest` flag through `sync_documents()` and
  `refresh_document_paths()`, set true only from `rebuild_rag()`. `.edit rag-docs`
  stays incremental. Re-embedding costs time and API spend, so `rebuild_rag()`
  now prints a one-line file-count warning first (no prompt: the path is
  reachable from a non-interactive CLI flag).

Attached-RAG guards:
- Block `.rebuild rag` / `--rebuild-rag` and `.edit rag-docs` on attached RAGs,
  which Coyote did not index and whose source documents it does not own.
- Add `Rag::driver()`, `Rag::is_attached()` and `Rag::file_count()`, and surface
  driver/attached through `Rag::export()` so `.info rag` shows them.

Adds 12 unit tests (1299 -> 1311), including the two gate tests pinning that a
forced re-ingest does not hash-skip while an ordinary refresh still does.
2026-08-10 11:04:51 -06:00
Dark-Alex-17 f404acdbca Merge branch 'main' 2026-08-06 16:39:02 -06:00
Dark-Alex-17 efa570267d feat(mcp): send RFC 8707 resource indicator in OAuth flows
CI / All (ubuntu-latest) (push) Failing after 27s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-06 16:13:06 -06:00
Dark-Alex-17 9aeb9e6e2e Merge branch 'main' 2026-08-06 13:13:03 -06:00
Dark-Alex-17 3607a180d9 fix: don't output thinking blocks for claude-based models
CI / All (ubuntu-latest) (push) Failing after 23s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-06 13:12:36 -06:00
Dark-Alex-17 d429def0f6 fix: additional edge case fix for duplicate tool call IDs in anthropic API calls
CI / All (ubuntu-latest) (push) Failing after 27s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-06 10:30:33 -06:00
Dark-Alex-17 e606eb7c49 fix: removed temperature modifier in librarian agent to mitigate invisible errors
CI / All (ubuntu-latest) (push) Failing after 27s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-05 16:10:19 -06:00
Dark-Alex-17 a8fb32b6bd fix: strip reasoning blocks for structured LLM output in graph agents 2026-08-05 16:10:03 -06:00
Dark-Alex-17 1f7b8417fa fix: prevent rare duplicate tool call IDs in long running claude prompts 2026-08-05 16:01:40 -06:00
Dark-Alex-17 9540345ec7 feat: Added loaded indicators to .list tools/mcp-servers/skills
CI / All (ubuntu-latest) (push) Failing after 27s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-05 13:11:24 -06:00
Dark-Alex-17 70b6d51b55 test: Implemented unit tests to prevent regression on agent reasoning effort inheritance
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-04 13:12:37 -06:00
Dark-Alex-17 9e5e8a60f2 fix: agents inherit global reasoning effort if unset 2026-08-04 13:09:13 -06:00
Dark-Alex-17 d6447603bc feat: hide .recover from tab completions when no session is active
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-04 12:05:23 -06:00
Dark-Alex-17 e6dc24beb5 feat: add a new .recover command for sessions to recover from errors
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-04 12:02:31 -06:00
Dark-Alex-17 e76c3efe4e feat: integrated the git-ssh-sign kit into the coyote sandbox kit
CI / All (ubuntu-latest) (push) Failing after 26s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-04 10:56:50 -06:00
Dark-Alex-17 888529f381 fix: infinite loop bug when attempting to interrupt a prompt exchange right before a session compression
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-31 16:00:26 -06:00
Dark-Alex-17 35e75e5b4f fix: ctrl-c inside of an auto-continue loop created an infinite loop
CI / All (ubuntu-latest) (push) Failing after 26s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-31 11:44:16 -06:00
Dark-Alex-17 4f38214681 fix: sbx update doesn't allow undefined fields in sbx spec
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-31 10:07:29 -06:00
Dark-Alex-17 69e1b98c44 fix: ctrl-c interruption doesn't discard session messages when throbber is showing
CI / All (ubuntu-latest) (push) Failing after 27s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-30 13:38:37 -06:00
Dark-Alex-17 0324436114 feat: ctrl-c interrupts ongoing prompt in a session, but lets the user inject more instructions mid-stream
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-30 10:21:19 -06:00
Dark-Alex-17 233c212d2a feat: improved function calling performance by allowing parallel tool calling
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-29 12:40:52 -06:00
Dark-Alex-17 e288b41365 fix: improper handling of fd-style globbing for directories in fs_glob 2026-07-29 12:40:33 -06:00
Dark-Alex-17 fbf6a6bdf4 fix: properly templated architect design doc path in starter commands
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-29 11:59:09 -06:00
Dark-Alex-17 57b72702b2 feat: created the architect and gatekeeper agents for dramatically improved coding performance
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-29 11:23:52 -06:00
Dark-Alex-17 38ba303c3c feat: Improved readability of session message exchange replays
CI / All (ubuntu-latest) (push) Failing after 26s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-29 10:15:35 -06:00
Dark-Alex-17 fc7bc0ff8f fix: .copy works when sessions are resumed 2026-07-29 09:50:40 -06:00
Dark-Alex-17 7f7ea758a7 fix: ACP session/prompt now drives the full tool-execution loop
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
run_prompt_turn previously performed a single completion and returned,
leaving tool calls unexecuted. It now mirrors start_directive's loop:
call -> execute tools -> merge results -> continue until no tool results,
then check the pending-agents guardrail before returning.

The pending-agents guardrail injects a reminder prompt when sub-agents
are pending, matching the same loop termination semantics as --headless.
The session is NOT exited between turns (unlike start_directive) so
multi-turn ACP conversations retain history across session/prompt calls.

Manual gate (tool-probe agent, fs_write + fs_read tools):
  id 3 result: {"output":"DONE:probe.txt","stopReason":"end_turn"}
  probe.txt exists: YES, content: hello
2026-07-28 15:42:51 -06:00
Dark-Alex-17 06b2c384e3 fix: ACP spec conformance — ContentBlock prompt params and protocolVersion type
CI / All (ubuntu-latest) (push) Failing after 26s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
DEFECT 1: session/prompt now accepts the spec-shaped params as primary:
  {"prompt": [{"type": "text", "text": "..."}]}
Text blocks are joined with newlines. Non-text block types are silently
ignored. The legacy params.text alias is preserved as a fallback. -32602
is returned only when neither a non-empty prompt array with text blocks
nor a non-empty text field is present.

DEFECT 2: initialize result now emits protocolVersion as integer 1
instead of the string "1", matching the ACP spec's InitializeResponse.

Tests: 4 new unit tests pin the spec-shaped prompt path, the non-text
block ignore behavior, the missing-both -32602 path, and the numeric
protocolVersion type.
2026-07-28 14:41:22 -06:00
Dark-Alex-17 d50de7c06a lint: fixed test ordering
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-27 19:52:51 -06:00
Dark-Alex-17 0956f08791 refactor: move ACP server dispatch into run() for shared flag setup
All CLI flag processing (--agent, --role, --rag, --model, --no-memory,
--no-stream, --no-workspace-instructions, etc.) now runs through run()
before the REPL/cmd split. The ACP server is dispatched right before
match is_repl, after apply_prelude and skills loading, so it benefits
from the complete context setup with no duplication or drift risk.
2026-07-27 19:44:11 -06:00
Dark-Alex-17 087d0c320c feat: apply --agent/--role/--rag/--model flags in --acp-server mode
These CLI flags were previously ignored because the ACP branch returned
before run() could apply them. Now the context is configured with the
requested agent, role, RAG index, and model before the server starts.
Session management remains protocol-driven via session/new and session/load.
2026-07-27 19:31:45 -06:00
Dark-Alex-17 2128390f99 fmt: applied formatting
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-27 19:26:29 -06:00
Dark-Alex-17 d008de1848 fix: restore stdout output for standalone --headless mode
RenderMode::Silent was incorrectly applied to --headless in addition to
--acp-server. Standalone headless should still display the LLM response;
only ACP mode requires stdout purity for JSON-RPC. The acp server's
run_prompt_turn already sets Silent before each prompt call.
2026-07-27 19:22:51 -06:00
Dark-Alex-17 d79787bf96 fix: suppress tool-call display in headless mode; initialize session on session/new 2026-07-27 19:14:28 -06:00
Dark-Alex-17 577c51b62f fix: skip stdin drain and set silent render mode when --acp-server is active 2026-07-27 18:59:09 -06:00
Dark-Alex-17 d462b09f80 feat: add headless profile to sbx-kit spec 2026-07-27 18:03:35 -06:00
Dark-Alex-17 b711e4983b feat: implement ACP user-interaction to request_permission bridge 2026-07-27 18:00:07 -06:00
Dark-Alex-17 6ae3efb06c feat: implement ACP session/load and session/cancel 2026-07-27 17:52:49 -06:00
Dark-Alex-17 02dd14394b feat: implement ACP session/prompt 2026-07-27 17:48:31 -06:00
Dark-Alex-17 f11d4ca760 feat: add ACP server skeleton with stdout-purity test 2026-07-27 17:31:47 -06:00
Dark-Alex-17 f1415067f2 feat: add --headless flag for unattended operation 2026-07-27 17:17:56 -06:00
Dark-Alex-17 af5c34fde5 Merge branch 'main' of github.com:Dark-Alex-17/coyote 2026-07-27 15:05:25 -06:00
Dark-Alex-17 2ffa278f2d test: testing potential nerdbox regression fix for coyote sandbox mode 2026-07-27 15:04:41 -06:00
Dark-Alex-17 f4cbee9611 docs: remove comment in spec.yaml about copying in coyote password file 2026-07-27 10:50:40 -06:00
Dark-Alex-17 e95fd1e06e ci: fix typo in coyote image tag; needs leading 'v'
CI / All (ubuntu-latest) (push) Failing after 23s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-27 09:51:23 -06:00
Dark-Alex-17 d79ea55e09 fix: include graph-agent descriptions in .agent <TAB> completions
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-27 09:34:36 -06:00
Dark-Alex-17 58d9d4c64e docs: updated help message for --fresh flag
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 18:30:45 -06:00
Dark-Alex-17 63a768aa46 fix: fresh wizard openai-compatible support
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 18:29:24 -06:00
Dark-Alex-17 2c3f671efa fix: config existence check for --fresh sandboxes
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 18:09:06 -06:00
Dark-Alex-17 8b0a536f4e feat: Dynamically detect if a selected client in the sandbox first run wizard supports oauth
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 18:07:30 -06:00
Dark-Alex-17 5a1bb569b4 feat: Add support for the --fresh flag again with host environment configuration injection
CI / All (ubuntu-latest) (push) Failing after 24s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 18:00:41 -06:00
Dark-Alex-17 0c12580836 fix: Improve coyote sandbox startup time 2026-07-24 17:23:08 -06:00
Dark-Alex-17 df909325a7 fmt: applied formatting
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-07-24 16:50:27 -06:00