Compare commits

...
7 Commits
Author SHA1 Message Date
Dark-Alex-17 ecda258d3a style: Cleaned up some minor styling issues 2026-08-11 13:04:27 -06:00
Dark-Alex-17 3e598065f8 fix(rag): serialize DuckDB extension installs to stop a Windows race
`ensure_extension` fell back to `INSTALL` whenever `LOAD` failed. With a cold
extension cache every thread's `LOAD` fails at once, so every thread ran
`INSTALL` concurrently for the same extension. DuckDB installs by downloading
to a temp file and then MOVING it into `~/.duckdb/extensions/...`; POSIX allows
replacing a file other handles hold open, so Linux and macOS survived, but
Windows rejects that move with "Access is denied" and the losing threads failed.

Guard the install step with a process-global mutex and re-check `LOAD` after
acquiring it. The re-check is what bounds the work to a single install: without
it every thread queued behind the winner would still run a redundant `INSTALL`
and repeat the same move over a file that is now open.

`LOAD` is per-connection, so it still runs on every connection; only `INSTALL`
is serialized. An already-installed extension takes the pre-lock fast path and
costs neither a lock nor network. The lock is never held across the connection
mutex, so it cannot invert lock order.

Traced with strace on a cold cache under default test parallelism: before, 17
threads moved files into the store (13 racing on vss alone); after, exactly one
rename per extension.
2026-08-10 16:13:36 -06:00
Dark-Alex-17 3abc30d633 fix(rag): emit an sbx kit v2 mixin and declare RAG credentials to the proxy
The RAG attach sidecar was written against the sbx kit v1 spec and still emitted schemaVersion "1" with network.allowedDomains, network.serviceDomains, network.serviceAuth, credentials.sources.<n>.env and environment.proxyManaged. Every one of those keys was removed in kit v2. Coyote does not validate mixins, it copies them byte-for-byte into spec.yaml, so the invalid document surfaced only as an opaque sbx failure with no indication of which mixin caused it.

generate_rag_sbx_mixin now builds the document from the shared serializer structs instead of a format! string, which is how the envelope drifted unnoticed in the first place. render_mixin_yaml and the RAG sidecar both go through a new render_mixin_document, giving one definition of the envelope and one enforcement point for the rule that every inject domain must also appear in permissions.network.allow.

Fix an auth bug the port exposed: inject_rag_secrets bound the API key with sbx secret set, but nothing ever emitted a matching credentials entry, so the proxy held a value with no inject rule and never rewrote the auth header. An attached RAG credential silently did not work inside the sandbox. The sidecar now declares that credential; a RAG with no API key declares none while still receiving egress.

Fix the service id: the bind passed the raw file stem instead of routing it through secret_service_id, so a RAG named My_Docs produced an illegal id. The bind and the generated credentials service now share that derivation and cannot disagree.

Retire sbx_domain_forms in favour of allow_entry_for_url, now pub(crate). It emitted both a bare host and host:port because v1 serviceDomains needed a bare key; v2 has no such need, so the extra entry is simply wrong. It also defaulted a schemeless host to port 6333 while normalize_base_url resolves it to http and port 80, meaning the allow entry named a port the client never dialled.
2026-08-10 15:58:40 -06:00
Dark-Alex-17 f68937611e fix(rag): install DuckDB vss and fts extensions when they are missing
The DuckDB schema init loaded the vss and fts extensions but nothing ever
installed them, so any machine without them already present failed with
'IO Error: Extension "vss.duckdb_extension" not found'. This surfaced as 13
failing tests in CI while passing locally, because local runs had the
extensions installed already.

Loading is attempted first so an extension that is already present costs
nothing and never touches the network; INSTALL is reached only once, on a
machine seeing the extension for the first time, and reports an actionable
message if it cannot download.

CI cached the extension directory but nothing populated it, so the cache
saved an empty directory forever. The cache key now derives from Cargo.lock
rather than a hardcoded DuckDB version, and a step on cache miss installs the
extensions so the post-job save has something to store.
2026-08-10 15:47:15 -06:00
Dark-Alex-17 a12cf84eb6 Merge remote-tracking branch 'refs/remotes/origin/main' 2026-08-10 15:41:50 -06:00
Dark-Alex-17 2b45e3a9b8 feat: improved wording and heuristic detection for sisyphus suite of agents
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-10 15:37:53 -06:00
Dark-Alex-17 91dbaf5533 feat: upgraded to sbx kit v2 spec for improved integration
CI / All (ubuntu-latest) (push) Failing after 25s
CI / All (macos-latest) (push) Has been cancelled
CI / All (windows-latest) (push) Has been cancelled
2026-08-10 15:26:16 -06:00
28 changed files with 2483 additions and 1047 deletions
+6 -1
View File
@@ -37,10 +37,15 @@ jobs:
- uses: Swatinem/rust-cache@v2
- name: Cache DuckDB Extensions
id: duckdb-extensions
uses: actions/cache@v4
with:
path: ~/.duckdb/extensions
key: duckdb-ext-${{ matrix.os }}-v1.5.5
key: duckdb-ext-${{ matrix.os }}-${{ hashFiles('Cargo.lock') }}
- name: Install DuckDB Extensions
if: steps.duckdb-extensions.outputs.cache-hit != 'true'
run: cargo test --all duckdb
- name: Test
run: cargo test --all
Generated
+144 -120
View File
@@ -58,9 +58,9 @@ checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
[[package]]
name = "android_system_properties"
version = "0.1.5"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311"
checksum = "ae221649c9976a6f6c56ae1facf410f3ddb33cc661c4b7b61020a912d4237fbc"
dependencies = [
"libc",
]
@@ -209,7 +209,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "17060e608fbc0809d62a996a65cdee9e7c441a979f40f2d1d2fbdce9eef60dad"
dependencies = [
"anyhow",
"base64",
"base64 0.22.1",
"convert_case 0.11.0",
"dirs",
"either",
@@ -321,7 +321,7 @@ dependencies = [
"arrow-schema",
"arrow-select",
"atoi",
"base64",
"base64 0.22.1",
"chrono",
"comfy-table",
"half",
@@ -445,9 +445,9 @@ dependencies = [
[[package]]
name = "async-trait"
version = "0.1.91"
version = "0.1.92"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae36dc4177970ef04fde5178d3e2429882def40e57a451f919c098f72baa6cec"
checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
dependencies = [
"proc-macro2",
"quote",
@@ -520,11 +520,11 @@ dependencies = [
[[package]]
name = "aws-lc-rs"
version = "1.17.3"
version = "1.18.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "00bdb5da18dac48ca2cc7cd4a98e533e8635a58e2361d13a1a4ee3888e0d72f1"
checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e"
dependencies = [
"aws-lc-sys 0.43.0",
"aws-lc-sys 0.44.0",
"zeroize",
]
@@ -543,9 +543,9 @@ dependencies = [
[[package]]
name = "aws-lc-sys"
version = "0.43.0"
version = "0.44.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "43103168cc76fe62678a375e722fc9cb3a0146159ac5828bc4f0dfd755c2224c"
checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483"
dependencies = [
"cc",
"cmake",
@@ -1052,6 +1052,12 @@ version = "0.22.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
[[package]]
name = "base64"
version = "0.23.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
[[package]]
name = "base64-simd"
version = "0.8.0"
@@ -1248,7 +1254,7 @@ dependencies = [
"cached_proc_macro_types",
"hashbrown 0.15.5",
"once_cell",
"thiserror 2.0.19",
"thiserror 2.0.20",
"web-time",
]
@@ -1278,9 +1284,9 @@ checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
[[package]]
name = "cc"
version = "1.4.0"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9"
checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -1382,9 +1388,9 @@ dependencies = [
[[package]]
name = "clap"
version = "4.6.5"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "301b56658598e48f3648647ac6fc887be7e7108eddfa4e9b63fcf3ec58c0cadf"
checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca"
dependencies = [
"clap_builder",
"clap_derive",
@@ -1392,9 +1398,9 @@ dependencies = [
[[package]]
name = "clap_builder"
version = "4.6.5"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94a65403d1a1bd28f7dc68eb8506e8874808ee5eecb59298de588e2e1407a078"
checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889"
dependencies = [
"anstream",
"anstyle",
@@ -1405,9 +1411,9 @@ dependencies = [
[[package]]
name = "clap_complete"
version = "4.6.8"
version = "4.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1f84a88507dbd05c695f2cb5e8558e747179134005e9893882dec964190ed89"
checksum = "3be2ad0423bdbbb0e25bc89add796f3559706d4a95e1bc98e4d9662a957b6a19"
dependencies = [
"clap",
"clap_lex",
@@ -1417,9 +1423,9 @@ dependencies = [
[[package]]
name = "clap_complete_nushell"
version = "4.6.1"
version = "4.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "933b05d5d83ff65fd7eaf5d106c792f2264908790a2642aca57429767b762ce2"
checksum = "ffb66bc82eb9c92b1727310ae2c5868df22ae7cf46185bc5c544a4fa71955e49"
dependencies = [
"clap",
"clap_complete",
@@ -1532,7 +1538,7 @@ dependencies = [
"lazy_static",
"serde",
"serde_yaml",
"thiserror 2.0.19",
"thiserror 2.0.20",
]
[[package]]
@@ -1612,9 +1618,9 @@ dependencies = [
[[package]]
name = "cookie"
version = "0.18.1"
version = "0.18.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ddef33a339a91ea89fb53151bd0a4689cfce27055c291dfa69945475d22c747"
checksum = "1a373e3602691c3cdea496d2f0ee5935151e6168fe87739483c463db1b2f2f87"
dependencies = [
"percent-encoding",
"time",
@@ -1667,7 +1673,7 @@ dependencies = [
"async-trait",
"aws-smithy-eventstream",
"aws-smithy-types",
"base64",
"base64 0.22.1",
"bincode 2.0.1",
"bitflags 2.13.1",
"bm25",
@@ -2033,7 +2039,7 @@ version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e"
dependencies = [
"thiserror 2.0.19",
"thiserror 2.0.20",
]
[[package]]
@@ -2487,9 +2493,9 @@ dependencies = [
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de"
[[package]]
name = "fixedbitset"
@@ -2562,9 +2568,9 @@ dependencies = [
[[package]]
name = "futures"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218"
checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3"
dependencies = [
"futures-channel",
"futures-core",
@@ -2577,9 +2583,9 @@ dependencies = [
[[package]]
name = "futures-channel"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae"
checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4"
dependencies = [
"futures-core",
"futures-sink",
@@ -2587,15 +2593,15 @@ dependencies = [
[[package]]
name = "futures-core"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7"
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
[[package]]
name = "futures-executor"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458"
checksum = "031b47cf1a3c6cc8bc2fc76cd437f521619387907d469316e7c0bc278f1f5432"
dependencies = [
"futures-core",
"futures-task",
@@ -2604,38 +2610,38 @@ dependencies = [
[[package]]
name = "futures-io"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a"
checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed"
[[package]]
name = "futures-macro"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b"
checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
name = "futures-sink"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307"
checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d"
[[package]]
name = "futures-task"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109"
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
[[package]]
name = "futures-util"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa"
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
dependencies = [
"futures-channel",
"futures-core",
@@ -2781,7 +2787,7 @@ dependencies = [
"azure_identity",
"azure_security_keyvault_secrets",
"backtrace",
"base64",
"base64 0.22.1",
"chacha20poly1305",
"chrono",
"clap",
@@ -2807,7 +2813,7 @@ dependencies = [
"serde_with",
"serde_yaml",
"tempfile",
"thiserror 2.0.19",
"thiserror 2.0.20",
"tokio",
"validator",
"which",
@@ -3214,7 +3220,7 @@ version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
dependencies = [
"base64",
"base64 0.22.1",
"bytes",
"futures-channel",
"futures-util",
@@ -3528,10 +3534,12 @@ dependencies = [
"defmt",
"jiff-core",
"jiff-static",
"jiff-tzdb-platform",
"log",
"portable-atomic",
"portable-atomic-util",
"serde_core",
"windows-link",
]
[[package]]
@@ -3555,6 +3563,21 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "jiff-tzdb"
version = "0.1.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e"
[[package]]
name = "jiff-tzdb-platform"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8"
dependencies = [
"jiff-tzdb",
]
[[package]]
name = "jni"
version = "0.22.4"
@@ -3567,7 +3590,7 @@ dependencies = [
"jni-sys",
"log",
"simd_cesu8",
"thiserror 2.0.19",
"thiserror 2.0.20",
"walkdir",
"windows-link",
]
@@ -3616,9 +3639,9 @@ dependencies = [
[[package]]
name = "js-sys"
version = "0.3.103"
version = "0.3.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102"
checksum = "0e0c1080212aad755ea003d18543e8768dd432c48819efd73a7bf1e39b7a5a3a"
dependencies = [
"cfg-if",
"futures-util",
@@ -3634,7 +3657,7 @@ dependencies = [
"jsonptr",
"serde",
"serde_json",
"thiserror 2.0.19",
"thiserror 2.0.20",
]
[[package]]
@@ -3653,7 +3676,7 @@ version = "10.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eba32bfb4ffdeaca3e34431072faf01745c9b26d25504aa7a6cf5684334fc4fc"
dependencies = [
"base64",
"base64 0.22.1",
"getrandom 0.2.17",
"js-sys",
"pem",
@@ -3845,7 +3868,7 @@ dependencies = [
"serde-value",
"serde_json",
"serde_yaml",
"thiserror 2.0.19",
"thiserror 2.0.20",
"thread-id",
"typemap-ors",
"unicode-segmentation",
@@ -3992,7 +4015,7 @@ dependencies = [
"mach2",
"nix 0.30.1",
"sysctl",
"thiserror 2.0.19",
"thiserror 2.0.20",
"widestring",
"windows 0.48.0",
]
@@ -4389,9 +4412,9 @@ checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
[[package]]
name = "open"
version = "5.4.0"
version = "5.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0b3d059e795d52b8a72fef45658620edd4d9c359b338564aa14391ffa511ed5"
checksum = "f9cfef937e9c486488c7e3d949ae31c0f1d06bdacd75b99c086cb35356e30408"
dependencies = [
"is-wsl",
"libc",
@@ -4566,7 +4589,7 @@ version = "3.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be"
dependencies = [
"base64",
"base64 0.22.1",
"serde_core",
]
@@ -4705,7 +4728,7 @@ version = "1.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da1d65da6dd5d1e44199ac0f58712d241c0f439f80adea8924d832384087f85"
dependencies = [
"base64",
"base64 0.22.1",
"indexmap 2.14.0",
"quick-xml 0.41.0",
"serde",
@@ -4725,9 +4748,9 @@ dependencies = [
[[package]]
name = "portable-atomic"
version = "1.14.0"
version = "1.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3"
checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
[[package]]
name = "portable-atomic-util"
@@ -4912,7 +4935,7 @@ dependencies = [
"rustc-hash",
"rustls 0.23.43",
"socket2 0.6.5",
"thiserror 2.0.19",
"thiserror 2.0.20",
"tokio",
"tracing",
"web-time",
@@ -4935,7 +4958,7 @@ dependencies = [
"rustls 0.23.43",
"rustls-pki-types",
"slab",
"thiserror 2.0.19",
"thiserror 2.0.20",
"tinyvec",
"tracing",
"web-time",
@@ -5086,7 +5109,7 @@ checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
dependencies = [
"getrandom 0.2.17",
"libredox",
"thiserror 2.0.19",
"thiserror 2.0.20",
]
[[package]]
@@ -5103,7 +5126,7 @@ dependencies = [
"serde",
"strip-ansi-escapes",
"strum",
"thiserror 2.0.19",
"thiserror 2.0.20",
"unicase",
"unicode-segmentation",
"unicode-width",
@@ -5143,9 +5166,9 @@ dependencies = [
[[package]]
name = "regex-automata"
version = "0.4.16"
version = "0.4.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad"
checksum = "ad8553b9b26413251cbf30e620595c7a41b3887f03da04579c0e6b0d6a06b4b2"
dependencies = [
"aho-corasick",
"memchr",
@@ -5170,7 +5193,7 @@ version = "0.12.28"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
dependencies = [
"base64",
"base64 0.22.1",
"bytes",
"futures-core",
"futures-util",
@@ -5214,7 +5237,7 @@ version = "0.13.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "219c5811de6525e5416c7d5d53bb656d3afdbc6c5af816e0802bcfa42dbdc1c3"
dependencies = [
"base64",
"base64 0.22.1",
"bytes",
"futures-channel",
"futures-core",
@@ -5285,7 +5308,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1d1f571c72940a19d9532fe52dbea8bc9912bf1d766c2970bb824056b86f3f59"
dependencies = [
"async-trait",
"base64",
"base64 0.22.1",
"chrono",
"futures",
"http 1.5.0",
@@ -5298,7 +5321,7 @@ dependencies = [
"serde",
"serde_json",
"sse-stream",
"thiserror 2.0.19",
"thiserror 2.0.20",
"tokio",
"tokio-stream",
"tokio-util",
@@ -5460,7 +5483,7 @@ dependencies = [
"once_cell",
"ring",
"rustls-pki-types",
"rustls-webpki 0.103.13",
"rustls-webpki 0.103.14",
"subtle",
"zeroize",
]
@@ -5501,7 +5524,7 @@ dependencies = [
"rustls 0.23.43",
"rustls-native-certs",
"rustls-platform-verifier-android",
"rustls-webpki 0.103.13",
"rustls-webpki 0.103.14",
"security-framework",
"security-framework-sys",
"webpki-root-certs",
@@ -5526,9 +5549,9 @@ dependencies = [
[[package]]
name = "rustls-webpki"
version = "0.103.13"
version = "0.103.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e"
checksum = "0527518605e68109d875e248ea259b6758801cf165e4b2c2733ae3b51f12535a"
dependencies = [
"aws-lc-rs",
"ring",
@@ -5828,16 +5851,17 @@ dependencies = [
[[package]]
name = "serde_with"
version = "3.21.0"
version = "3.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "76a5c54c7310e7b8b9577c286d7e399ddd876c3e12b3ed917a8aabc4b96e9e8c"
checksum = "ee78f1fbe43ac4a0e47aadb3dbd357b69eb0d3793e948624cd03dd2750ab1c0a"
dependencies = [
"base64",
"base64 0.22.1",
"bs58",
"chrono",
"hex",
"indexmap 1.9.3",
"indexmap 2.14.0",
"jiff",
"schemars 0.9.0",
"schemars 1.2.2",
"serde_core",
@@ -5848,9 +5872,9 @@ dependencies = [
[[package]]
name = "serde_with_macros"
version = "3.21.0"
version = "3.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "84d57bc0c8b9a17920c178daa6bb924850d54a9c97ab45194bb8c17ad66bb660"
checksum = "8705578779c2b6bd90d84d66eb2e206b708b1a4d7b9f17641b293545bf1c7e46"
dependencies = [
"darling 0.23.0",
"proc-macro2",
@@ -6055,7 +6079,7 @@ checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d"
dependencies = [
"num-bigint",
"num-traits",
"thiserror 2.0.19",
"thiserror 2.0.20",
"time",
]
@@ -6283,7 +6307,7 @@ dependencies = [
"serde",
"serde_derive",
"serde_json",
"thiserror 2.0.19",
"thiserror 2.0.20",
"walkdir",
]
@@ -6417,11 +6441,11 @@ dependencies = [
[[package]]
name = "thiserror"
version = "2.0.19"
version = "2.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9"
checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f"
dependencies = [
"thiserror-impl 2.0.19",
"thiserror-impl 2.0.20",
]
[[package]]
@@ -6437,9 +6461,9 @@ dependencies = [
[[package]]
name = "thiserror-impl"
version = "2.0.19"
version = "2.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd"
checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
dependencies = [
"proc-macro2",
"quote",
@@ -6658,7 +6682,7 @@ checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef"
dependencies = [
"async-trait",
"axum",
"base64",
"base64 0.22.1",
"bytes",
"h2 0.4.15",
"http 1.5.0",
@@ -6779,9 +6803,9 @@ dependencies = [
[[package]]
name = "tree-sitter"
version = "0.26.11"
version = "0.26.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af1c71c1c4cc0920b20d6b0f6572e7682cd07a6a2faec71067a31fa394c586df"
checksum = "83c567a8e18ae93f20982c90370b16fd24023aeaf52f6052b96957ab253a0fec"
dependencies = [
"cc",
"regex",
@@ -6855,7 +6879,7 @@ version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4dd1eb4a538c1ab3d5c05437129bc16891296146b23c9b0bb3f5df99f5b3a18d"
dependencies = [
"base64",
"base64 0.22.1",
"bytes",
"futures",
"serde",
@@ -6870,7 +6894,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e632235c99ae896a3c451d1ead00cea11a2219aeda1b35a74027fe99ea3f3b72"
dependencies = [
"async-trait",
"base64",
"base64 0.22.1",
"dyn-clone",
"futures",
"getrandom 0.3.4",
@@ -6981,11 +7005,11 @@ checksum = "6d49784317cd0d1ee7ec5c716dd598ec5b4483ea832a2dced265471cc0f690ae"
[[package]]
name = "ureq"
version = "3.3.0"
version = "3.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dea7109cdcd5864d4eeb1b58a1648dc9bf520360d7af16ec26d0a9354bafcfc0"
checksum = "972d7902c8735f2695410b8aed7df6ed12a47394aa1c8d7af49f0497b731a94d"
dependencies = [
"base64",
"base64 0.23.1",
"cookie_store",
"encoding_rs",
"flate2",
@@ -7003,11 +7027,11 @@ dependencies = [
[[package]]
name = "ureq-proto"
version = "0.6.0"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e994ba84b0bd1b1b0cf92878b7ef898a5c1760108fe7b6010327e274917a808c"
checksum = "da5f78b09e6941e1a0f2e30e695e4b120377b54d5e0aec11b594bb57b3971613"
dependencies = [
"base64",
"base64 0.23.1",
"http 1.5.0",
"httparse",
"log",
@@ -7159,9 +7183,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen"
version = "0.2.126"
version = "0.2.127"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4"
checksum = "1b70935747edd64d89de3efa29d73789b806c15798f8e7dca4d8ac356b50ce70"
dependencies = [
"cfg-if",
"once_cell",
@@ -7172,9 +7196,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.76"
version = "0.4.77"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c62df1340f32221cb9c54d6a27b030e3dba64361d4a95bed55f9aacb44da291d"
checksum = "6b7777d5cc23d0e91404e53ce2d5e8ec7acae3026b16233dba62cd3246457950"
dependencies = [
"js-sys",
"wasm-bindgen",
@@ -7182,9 +7206,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.126"
version = "0.2.127"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1"
checksum = "77775f8f3f7217702089053b94958f8f54061a3f663417df76e19cbdcca29bc1"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
@@ -7192,9 +7216,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.126"
version = "0.2.127"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e"
checksum = "e11d33f857dc2fb11b8bc75aee111aa9cbeb12cd9f25efd3d4c2a3dd4e235284"
dependencies = [
"bumpalo",
"proc-macro2",
@@ -7205,9 +7229,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.126"
version = "0.2.127"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24"
checksum = "7ef64dbcc55df09c7e5a46182d181c2cfa3e925f3da937ea764728b4bbb9dcbf"
dependencies = [
"unicode-ident",
]
@@ -7310,9 +7334,9 @@ dependencies = [
[[package]]
name = "web-sys"
version = "0.3.103"
version = "0.3.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8622dcb61c0bcc9fffa6938bed81210af2da9a7e4a1a834b2e37a59b6dfb6141"
checksum = "c435338968042f4f59a557f690a253676d47ce13ceb55d70100e7facf6620a30"
dependencies = [
"js-sys",
"wasm-bindgen",
@@ -7749,7 +7773,7 @@ dependencies = [
"log",
"os_pipe",
"rustix 1.1.4",
"thiserror 2.0.19",
"thiserror 2.0.20",
"tree_magic_mini",
"wayland-backend",
"wayland-client",
@@ -7844,18 +7868,18 @@ dependencies = [
[[package]]
name = "zerocopy"
version = "0.8.55"
version = "0.8.56"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb"
checksum = "556764e583adb45a9f8d413c2a147fa7e8d821e48e12b14fd560b607998b75eb"
dependencies = [
"zerocopy-derive",
]
[[package]]
name = "zerocopy-derive"
version = "0.8.55"
version = "0.8.56"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb"
checksum = "f2ab42fc20575779bd240faa45f94a74256f755c0fa9e89f0ede20d91d0cdfc1"
dependencies = [
"proc-macro2",
"quote",
@@ -7957,9 +7981,9 @@ version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dba6063ff82cdbd9a765add16d369abe81e520f836054e997c2db217ceca40c0"
dependencies = [
"base64",
"base64 0.22.1",
"ed25519-dalek",
"thiserror 2.0.19",
"thiserror 2.0.20",
]
[[package]]
+97 -21
View File
@@ -40,15 +40,57 @@ _write_project_cache() {
_detect_heuristic() {
local dir="$1"
local runner="" runner_type="" runner_targets=""
if [[ -f "${dir}/Taskfile.yml" || -f "${dir}/Taskfile.yaml" || -f "${dir}/taskfile.yml" || -f "${dir}/taskfile.yaml" ]]; then
runner="task" runner_type="taskfile"
runner_targets=$( (cd "${dir}" && task --list-all 2>/dev/null | sed -n 's/^\* \([^:[:space:]]*\):.*/\1/p') || true)
elif [[ -f "${dir}/justfile" || -f "${dir}/Justfile" ]]; then
runner="just" runner_type="just"
runner_targets=$( (cd "${dir}" && just --summary 2>/dev/null | tr ' ' '\n') || true)
elif [[ -f "${dir}/Makefile" || -f "${dir}/makefile" || -f "${dir}/GNUmakefile" ]]; then
runner="make" runner_type="make"
local mk mkfiles=()
for mk in Makefile makefile GNUmakefile; do
[[ -f "${dir}/${mk}" ]] && mkfiles+=("${dir}/${mk}")
done
runner_targets=$(sed -n 's/^\([A-Za-z0-9_][A-Za-z0-9_.-]*\):\([^=].*\|\)$/\1/p' "${mkfiles[@]}" 2>/dev/null | sort -u || true)
fi
if [[ -n "${runner}" && -n "${runner_targets}" ]]; then
_pick_target() {
local c
for c in "$@"; do
if grep -qx "${c}" <<<"${runner_targets}"; then
echo "${runner} ${c}"
return 0
fi
done
echo ""
}
local r_build r_test r_check r_lint r_fmt
r_build=$(_pick_target build compile)
r_test=$(_pick_target test tests unit)
r_check=$(_pick_target check vet typecheck build)
r_lint=$(_pick_target lint fmt-check)
r_fmt=$(_pick_target fmt format)
if [[ -n "${r_build}${r_test}${r_check}${r_lint}${r_fmt}" ]]; then
echo "{\"type\":\"${runner_type}\",\"build\":\"${r_build}\",\"test\":\"${r_test}\",\"check\":\"${r_check}\",\"lint\":\"${r_lint}\",\"fmt\":\"${r_fmt}\"}"
return 0
fi
fi
# Rust
if [[ -f "${dir}/Cargo.toml" ]]; then
echo '{"type":"rust","build":"cargo build","test":"cargo test","check":"cargo check"}'
echo '{"type":"rust","build":"cargo build","test":"cargo test","check":"cargo check","lint":"cargo clippy --no-deps -- -D warnings","fmt":"cargo fmt"}'
return 0
fi
# Go
if [[ -f "${dir}/go.mod" ]]; then
echo '{"type":"go","build":"go build ./...","test":"go test ./...","check":"go vet ./..."}'
local go_lint=""
if compgen -G "${dir}/.golangci.*" &>/dev/null && command -v golangci-lint &>/dev/null; then
go_lint="golangci-lint run"
fi
echo "{\"type\":\"go\",\"build\":\"go build ./...\",\"test\":\"go test ./...\",\"check\":\"go vet ./...\",\"lint\":\"${go_lint}\",\"fmt\":\"gofmt -w .\"}"
return 0
fi
@@ -65,7 +107,25 @@ _detect_heuristic() {
[[ -f "${dir}/pnpm-lock.yaml" ]] && pm="pnpm"
[[ -f "${dir}/yarn.lock" ]] && pm="yarn"
echo "{\"type\":\"nodejs\",\"build\":\"${pm} run build\",\"test\":\"${pm} test\",\"check\":\"${pm} run lint\"}"
# Emit only scripts the manifest actually declares (same introspection
# contract as the runner tier: never guess a target into existence).
_pkg_script() {
local s
for s in "$@"; do
if jq -e --arg s "$s" '.scripts[$s] // empty' "${dir}/package.json" &>/dev/null; then
echo "${pm} run ${s}"
return 0
fi
done
echo ""
}
local p_build p_test p_check p_lint p_fmt
p_build=$(_pkg_script build compile)
p_test=$(_pkg_script test)
p_check=$(_pkg_script check typecheck tsc)
p_lint=$(_pkg_script lint)
p_fmt=$(_pkg_script fmt format prettier)
echo "{\"type\":\"nodejs\",\"build\":\"${p_build}\",\"test\":\"${p_test}\",\"check\":\"${p_check}\",\"lint\":\"${p_lint}\",\"fmt\":\"${p_fmt}\"}"
return 0
fi
@@ -82,7 +142,7 @@ _detect_heuristic() {
check_cmd="uv run ruff check ."
fi
echo "{\"type\":\"python\",\"build\":\"\",\"test\":\"${test_cmd}\",\"check\":\"${check_cmd}\"}"
echo "{\"type\":\"python\",\"build\":\"\",\"test\":\"${test_cmd}\",\"check\":\"${check_cmd}\",\"lint\":\"${check_cmd}\",\"fmt\":\"ruff format .\"}"
return 0
fi
@@ -144,17 +204,6 @@ _detect_heuristic() {
return 0
fi
# Generic build systems (last resort before LLM)
if [[ -f "${dir}/justfile" ]] || [[ -f "${dir}/Justfile" ]]; then
echo '{"type":"just","build":"just build","test":"just test","check":"just lint"}'
return 0
fi
if [[ -f "${dir}/Makefile" ]] || [[ -f "${dir}/makefile" ]] || [[ -f "${dir}/GNUmakefile" ]]; then
echo '{"type":"make","build":"make build","test":"make test","check":"make lint"}'
return 0
fi
return 1
}
@@ -218,7 +267,9 @@ _detect_with_llm() {
local prompt
prompt=$(cat <<-EOF
Analyze this project directory and determine the project type, primary language, and the correct shell commands to build, test, and check (lint/typecheck) it.
Analyze this project directory and determine the project type, primary language, and the correct shell commands to build, test, check (typecheck/vet), lint, and format it.
PRIORITY RULE: if the project declares its own task-runner interface (a Taskfile, justfile, Makefile, package.json scripts, or similar), those declared targets ARE the correct commands — prefer them over generic ecosystem defaults, and never invent a target the interface does not declare.
EOF
)
@@ -226,12 +277,12 @@ _detect_with_llm() {
prompt+=$(cat <<-EOF
Respond with ONLY a valid JSON object. No markdown fences, no explanation, no extra text.
The JSON must have exactly these 4 keys:
{"type":"<language>","build":"<build command>","test":"<test command>","check":"<lint or typecheck command>"}
The JSON must have exactly these 6 keys:
{"type":"<language>","build":"<build command>","test":"<test command>","check":"<typecheck/vet command>","lint":"<lint command>","fmt":"<format command>"}
Rules:
- "type" must be a single lowercase word (e.g. rust, go, python, nodejs, java, ruby, elixir, cpp, c, zig, haskell, scala, kotlin, dart, swift, php, dotnet, etc.)
- If a command doesn't apply to this project, use an empty string, ""
- If a command doesn't apply to this project, use an empty string, "" — NEVER guess a command that might not exist; a wrongly-guessed command is worse than an empty one
- Use the most standard/common commands for the detected ecosystem
- If you detect a package manager lockfile, use that package manager (e.g. pnpm over npm)
EOF
@@ -244,7 +295,7 @@ _detect_with_llm() {
llm_response=$(echo "${llm_response}" | grep -o '{[^}]*}' | head -1)
if echo "${llm_response}" | jq -e '.type and .build != null and .test != null and .check != null' &>/dev/null; then
echo "${llm_response}" | jq -c '{type: (.type // "unknown"), build: (.build // ""), test: (.test // ""), check: (.check // "")}'
echo "${llm_response}" | jq -c '{type: (.type // "unknown"), build: (.build // ""), test: (.test // ""), check: (.check // ""), lint: (.lint // ""), fmt: (.fmt // "")}'
return 0
fi
@@ -258,7 +309,7 @@ detect_project() {
local cached
if cached=$(_read_project_cache "${dir}"); then
echo "${cached}" | jq -c '{type, build, test, check}'
echo "${cached}" | jq -c '{type, build, test, check, lint: (.lint // ""), fmt: (.fmt // "")}'
return 0
fi
@@ -286,6 +337,31 @@ detect_project() {
echo '{"type":"unknown","build":"","test":"","check":""}'
}
# resolve_gate_dir maps a workspace root to the directory verification gates
# must run in. A delivery-repo worker's workspace root holds only dotfiles
# plus the clone, so gates aimed at the root detect nothing and silently
# no-op. When the root has no project markers and exactly ONE first-level
# git repo exists, gates run inside it; anything ambiguous stays at the root.
resolve_gate_dir() {
local dir="${1:-.}"
local m
for m in Taskfile.yml Taskfile.yaml taskfile.yml Cargo.toml go.mod package.json pyproject.toml setup.py pom.xml build.gradle mix.exs Gemfile composer.json Makefile justfile Justfile CMakeLists.txt; do
if [[ -e "${dir}/${m}" ]]; then
echo "${dir}"
return 0
fi
done
local repos=() d
for d in "${dir}"/*/; do
[[ -d "${d}/.git" ]] && repos+=("${d}")
done
if [[ ${#repos[@]} -eq 1 ]]; then
echo "${repos[0]%/}"
return 0
fi
echo "${dir}"
}
###########################
## FILE SEARCH UTILITIES ##
###########################
+6 -1
View File
@@ -227,6 +227,11 @@ nodes:
on unfamiliar lints, etc.).
4. No dead code, no commented-out blocks, no premature abstractions.
5. End your turn when editing is done. The graph runs verification next.
6. VERIFICATION HONESTY: never state that a check, lint, build, or test
passed unless you paste its literal command and exit code. A gate
that did not run is UNVERIFIED — say so. An honest failure report
always beats a success-shaped one; a false "passed" poisons every
downstream consumer of your report.
Project directory: {{project_dir}}
prompt: |
@@ -248,7 +253,7 @@ nodes:
- fs_write
- fs_patch
- execute_command
max_iterations: 30
max_iterations: 100
state_updates:
last_node_output: '{{output}}'
fallback: end_failure
+2 -1
View File
@@ -13,6 +13,7 @@ else
fi
project_dir=$(echo "$state" | jq -r '.project_dir // "."')
project_dir=$(resolve_gate_dir "$project_dir")
if [[ -n "${BUILD_CMD:-}" ]]; then
cmd="$BUILD_CMD"
@@ -24,7 +25,7 @@ fi
if [[ -z "$cmd" || "$cmd" == "null" ]]; then
jq -nc '{
"build_ok": true,
"build_output": "(no build/check command available for this project type)",
"build_output": "(GATE NOT RUN: no build/check command configured or detected. This is NOT evidence that the build passed — set BUILD_CMD, and never report the build as verified.)",
"_next": "verify_tests"
}'
exit 0
+2 -1
View File
@@ -13,6 +13,7 @@ else
fi
project_dir=$(echo "$state" | jq -r '.project_dir // "."')
project_dir=$(resolve_gate_dir "$project_dir")
if [[ -n "${TEST_CMD:-}" ]]; then
cmd="$TEST_CMD"
@@ -24,7 +25,7 @@ fi
if [[ -z "$cmd" || "$cmd" == "null" ]]; then
jq -nc '{
"tests_ok": true,
"tests_output": "(no test command available for this project type)",
"tests_output": "(GATE NOT RUN: no test command configured or detected. This is NOT evidence that tests passed — set TEST_CMD, and never report the suite as green.)",
"_next": "self_review"
}'
exit 0
+6
View File
@@ -266,6 +266,12 @@ instructions: |
**No evidence = not complete.** Mark a todo `completed` only after evidence is collected.
### Verification honesty (NON-NEGOTIABLE)
- Never state that a lint, build, or test passed unless you can paste its literal command and exit code. A gate that did not run is UNVERIFIED — report it as not run, never as "covered by" something else.
- Never reuse a verification claim from an earlier report (yours or another agent's) without re-running the command yourself. Prior reports are unverified context, not evidence.
- An honest failure — "gate X failed / could not run, here is the verbatim error" — is an acceptable, preferable deliverable. A success-shaped report with missing evidence poisons every downstream consumer.
### Independent code review (post-coder, non-trivial work)
After completing delegated `coder` work, spawn `code-reviewer` for an independent review pass if ANY of these are true:
+34 -7
View File
@@ -1,11 +1,38 @@
schemaVersion: '1'
schemaVersion: '2'
kind: mixin
name: sisyphus-ddg
description: >
Allows Sisyphus to hit all domains since it utilizes the DuckDuckGo
MCP server. This allows the MCP server to actually perform web searches
on arbitrary domains and retrieve info for the agent.
Allows Sisyphus to reach DuckDuckGo plus a curated set of common
content domains for its web-search MCP server. Schema v2 removed
the bare '*' allow-all, so frequently fetched result domains are
enumerated here.
network:
allowedDomains:
- '*'
agentInstructions:
content: |
Web search runs against an enumerated network allow list. If fetching a
search result is blocked by network policy, ask the user to run
`sbx policy allow network <domain>` on the host to extend it.
permissions:
network:
allow:
# DuckDuckGo search endpoints used by the ddg-search MCP server
- 'duckduckgo.com'
- 'html.duckduckgo.com'
- 'lite.duckduckgo.com'
# Common content/result domains fetched from search results
# ('*.host' matches exactly one label and not the bare host itself)
- '*.wikipedia.org'
- 'github.com'
- '*.githubusercontent.com'
- 'stackoverflow.com'
- '*.stackexchange.com'
- 'developer.mozilla.org'
- 'docs.python.org'
- 'doc.rust-lang.org'
- 'docs.rs'
- 'crates.io'
- 'pypi.org'
- 'www.npmjs.com'
# Jina reader fallback for fetching arbitrary pages as markdown
- 'r.jina.ai'
+6
View File
@@ -439,6 +439,12 @@ nodes:
staleness report, gate decisions, and fix loop history. Downstream
plan updates come from the sweep results.
VERIFICATION HONESTY: evidence marked "GATE NOT RUN" means that gate
is UNVERIFIED — record it as not run; never paraphrase a skipped gate
as covered, passing, or handled elsewhere. A handoff that admits an
unverified gate is correct; one that dresses it up as verified poisons
every downstream reader.
Then append durable, step-independent facts (if any) to {{notes_path}}
- create the file if missing, never rewrite existing entries.
@@ -13,6 +13,7 @@ else
fi
project_dir=$(echo "$state" | jq -r '.project_dir // "."')
project_dir=$(resolve_gate_dir "$project_dir")
if [[ -n "${BUILD_CMD:-}" ]]; then
cmd="$BUILD_CMD"
@@ -24,7 +25,7 @@ fi
if [[ -z "$cmd" || "$cmd" == "null" ]]; then
jq -nc '{
"build_ok": true,
"build_output": "(no build/check command available for this project type)",
"build_output": "(GATE NOT RUN: no build/check command configured or detected. This is NOT evidence that the build passed — set BUILD_CMD, and never report the build as verified.)",
"_next": "verify_tests"
}'
exit 0
@@ -13,19 +13,18 @@ else
fi
project_dir=$(echo "$state" | jq -r '.project_dir // "."')
project_type=$(detect_project "$project_dir" | jq -r '.type // "unknown"')
project_dir=$(resolve_gate_dir "$project_dir")
project_info=$(detect_project "$project_dir")
project_type=$(echo "$project_info" | jq -r '.type // "unknown"')
format_cmd="${FORMAT_CMD:-}"
if [[ -z "$format_cmd" ]]; then
case "$project_type" in
rust) format_cmd="cargo fmt" ;;
go) format_cmd="gofmt -w ." ;;
python) command -v ruff &>/dev/null && format_cmd="ruff format ." ;;
esac
format_cmd=$(echo "$project_info" | jq -r '.fmt // ""')
fi
if [[ "$format_cmd" == "null" ]]; then format_cmd=""; fi
if [[ -z "$format_cmd" ]]; then
format_output="(no format command configured for project type '$project_type'; skipped. Set FORMAT_CMD to enable.)"
format_output="(GATE NOT RUN: no format command configured or detected for project type '$project_type'. This is NOT evidence that formatting is clean. Set FORMAT_CMD to enable.)"
else
fmt_rc=0
fmt_out=$(cd "$project_dir" && eval "$format_cmd" 2>&1) || fmt_rc=$?
@@ -37,12 +36,18 @@ fi
lint_cmd="${LINT_CMD:-}"
if [[ -z "$lint_cmd" ]]; then
lint_cmd=$(echo "$project_info" | jq -r '.lint // ""')
fi
# The skip message must read as a WARNING, never a reassurance: the previous
# wording ("linting is covered by the build/check command") was quoted
# verbatim by workers as false evidence that linting passed
if [[ -z "$lint_cmd" || "$lint_cmd" == "null" ]]; then
jq -nc \
--arg fo "$format_output" \
'{
"format_output": $fo,
"lint_ok": true,
"lint_output": "(no LINT_CMD configured; linting is covered by the build/check command)",
"lint_output": "(GATE NOT RUN: no lint command configured or detected. This is NOT evidence that linting passed — set LINT_CMD or add a Taskfile lint target, and never report linting as covered.)",
"_next": "verify_build"
}'
exit 0
@@ -13,6 +13,7 @@ else
fi
project_dir=$(echo "$state" | jq -r '.project_dir // "."')
project_dir=$(resolve_gate_dir "$project_dir")
if [[ -n "${TEST_CMD:-}" ]]; then
cmd="$TEST_CMD"
@@ -24,7 +25,7 @@ fi
if [[ -z "$cmd" || "$cmd" == "null" ]]; then
jq -nc '{
"tests_ok": true,
"tests_output": "(no test command available for this project type)",
"tests_output": "(GATE NOT RUN: no test command configured or detected. This is NOT evidence that tests passed — set TEST_CMD, and never report the suite as green.)",
"_next": "edge_case_sweep"
}'
exit 0
+37 -37
View File
@@ -1,4 +1,4 @@
schemaVersion: "1"
schemaVersion: '2'
kind: mixin
name: built-in-tools
description: >
@@ -6,39 +6,39 @@ description: >
global tools and the default MCP server set. Auto-applied by Coyote's sbx
mixin discovery when running `coyote --sandbox`.
network:
allowedDomains:
# fetch_url_via_jina + jina reader fallback
- "r.jina.ai:443"
# get_current_weather (.sh, .py, .ts)
- "wttr.in:443"
# search_arxiv (the .sh tool still uses http://, so :80 is required until fixed)
- "export.arxiv.org:443"
- "export.arxiv.org:80"
# search_arxiv + search_wikipedia may follow DOI redirects
- "doi.org:443"
# search_wikipedia
- "en.wikipedia.org:443"
# search_wolframalpha
- "api.wolframalpha.com:443"
# web_search_perplexity
- "api.perplexity.ai:443"
# web_search_tavily
- "api.tavily.com:443"
# send_twilio
- "api.twilio.com:443"
# MCP: github (built-in mcp.json: api.githubcopilot.com)
- "api.githubcopilot.com:443"
# MCP: atlassian (built-in mcp.json: mcp-remote -> mcp.atlassian.com)
- "mcp.atlassian.com:443"
# MCP: ddg-search (built-in mcp.json: uvx duckduckgo-mcp-server)
- "duckduckgo.com:443"
- "html.duckduckgo.com:443"
- "lite.duckduckgo.com:443"
# MCP: npx-based servers (mcp-remote) pull from npm
- "registry.npmjs.org:443"
# MCP: docker server may pull images from common registries
- "ghcr.io:443"
- "registry-1.docker.io:443"
- "auth.docker.io:443"
- "production.cloudflare.docker.com:443"
permissions:
network:
allow:
# fetch_url_via_jina + jina reader fallback
- 'r.jina.ai'
# get_current_weather (.sh, .py, .ts)
- 'wttr.in'
# search_arxiv (the .sh tool still uses http://, so :80 is required until fixed)
- 'export.arxiv.org'
- 'export.arxiv.org:80'
# search_arxiv + search_wikipedia may follow DOI redirects
- 'doi.org'
# search_wikipedia
- 'en.wikipedia.org'
# search_wolframalpha
- 'api.wolframalpha.com'
# web_search_perplexity
- 'api.perplexity.ai'
# web_search_tavily
- 'api.tavily.com'
# send_twilio
- 'api.twilio.com'
# MCP: github (built-in mcp.json: api.githubcopilot.com)
- 'api.githubcopilot.com'
# MCP: atlassian (built-in mcp.json: mcp-remote -> mcp.atlassian.com)
- 'mcp.atlassian.com'
# MCP: ddg-search (built-in mcp.json: uvx duckduckgo-mcp-server)
- 'duckduckgo.com'
- 'html.duckduckgo.com'
- 'lite.duckduckgo.com'
# MCP: npx-based servers (mcp-remote) pull from npm
- 'registry.npmjs.org'
# MCP: docker server may pull images from common registries
- 'ghcr.io'
- 'registry-1.docker.io'
- 'auth.docker.io'
+287 -242
View File
@@ -4,7 +4,7 @@
# sbx create --kit ./sbx-kit/ coyote --name testing .
# sbx cp $HOME/.config/coyote/ testing:/home/agent/.config/
# sbx run testing --kit ./sbx-kit/
schemaVersion: '1'
schemaVersion: '2'
kind: sandbox
name: coyote
displayName: Coyote
@@ -14,198 +14,255 @@ description: >
sandbox:
image: 'darkalex17/coyote:v0.8.3'
aiFilename: COYOTE.md
entrypoint:
run: ['bash', '-lc', 'exec /home/agent/.cargo/bin/coyote']
entrypoint: ['bash', '-lc', 'exec /home/agent/.cargo/bin/coyote']
network:
# Proxy-managed LLM providers: the proxy substitutes `proxy-managed` for
# the env var inside the sandbox and rewrites the auth header per
# serviceAuth at request time. Multiple domains may map to one service
# (e.g. jina) so they share a single credential.
serviceDomains:
api.openai.com: openai
api.anthropic.com: anthropic
generativelanguage.googleapis.com: gemini
api.cohere.ai: cohere
api.groq.com: groq
openrouter.ai: openrouter
api.ai21.com: ai21
api.cloudflare.com: cloudflare
api.deepinfra.com: deepinfra
api.deepseek.com: deepseek
api.mistral.ai: mistral
api.perplexity.ai: perplexity
api.voyageai.com: voyageai
api.x.ai: xai
api.jina.ai: jina
r.jina.ai: jina
qianfan.baidubce.com: ernie
api.hunyuan.cloud.tencent.com: hunyuan
api.minimax.chat: minimax
api.moonshot.cn: moonshot
dashscope.aliyuncs.com: qianwen
open.bigmodel.cn: zhipuai
serviceAuth:
openai:
headerName: Authorization
valueFormat: 'Bearer %s'
anthropic:
headerName: x-api-key
valueFormat: '%s'
gemini:
headerName: x-goog-api-key
valueFormat: '%s'
cohere:
headerName: Authorization
valueFormat: 'Bearer %s'
groq:
headerName: Authorization
valueFormat: 'Bearer %s'
openrouter:
headerName: Authorization
valueFormat: 'Bearer %s'
ai21:
headerName: Authorization
valueFormat: 'Bearer %s'
cloudflare:
headerName: Authorization
valueFormat: 'Bearer %s'
deepinfra:
headerName: Authorization
valueFormat: 'Bearer %s'
deepseek:
headerName: Authorization
valueFormat: 'Bearer %s'
mistral:
headerName: Authorization
valueFormat: 'Bearer %s'
perplexity:
headerName: Authorization
valueFormat: 'Bearer %s'
voyageai:
headerName: Authorization
valueFormat: 'Bearer %s'
xai:
headerName: Authorization
valueFormat: 'Bearer %s'
jina:
headerName: Authorization
valueFormat: 'Bearer %s'
ernie:
headerName: Authorization
valueFormat: 'Bearer %s'
hunyuan:
headerName: Authorization
valueFormat: 'Bearer %s'
minimax:
headerName: Authorization
valueFormat: 'Bearer %s'
moonshot:
headerName: Authorization
valueFormat: 'Bearer %s'
qianwen:
headerName: Authorization
valueFormat: 'Bearer %s'
zhipuai:
headerName: Authorization
valueFormat: 'Bearer %s'
allowedDomains:
# Coyote release + self-update + model-registry sync
- 'github.com:443'
- 'api.github.com:443'
- 'raw.githubusercontent.com:443'
- 'objects.githubusercontent.com:443'
- '*.githubusercontent.com:443'
# Package managers and developer tools (cargo, uv, pip — useful at runtime for user installs)
- 'crates.io:443'
- 'static.crates.io:443'
- 'pypi.org:443'
- 'files.pythonhosted.org:443'
- 'astral.sh:443'
- 'sh.rustup.rs:443'
- 'static.rust-lang.org:443'
permissions:
network:
allow:
# Coyote release + self-update + model-registry sync
- 'github.com'
- 'api.github.com'
- 'raw.githubusercontent.com'
- 'objects.githubusercontent.com'
- '*.githubusercontent.com'
# Package managers and developer tools (cargo, uv, pip — useful at runtime for user installs)
- 'crates.io'
- 'static.crates.io'
- 'pypi.org'
- 'files.pythonhosted.org'
- 'astral.sh'
- 'sh.rustup.rs'
- 'static.rust-lang.org'
# LLM model OAuth + API endpoints
- 'claude.ai:443'
- 'console.anthropic.com:443'
- 'accounts.google.com:443'
# *.googleapis.com covers oauth2 + userinfo + VertexAI regional endpoints
# (*-aiplatform.googleapis.com). Do not narrow without re-checking VertexAI.
- '*.googleapis.com:443'
# LLM model OAuth + API endpoints
- 'claude.ai'
- 'console.anthropic.com'
- 'accounts.google.com'
# *.googleapis.com covers oauth2 + userinfo + VertexAI regional endpoints
# (*-aiplatform.googleapis.com). Do not narrow without re-checking VertexAI.
- '*.googleapis.com'
# Bedrock and GitHub Models use signed / GitHub-PAT auth that the proxy
# cannot rewrite. Domains are allow-listed; credentials must be injected
# separately (see README "Extending").
- '*.amazonaws.com:443'
- 'models.inference.ai.azure.com:443'
# Bedrock and GitHub Models use signed / GitHub-PAT auth that the proxy
# cannot rewrite; credentials must be injected separately (see README
# "Extending"). NOTE: '*.amazonaws.com' matches exactly ONE label, so
# two-label regional Bedrock hosts must be enumerated explicitly
# ('**.' is declared but not yet enforced by sbx). Add your region
# via a mixin if it's missing below.
- '*.amazonaws.com'
- 'bedrock-runtime.us-east-1.amazonaws.com'
- 'bedrock-runtime.us-east-2.amazonaws.com'
- 'bedrock-runtime.us-west-2.amazonaws.com'
- 'bedrock-runtime.eu-west-1.amazonaws.com'
- 'bedrock-runtime.eu-central-1.amazonaws.com'
- 'bedrock-runtime.ap-southeast-2.amazonaws.com'
- 'bedrock-runtime.ap-northeast-1.amazonaws.com'
- 'models.inference.ai.azure.com'
# Proxy-managed LLM provider APIs. Every credentials[].apiKey.inject
# domain below MUST also appear here. sbx does not derive allow entries
# from inject rules.
- 'api.openai.com'
- 'api.anthropic.com'
- 'generativelanguage.googleapis.com'
- 'api.cohere.ai'
- 'api.groq.com'
- 'openrouter.ai'
- 'api.ai21.com'
- 'api.cloudflare.com'
- 'api.deepinfra.com'
- 'api.deepseek.com'
- 'api.mistral.ai'
- 'api.perplexity.ai'
- 'api.voyageai.com'
- 'api.x.ai'
- 'api.jina.ai'
- 'r.jina.ai'
- 'qianfan.baidubce.com'
- 'api.hunyuan.cloud.tencent.com'
- 'api.minimax.chat'
- 'api.moonshot.cn'
- 'dashscope.aliyuncs.com'
- 'open.bigmodel.cn'
# Proxy-managed LLM providers: inside the sandbox each apiKey env var holds
# the `proxy-managed` sentinel; the proxy injects the real value into the
# request header per the inject rules at request time. Values are bound by
# the user via credential bindings (`sbx secret set <service>`); Coyote
# pre-seeds them from its vault at launch. Multiple domains may map to one
# service (e.g. jina) so they share a single credential.
credentials:
sources:
openai:
env:
- OPENAI_API_KEY
anthropic:
env:
- ANTHROPIC_API_KEY
gemini:
env:
- GEMINI_API_KEY
- GOOGLE_API_KEY
cohere:
env:
- COHERE_API_KEY
groq:
env:
- GROQ_API_KEY
openrouter:
env:
- OPENROUTER_API_KEY
ai21:
env:
- AI21_API_KEY
cloudflare:
env:
- CLOUDFLARE_API_KEY
deepinfra:
env:
- DEEPINFRA_API_KEY
deepseek:
env:
- DEEPSEEK_API_KEY
mistral:
env:
- MISTRAL_API_KEY
perplexity:
env:
- PERPLEXITY_API_KEY
voyageai:
env:
- VOYAGE_API_KEY
xai:
env:
- XAI_API_KEY
jina:
env:
- JINA_API_KEY
ernie:
env:
- ERNIE_API_KEY
hunyuan:
env:
- HUNYUAN_API_KEY
minimax:
env:
- MINIMAX_API_KEY
moonshot:
env:
- MOONSHOT_API_KEY
qianwen:
env:
- DASHSCOPE_API_KEY
zhipuai:
env:
- ZHIPUAI_API_KEY
- service: openai
description: OpenAI API key, injected on api.openai.com
apiKey:
name: OPENAI_API_KEY
proxyManaged: true
inject:
- domain: api.openai.com
scheme: bearer
- service: anthropic
description: Anthropic API key, injected as x-api-key on api.anthropic.com
apiKey:
name: ANTHROPIC_API_KEY
proxyManaged: true
inject:
- domain: api.anthropic.com
header: x-api-key
format: '%s'
- service: gemini
description: Google Gemini API key, injected as x-goog-api-key on generativelanguage.googleapis.com
apiKey:
name: GEMINI_API_KEY
proxyManaged: true
inject:
- domain: generativelanguage.googleapis.com
header: x-goog-api-key
format: '%s'
- service: cohere
description: Cohere API key, injected on api.cohere.ai
apiKey:
name: COHERE_API_KEY
proxyManaged: true
inject:
- domain: api.cohere.ai
scheme: bearer
- service: groq
description: Groq API key, injected on api.groq.com
apiKey:
name: GROQ_API_KEY
proxyManaged: true
inject:
- domain: api.groq.com
scheme: bearer
- service: openrouter
description: OpenRouter API key, injected on openrouter.ai
apiKey:
name: OPENROUTER_API_KEY
proxyManaged: true
inject:
- domain: openrouter.ai
scheme: bearer
- service: ai21
description: AI21 Labs API key, injected on api.ai21.com
apiKey:
name: AI21_API_KEY
proxyManaged: true
inject:
- domain: api.ai21.com
scheme: bearer
- service: cloudflare
description: Cloudflare Workers AI API key, injected on api.cloudflare.com
apiKey:
name: CLOUDFLARE_API_KEY
proxyManaged: true
inject:
- domain: api.cloudflare.com
scheme: bearer
- service: deepinfra
description: DeepInfra API key, injected on api.deepinfra.com
apiKey:
name: DEEPINFRA_API_KEY
proxyManaged: true
inject:
- domain: api.deepinfra.com
scheme: bearer
- service: deepseek
description: DeepSeek API key, injected on api.deepseek.com
apiKey:
name: DEEPSEEK_API_KEY
proxyManaged: true
inject:
- domain: api.deepseek.com
scheme: bearer
- service: mistral
description: Mistral API key, injected on api.mistral.ai
apiKey:
name: MISTRAL_API_KEY
proxyManaged: true
inject:
- domain: api.mistral.ai
scheme: bearer
- service: perplexity
description: Perplexity API key, injected on api.perplexity.ai
apiKey:
name: PERPLEXITY_API_KEY
proxyManaged: true
inject:
- domain: api.perplexity.ai
scheme: bearer
- service: voyageai
description: Voyage AI API key, injected on api.voyageai.com
apiKey:
name: VOYAGE_API_KEY
proxyManaged: true
inject:
- domain: api.voyageai.com
scheme: bearer
- service: xai
description: xAI (Grok) API key, injected on api.x.ai
apiKey:
name: XAI_API_KEY
proxyManaged: true
inject:
- domain: api.x.ai
scheme: bearer
- service: jina
description: Jina API key, injected on api.jina.ai and r.jina.ai
apiKey:
name: JINA_API_KEY
proxyManaged: true
inject:
- domain: api.jina.ai
scheme: bearer
- domain: r.jina.ai
scheme: bearer
- service: ernie
description: Baidu ERNIE API key, injected on qianfan.baidubce.com
apiKey:
name: ERNIE_API_KEY
proxyManaged: true
inject:
- domain: qianfan.baidubce.com
scheme: bearer
- service: hunyuan
description: Tencent Hunyuan API key, injected on api.hunyuan.cloud.tencent.com
apiKey:
name: HUNYUAN_API_KEY
proxyManaged: true
inject:
- domain: api.hunyuan.cloud.tencent.com
scheme: bearer
- service: minimax
description: MiniMax API key, injected on api.minimax.chat
apiKey:
name: MINIMAX_API_KEY
proxyManaged: true
inject:
- domain: api.minimax.chat
scheme: bearer
- service: moonshot
description: Moonshot AI API key, injected on api.moonshot.cn
apiKey:
name: MOONSHOT_API_KEY
proxyManaged: true
inject:
- domain: api.moonshot.cn
scheme: bearer
- service: qianwen
description: Alibaba Qianwen (DashScope) API key, injected on dashscope.aliyuncs.com
apiKey:
name: DASHSCOPE_API_KEY
proxyManaged: true
inject:
- domain: dashscope.aliyuncs.com
scheme: bearer
- service: zhipuai
description: Zhipu AI (GLM) API key, injected on open.bigmodel.cn
apiKey:
name: ZHIPUAI_API_KEY
proxyManaged: true
inject:
- domain: open.bigmodel.cn
scheme: bearer
environment:
variables:
@@ -213,32 +270,14 @@ environment:
COYOTE_LOG_LEVEL: INFO
COYOTE_CONFIG_DIR: /home/agent/.config/coyote
EDITOR: nano
proxyManaged:
- OPENAI_API_KEY
- ANTHROPIC_API_KEY
- GEMINI_API_KEY
- GOOGLE_API_KEY
- COHERE_API_KEY
- GROQ_API_KEY
- OPENROUTER_API_KEY
- AI21_API_KEY
- CLOUDFLARE_API_KEY
- DEEPINFRA_API_KEY
- DEEPSEEK_API_KEY
- MISTRAL_API_KEY
- PERPLEXITY_API_KEY
- VOYAGE_API_KEY
- XAI_API_KEY
- JINA_API_KEY
- ERNIE_API_KEY
- HUNYUAN_API_KEY
- MINIMAX_API_KEY
- MOONSHOT_API_KEY
- DASHSCOPE_API_KEY
- ZHIPUAI_API_KEY
# Alias for the gemini credential: v2 apiKey supports a single env name
# (GEMINI_API_KEY above). Coyote also recognizes GOOGLE_API_KEY, so keep
# it set to the sentinel. Header injection happens per-domain regardless
# of which env var the app reads.
GOOGLE_API_KEY: proxy-managed
commands:
initFiles:
setup:
files:
- path: /home/agent/.config/git/ssh-signing-key-command
mode: '0755'
description: Resolve the forwarded SSH agent key for Git SSH signing
@@ -290,39 +329,45 @@ commands:
background: false
description: Bootstrap Coyote config directory on first sandbox start
agentContext: |
## Sandbox environment
agentInstructions:
filename: COYOTE.md
content: |
## Sandbox environment
You are running inside a Docker sandbox launched via `sbx run coyote`. The
user's project workspace is mounted at its absolute host path and is the
current working directory. `sudo` is passwordless; use it for system
package installs.
You are running inside a Docker sandbox launched via `sbx run coyote`. The
user's project workspace is mounted at its absolute host path and is the
current working directory. `sudo` is passwordless; use it for system
package installs.
Coyote's configuration lives at `~/.config/coyote/` and logs at
`~/.cache/coyote/coyote.log`. Persistence is enabled, so config, sessions,
vault state, OAuth tokens, and installed tools survive sandbox restarts.
Coyote's configuration lives at `~/.config/coyote/` and logs at
`~/.cache/coyote/coyote.log`. Persistence is enabled, so config, sessions,
vault state, OAuth tokens, and installed tools survive sandbox restarts.
LLM provider credentials are forwarded by the sandbox HTTP proxy. The
following provider env vars are recognized - export the ones you use on
the host before running `sbx run coyote`:
LLM provider credentials are forwarded by the sandbox HTTP proxy via
credential bindings. Coyote pre-seeds them from its vault at launch
(`sbx secret set <service>`); users can also bind values manually on the
host with `sbx secret set <service>` or `sbx secret import`. Recognized
services:
OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY / GOOGLE_API_KEY,
COHERE_API_KEY, GROQ_API_KEY, OPENROUTER_API_KEY, AI21_API_KEY,
CLOUDFLARE_API_KEY, DEEPINFRA_API_KEY, DEEPSEEK_API_KEY,
MISTRAL_API_KEY, PERPLEXITY_API_KEY, VOYAGE_API_KEY, XAI_API_KEY,
JINA_API_KEY, ERNIE_API_KEY, HUNYUAN_API_KEY, MINIMAX_API_KEY,
MOONSHOT_API_KEY, DASHSCOPE_API_KEY (Qwen), ZHIPUAI_API_KEY
openai, anthropic, gemini, cohere, groq, openrouter, ai21,
cloudflare, deepinfra, deepseek, mistral, perplexity, voyageai,
xai, jina, ernie, hunyuan, minimax, moonshot, qianwen, zhipuai
Inside the sandbox these appear as the placeholder string `proxy-managed`;
the proxy substitutes the real value at request time. OAuth flows for
Claude Pro/Max and Gemini are also allow-listed.
Inside the sandbox the corresponding env vars (OPENAI_API_KEY, etc.)
hold the placeholder string `proxy-managed`; the proxy substitutes the
real value at request time. OAuth flows for Claude Pro/Max and Gemini
are also allow-listed.
Bedrock (AWS) and VertexAI (Google Cloud) use signed/OAuth-token requests
that the proxy cannot rewrite. Their domains are allow-listed but you must
inject credentials yourself via `sbx run --env AWS_ACCESS_KEY_ID=...` or
a mixin kit that mounts a service-account JSON.
Bedrock (AWS) and VertexAI (Google Cloud) use signed/OAuth-token requests
that the proxy cannot rewrite, so you must inject credentials yourself via
`sbx run --env AWS_ACCESS_KEY_ID=...` or a mixin kit that mounts a
service-account JSON. VertexAI regional endpoints are allow-listed via
`*.googleapis.com`. Bedrock runtime endpoints are allow-listed for
us-east-1/2, us-west-2, eu-west-1, eu-central-1, ap-southeast-2, and
ap-northeast-1 only; other regions need a mixin allow entry
(`bedrock-runtime.<region>.amazonaws.com`).
Useful first-run commands:
- `coyote --info` # show config paths and resolved settings
- `coyote --list-secrets` # initialise the local vault
- `coyote --authenticate <client>` # OAuth flow (Claude Pro/Max, Gemini)
Useful first-run commands:
- `coyote --info` # show config paths and resolved settings
- `coyote --list-secrets` # initialise the local vault
- `coyote --authenticate <client>` # OAuth flow (Claude Pro/Max, Gemini)
+2 -23
View File
@@ -414,11 +414,10 @@ pub fn list_rags() -> Vec<String> {
for entry in rd.flatten() {
let name = entry.file_name();
if let Some(name) = name.to_string_lossy().strip_suffix(".yaml") {
// Sidecars are not RAGs. `.duckdb` files are already excluded by
// the `.yaml` suffix check above; this rejects `<name>.sbx-mixin`.
if is_rag_sidecar_name(name) {
continue;
}
names.push(name.to_string());
}
}
@@ -429,24 +428,10 @@ pub fn list_rags() -> Vec<String> {
}
}
/// True for the sidecar YAML files that must never be listed or deleted as RAGs.
/// `name` is the already-stripped stem (i.e. after `strip_suffix(".yaml")`).
/// Uses `ends_with`, not `contains('.')`, so a RAG legitimately named "v2.docs" is
/// not rejected.
pub(crate) fn is_rag_sidecar_name(name: &str) -> bool {
name.ends_with(".sbx-mixin")
}
/// Remove every sidecar belonging to RAG `name` in `dir`. Missing files are NOT an
/// error. A failure to remove an EXISTING mixin IS an error and must propagate — a
/// silently-orphaned mixin keeps a sandbox network permission alive after the user
/// believes it is gone. The `.duckdb` orphan is only wasted disk, so its removal
/// failure is ignorable; the asymmetry is deliberate.
///
/// Callers must run this BEFORE unlinking the primary `.yaml`. If the YAML goes first
/// and this then fails, the RAG disappears from `list_rags()` — so the user can no
/// longer select it to retry — while its `allowedDomains` entry keeps being injected
/// into every sandbox launch.
pub(crate) fn remove_rag_sidecars(dir: &Path, name: &str) -> Result<()> {
let duckdb_path = dir.join(format!("{name}.duckdb"));
if duckdb_path.exists() {
@@ -463,6 +448,7 @@ pub(crate) fn remove_rag_sidecars(dir: &Path, name: &str) -> Result<()> {
)
})?;
}
Ok(())
}
@@ -889,8 +875,6 @@ mod tests {
let _ = fs::remove_dir_all(&root);
}
/// Unique temp dir for the sidecar helper tests. These take `dir: &Path` directly,
/// so no env-var mutation and therefore no `#[serial]` is needed.
fn sidecar_temp_dir(label: &str) -> PathBuf {
let unique = time::SystemTime::now()
.duration_since(time::UNIX_EPOCH)
@@ -903,7 +887,6 @@ mod tests {
#[test]
fn is_rag_sidecar_name_accepts_dotted_rag_names() {
// A RAG legitimately named "v2.docs" must not be mistaken for a sidecar.
assert!(!is_rag_sidecar_name("v2.docs"));
assert!(!is_rag_sidecar_name("myrag"));
assert!(is_rag_sidecar_name("myrag.sbx-mixin"));
@@ -940,8 +923,6 @@ mod tests {
let root = sidecar_temp_dir("rag-sidecars-order");
let yaml = root.join("docs.yaml");
fs::write(&yaml, "rag").unwrap();
// A non-empty DIRECTORY at the mixin path makes remove_file fail, standing in
// for any real removal failure (permissions, a busy mount).
let mixin = root.join("docs.sbx-mixin.yaml");
fs::create_dir_all(&mixin).unwrap();
fs::write(mixin.join("blocker"), "x").unwrap();
@@ -952,8 +933,6 @@ mod tests {
.contains("Failed to remove the sandbox mixin"),
"got: {err}"
);
// The whole point of removing sidecars first: the RAG is still on disk, still
// listed, and the deletion is retryable.
assert!(
yaml.exists(),
"the .yaml must survive a sidecar-removal failure so the delete is retryable"
+2 -24
View File
@@ -142,11 +142,6 @@ pub struct RequestContext {
pub role: Option<Role>,
pub session: Option<Session>,
pub rag: Option<Arc<Rag>>,
/// The cache key `self.rag` was actually inserted under, carried rather than
/// reconstructed. Reconstruction was the bug: the invalidation sites do not have
/// the information needed to rebuild the key (agent RAGs are inserted under the
/// AGENT's name but `rag.name()` is the constant "rag"), so insert and invalidate
/// silently disagreed. `None` for the temp RAG, which bypasses the cache entirely.
pub rag_key: Option<RagKey>,
pub agent: Option<Agent>,
@@ -4122,14 +4117,10 @@ impl RequestContext {
}
let app = self.app.config.clone();
// Hoisted: `rag_cache` below borrows `self`, so the loader closure cannot
// reach through `self` for the vault. `GlobalVault` is an Arc, so this is cheap.
let vault = self.app.vault.clone();
let rag_cache = self.rag_cache();
let working_mode = self.working_mode;
// The key is returned alongside the Rag rather than assigned inside the match:
// `rag_cache` borrows `self`, so writing `self.rag_key` there is E0506.
let (rag, rag_key): (Arc<Rag>, Option<RagKey>) = match rag {
None => {
let rag_path = self.rag_file(super::TEMP_RAG_NAME);
@@ -4138,7 +4129,6 @@ impl RequestContext {
format!("Failed to cleanup previous '{}' rag", super::TEMP_RAG_NAME)
})?;
}
// The temp RAG is never inserted into the cache, so it has no key.
(
Arc::new(
Rag::init(
@@ -4198,13 +4188,9 @@ impl RequestContext {
let vault = self.app.vault.clone();
let rag = Rag::attach(app, &vault, name, &rag_path).await?;
let rag = Arc::new(rag);
// Populate the cache so a later `.rag <name>` reuses this instance rather
// than re-running the network preflight. Attach is always a global RAG.
let key = RagKey::Named(name.to_string());
self.rag_cache().insert(key.clone(), &rag);
self.rag = Some(rag);
// Carried so invalidation in rebuild_rag()/edit_rag_docs() can find this
// entry; without it a stale Arc would linger in the cache all session.
self.rag_key = Some(key);
Ok(())
}
@@ -4217,7 +4203,7 @@ impl RequestContext {
if rag.is_attached() {
bail!(
"Cannot edit documents on an attached RAG Coyote does not own its source documents."
"Cannot edit documents on an attached RAG; Coyote does not own its source documents."
);
}
@@ -4270,7 +4256,7 @@ impl RequestContext {
if rag.is_attached() {
bail!(
"Cannot rebuild an attached RAG Coyote does not own its source documents. \
"Cannot rebuild an attached RAG; Coyote does not own its source documents. \
Re-index from the system that originally created '{}'.",
rag.name()
);
@@ -4716,8 +4702,6 @@ mod tests {
)
.unwrap();
// Stand in for the state `.rag docs` leaves behind: `use_rag` sets `rag` and
// `rag_key` together, so a named key is live when the agent is entered.
ctx.rag_key = Some(RagKey::Named("docs".to_string()));
tokio::runtime::Builder::new_current_thread()
@@ -4730,9 +4714,6 @@ mod tests {
.unwrap();
});
// This agent has no RAG, so `rag` is None and `rag_key` must be None as well.
// Carrying `Named("docs")` across the transition would point `.rebuild rag`
// at an unrelated RAG's cache entry.
assert!(ctx.rag.is_none());
assert_eq!(ctx.rag_key, None);
}
@@ -6122,9 +6103,6 @@ mod tests {
assert!(paths::list_rags().is_empty());
}
/// A `<name>.sbx-mixin.yaml` sidecar must not appear as a phantom RAG in TAB
/// completion or `.list rag`. A RAG whose name legitimately contains a dot must
/// still be listed — the filter uses `ends_with`, not `contains('.')`.
#[test]
#[serial]
fn list_rags_skips_sbx_mixin_sidecars() {
+260 -210
View File
@@ -12,10 +12,8 @@ mod splitter;
use self::graph::{KnowledgeGraph, extract_entities};
use self::provider::RagProvider;
// `providers::duckdb_path_from_yaml(path)` is called through the module path in
// `create()`, so `providers` itself must stay in scope — do not collapse it into
// the `use` below.
use self::providers::{DuckDbProvider, QdrantProvider, YamlProvider};
use crate::sandbox::mcp_credentials;
use crate::vault::{Vault, interpolate_secrets};
use anyhow::{Context, Result, anyhow, bail};
@@ -64,10 +62,7 @@ pub struct Rag {
name: String,
path: String,
embedding_model: Model,
// Local BM25: keyword search + graph seeding. Always built from `data.files`
// regardless of driver, and kept on `Rag` so the sync `graph_search` can use it.
bm25: SearchEngine<DocumentId>,
// Vector storage + content retrieval.
provider: Box<dyn RagProvider>,
data: RagData,
last_sources: RwLock<Option<String>>,
@@ -125,8 +120,7 @@ pub struct RagInitConfig {
pub extractor_model: Option<String>,
pub extractor_prompt: Option<String>,
pub graph_hops: Option<usize>,
/// `None` -> "yaml". No serde attribute: this struct derives only
/// `Debug, Clone, Default` and is built in Rust, never deserialized.
/// `None` -> "yaml"
pub driver: Option<String>,
}
@@ -366,8 +360,6 @@ impl Rag {
// plaintext.
let data: RagData = serde_yaml::from_str(&raw_content).with_context(err)?;
// Validated before the match so the rule applies to every driver, including
// the sync fallthrough below.
data.validate().with_context(err)?;
match data.driver.as_str() {
@@ -383,7 +375,6 @@ impl Rag {
.context("qdrant driver requires 'collection' in driver_config")?
.clone();
// Resolved out of band and kept in a local; it never enters `data`.
let api_key: Option<String> = match data.driver_config.get("api_key") {
Some(placeholder) => {
let (resolved, _) =
@@ -410,14 +401,10 @@ impl Rag {
last_sources: RwLock::new(None),
})
}
// yaml/duckdb take the sync path. It re-reads and re-parses the file;
// that cost is accepted to keep every existing caller untouched.
_ => Self::load(app, name, path),
}
}
/// Connects to a pre-existing external collection. Coyote is a query-only
/// client here: it never indexes documents into it.
pub async fn attach(
app: &AppConfig,
vault: &Vault,
@@ -434,8 +421,6 @@ impl Rag {
let host = Text::new("Host (e.g. qdrant.company.com:6333):")
.with_validator(required!("This field is required"))
.with_validator(|input: &str| {
// Bracketed IPv6 literals would produce a malformed sandbox
// allowedDomains entry, so refuse them at the prompt.
Ok(if input.contains('[') || input.contains(']') {
Validation::Invalid(
"Bracketed IPv6 literals are not supported; use a hostname.".into(),
@@ -568,14 +553,20 @@ impl Rag {
rag.save()?;
println!("✓ Attached '{name}' → collection '{collection}' on {host}.");
let env_var = rag_env_var_name(name);
let env_var = api_key_entry.as_ref().map(|_| rag_env_var_name(name));
let (header_name, value_format) = driver_auth_header(driver);
generate_rag_sbx_mixin(save_path, &host, name, &env_var, header_name, value_format)?;
generate_rag_sbx_mixin(
save_path,
&host,
name,
env_var.as_deref(),
header_name,
value_format,
)?;
Ok(rag)
}
/// `mut data` — the duckdb arm rehydrates `data.vectors` from the sidecar.
pub fn create(app: &AppConfig, name: &str, path: &Path, mut data: RagData) -> Result<Self> {
// Deliberately does NOT call rebuild_indexes: both callers construct the Rag
// before any documents are added, so rebuilding empty data would be a no-op.
@@ -595,11 +586,11 @@ impl Rag {
// Guarded on is_empty() so a caller that already has vectors in memory
// is never overwritten by an empty table.
//
// 🔴 `?`, NOT `unwrap_or_default()`. A hydration failure must propagate.
// Degrading to an empty map here loads a RAG that looks healthy, answers
// every query with nothing, and then loses the store permanently on the
// first `.edit rag-docs`. The legitimate "nothing indexed yet" case is
// already Ok(empty) open() runs CREATE TABLE IF NOT EXISTS so `?`
// WARNING: `?`, NOT `unwrap_or_default()`. A hydration failure must
// propagate. Degrading to an empty map here loads a RAG that looks healthy,
// answers every query with nothing, and then loses the store permanently
// on the first `.edit rag-docs`. The legitimate "nothing indexed yet" case
// is already Ok(empty) (open() runs CREATE TABLE IF NOT EXISTS) so `?`
// costs a new RAG nothing.
if data.vectors.is_empty() {
data.vectors = duck.read_all_vectors()?;
@@ -614,7 +605,6 @@ impl Rag {
use Rag::attach() or Rag::load_async() instead"
),
_ => {
// "yaml" and any unknown driver — in-memory HNSW.
let bm25 = data.build_bm25();
(Box::new(YamlProvider::from_data(&data)), bm25)
}
@@ -722,7 +712,7 @@ impl Rag {
// `data.files` is empty for an attached RAG; the local index is not the
// source of truth. A static label is honest, an empty list is not.
*self.last_sources.write() =
Some("[attached RAG — source list unavailable]".to_string());
Some("[Using attached RAG. Source list unavailable]".to_string());
return;
}
let mut sources: IndexMap<String, Vec<String>> = IndexMap::new();
@@ -914,6 +904,7 @@ impl Rag {
if self.data.attached {
return format!("- {}", self.data.attached_source_label());
}
let mut seen = IndexSet::new();
for id in ids {
let (file_index, _) = id.split();
@@ -1194,20 +1185,13 @@ impl Rag {
let keyword_search_results: Vec<(DocumentId, f32)> =
if self.provider.has_native_keyword_search() {
// Keyword is ONE of three RRF rankers (vector + keyword + graph);
// its absence is survivable and produces a slightly worse ranking,
// whereas a `?` here turns a provider FTS fault into TOTAL query
// failure — the user gets an error instead of the results the
// vector and graph rankers already retrieved. Degrade, do not
// propagate, and do not silently swap in the local BM25 either:
// that would change the ranking algorithm mid-query.
match self.provider.keyword_search(query, top_k).await {
Ok(v) => v,
Err(e) => {
self.provider
.keyword_search(query, top_k)
.await
.unwrap_or_else(|e| {
warn!("native keyword search failed, dropping the keyword ranker: {e}");
Vec::new()
}
}
})
} else {
self.keyword_search(query, top_k, 0.0)
};
@@ -1223,8 +1207,6 @@ impl Rag {
.concat()
.into_iter()
.collect();
// `ids` is an `IndexSet` here, not the `Vec` of the RRF branch below,
// and `&IndexSet<_>` does not coerce to `&[DocumentId]`.
let ids: Vec<DocumentId> = ids.into_iter().collect();
let fetched = self.provider.fetch_content(&ids).await?;
// Build both vectors from the SAME source in the SAME iteration —
@@ -1268,8 +1250,6 @@ impl Rag {
ids
}
};
// `ids` is the ranked list; `fetch_content` preserves that order per the
// trait's ordering contract, so the result is returned as-is.
let output = self.provider.fetch_content(&ids).await?;
Ok(output)
}
@@ -1300,7 +1280,7 @@ impl Rag {
Ok(merge_vector_results(results))
}
/// Local in-memory BM25 over `data.files` empty for attached RAGs, which is
/// Local in-memory BM25 over `data.files`. This is empty for attached RAGs, which is
/// correct: they have no local text.
fn keyword_search(&self, query: &str, top_k: usize, min_score: f32) -> Vec<(DocumentId, f32)> {
let results = self.bm25.search(query, top_k);
@@ -1471,9 +1451,6 @@ pub struct RagData {
pub driver: String,
#[serde(default)]
pub attached: bool,
/// Driver-specific connection parameters (qdrant: `host`, `collection`, `api_key`).
/// Secret-bearing values are stored as `{{SECRET_NAME}}` placeholders and resolved
/// out of band at load time, so a resolved credential never reaches disk.
#[serde(default, skip_serializing_if = "IndexMap::is_empty")]
pub driver_config: IndexMap<String, String>,
@@ -1581,6 +1558,7 @@ impl RagData {
no results with no error. Set `top_k:` in the RAG YAML."
);
}
if !self.attached {
if self.chunk_size == 0 {
bail!(
@@ -1589,6 +1567,7 @@ impl RagData {
embedding batches. Set `chunk_size:` in the RAG YAML."
);
}
if self.chunk_overlap >= self.chunk_size {
bail!(
"chunk_overlap ({}) must be strictly less than chunk_size ({}).",
@@ -1597,6 +1576,7 @@ impl RagData {
);
}
}
match (self.driver.as_str(), self.attached) {
("yaml", false) => Ok(()),
("duckdb", false) => Ok(()),
@@ -1618,7 +1598,7 @@ impl RagData {
/// Every (DocumentId, &RagDocument) in the corpus, in `files` order.
///
/// This NOT `vectors` is the authoritative document id space. BM25, the
/// This, NOT `vectors`, is the authoritative document id space. BM25, the
/// knowledge graph and content lookup all key off it; `vectors` is a subset,
/// since `add`'s zip truncates whenever fewer embeddings come back than
/// document ids were sent.
@@ -1761,92 +1741,90 @@ impl DocumentId {
/// byte-for-byte to `spec.yaml` inside a kit dir handed to `sbx create --kit`,
/// with no `kind` rewrite — so an envelope-less file can break the launch
/// itself, not merely this RAG's traffic.
/// 2. `allowedDomains` entries carry a port; `serviceDomains` keys are bare
/// hostnames. The asymmetry is deliberate.
/// 2. The `service` declared here must be the id the host binds the value
/// under with `sbx secret set`; both derive from the RAG name through
/// `secret_service_id`, so they cannot spell it differently.
///
/// `api_key_env` is `None` for a store that needs no credential: the host is
/// still allowed, but no binding is declared, because nothing binds a value.
fn generate_rag_sbx_mixin(
rag_yaml_path: &Path,
host: &str,
service_name: &str,
env_var: &str,
api_key_env: Option<&str>,
header_name: &str,
value_format: &str,
) -> Result<()> {
let (bare_host, allowed_domain) = sbx_domain_forms(host);
let base_url = QdrantProvider::normalize_base_url(host);
let Some(allow_entry) = mcp_credentials::allow_entry_for_url(&base_url) else {
eprintln!(
"Warning: host '{host}' has no representation in the sbx network allow \
grammar, so no sandbox mixin was written for RAG '{service_name}'. \
Queries to this RAG will be blocked inside the sandbox."
);
return Ok(());
};
let credentials = api_key_env
.map(|env_var| mcp_credentials::CredentialEntry {
service: mcp_credentials::secret_service_id(service_name),
description: format!("API key for the attached RAG '{service_name}'"),
api_key: mcp_credentials::ApiKey {
name: env_var.to_string(),
proxy_managed: true,
inject: vec![rag_inject_rule(&allow_entry, header_name, value_format)],
},
})
.into_iter()
.collect();
let mixin_path = rag_yaml_path.with_extension("sbx-mixin.yaml");
let content = format!(
r#"schemaVersion: "1"
kind: mixin
name: rag-{service_name}
description: >
Auto-generated by the Coyote attach wizard for RAG '{service_name}'. Allows
outbound traffic to its external vector store and tells the sbx proxy which
header to rewrite with the stored credential. Do not edit manually.
network:
allowedDomains:
- "{allowed_domain}"
serviceDomains:
{bare_host}: {service_name}
serviceAuth:
{service_name}:
headerName: {header_name}
valueFormat: "{value_format}"
credentials:
sources:
{service_name}:
env:
- {env_var}
environment:
proxyManaged:
- {env_var}
"#
);
let content = mcp_credentials::render_mixin_document(
&format!("rag-{service_name}"),
&format!(
"Auto-generated by the Coyote attach wizard for RAG '{service_name}'. Allows \
outbound traffic to its external vector store and declares the credential the \
sbx proxy injects into each request. Do not edit manually."
),
credentials,
&[allow_entry],
)?;
fs::write(&mixin_path, &content).with_context(|| {
format!(
"Failed to write sandbox mixin to '{}'",
mixin_path.display()
)
})?;
println!("✓ Sandbox mixin: '{}'.", mixin_path.display());
Ok(())
}
/// Splits a user-supplied host into the two forms sbx needs:
/// `(bare_host, allowed_domain)`.
///
/// `bare_host` is the `serviceDomains` KEY — hostname only, no scheme, no port.
/// `allowed_domain` is an `allowedDomains` ENTRY — always `host:port`.
///
/// The rule: emit the host verbatim when it already carries a numeric port,
/// otherwise append the default for the scheme. Stripping the port fails
/// silently — the sandbox just denies the connection, with no compile or test
/// signal. Defaults match `normalize_base_url`, which assumes `http://` when no
/// scheme is given: plain host → 6333, explicit `https://` → 443.
fn sbx_domain_forms(host: &str) -> (String, String) {
let is_https = host.starts_with("https://");
let hostport = host
.strip_prefix("https://")
.or_else(|| host.strip_prefix("http://"))
.unwrap_or(host)
.trim_end_matches('/')
// A trailing ':' with no digits is a typo, not a port. Trim it so the
// default-port arm cannot emit "host::6333".
.trim_end_matches(':');
match hostport.rsplit_once(':') {
Some((h, p)) if !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit()) => {
(h.to_string(), hostport.to_string())
fn rag_inject_rule(
domain: &str,
header_name: &str,
value_format: &str,
) -> mcp_credentials::InjectRule {
if header_name.eq_ignore_ascii_case("authorization")
&& value_format.eq_ignore_ascii_case("Bearer %s")
{
mcp_credentials::InjectRule {
domain: domain.to_string(),
header: None,
format: None,
scheme: Some("bearer".to_string()),
}
_ => {
let port = if is_https { 443 } else { 6333 };
(hostport.to_string(), format!("{hostport}:{port}"))
} else {
mcp_credentials::InjectRule {
domain: domain.to_string(),
header: Some(header_name.to_string()),
format: Some(value_format.to_string()),
scheme: None,
}
}
}
/// Derives a deterministic env var name from a RAG name:
/// `company-docs` → `COMPANY_DOCS_API_KEY`.
fn rag_env_var_name(rag_name: &str) -> String {
format!(
"{}_API_KEY",
@@ -1854,8 +1832,6 @@ fn rag_env_var_name(rag_name: &str) -> String {
)
}
/// How a driver authenticates its HTTP requests. Qdrant uses a bare `api-key`
/// header rather than `Authorization: Bearer`.
fn driver_auth_header(driver: &str) -> (&'static str, &'static str) {
match driver {
"qdrant" => ("api-key", "%s"),
@@ -2093,8 +2069,6 @@ fn find_hash_skip(
/// so the pool is bounded by `top_k * query_chunks`, and `reciprocal_rank_fusion`
/// truncates to `top_k` itself. Capping here would let whichever query chunk has
/// the strongest absolute scores crowd out every other chunk's hits.
///
/// Free function (not a method) so it is unit-testable without an embeddings client.
fn merge_vector_results(mut results: Vec<(DocumentId, f32)>) -> Vec<(DocumentId, f32)> {
debug_assert!(
results.iter().all(|(_, score)| score.is_finite()),
@@ -2150,16 +2124,17 @@ fn embedding_dim_for_model(model_id: &str) -> usize {
#[cfg(test)]
mod tests {
use super::*;
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
/// Scratch directory for tests that must write a real file.
struct TempDir {
path: std::path::PathBuf,
path: PathBuf,
}
impl TempDir {
fn new(tag: &str) -> Self {
let unique = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
let unique = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos();
let path = env::temp_dir().join(format!("coyote-rag-{tag}-{unique}"));
@@ -2226,99 +2201,164 @@ mod tests {
);
}
/// `allowedDomains` entries are host:PORT; `serviceDomains` keys are bare
/// hostnames. Stripping the port breaks sandbox whitelisting silently — no
/// compile error, no runtime error on the host, just a denied connection
/// inside the sandbox. This test is the only signal.
#[test]
fn sbx_domain_forms_keeps_explicit_ports_and_defaults_the_rest() {
assert_eq!(
sbx_domain_forms("rag.example.com:6333"),
("rag.example.com".into(), "rag.example.com:6333".into())
);
// A non-default explicit port must survive.
assert_eq!(
sbx_domain_forms("rag.example.com:7777").1,
"rag.example.com:7777"
);
// Bare host → Qdrant's REST default, matching normalize_base_url.
assert_eq!(
sbx_domain_forms("rag.example.com"),
("rag.example.com".into(), "rag.example.com:6333".into())
);
// The scheme is stripped; http keeps 6333.
assert_eq!(
sbx_domain_forms("http://localhost:6333").1,
"localhost:6333"
);
// https with no port → 443 (the Qdrant Cloud shape).
assert_eq!(
sbx_domain_forms("https://xyz.cloud.qdrant.io"),
(
"xyz.cloud.qdrant.io".into(),
"xyz.cloud.qdrant.io:443".into()
)
);
// A dangling colon is a typo, not a port: trimmed, then defaulted.
assert_eq!(
sbx_domain_forms("rag.example.com:").1,
"rag.example.com:6333"
);
}
/// The generated mixin must carry the schema envelope: `wrap_mixin_as_kit`
/// copies it verbatim to `spec.yaml` for `sbx create --kit`, with no `kind`
/// rewrite, so an envelope-less file can break the launch itself.
#[test]
fn generated_sbx_mixin_carries_the_schema_envelope() {
/// Renders a sidecar into a scratch dir and returns its text plus the
/// parsed document. Parsing is not optional: a malformed document is
/// otherwise copied verbatim into `spec.yaml` and only rejected by sbx.
fn render_rag_mixin(
host: &str,
name: &str,
api_key_env: Option<&str>,
header_name: &str,
value_format: &str,
) -> (String, serde_yaml::Value) {
let dir = TempDir::new("mixin");
let yaml_path = dir.path.join("company-docs.yaml");
let yaml_path = dir.path.join(format!("{name}.yaml"));
generate_rag_sbx_mixin(
&yaml_path,
"rag.example.com",
"company-docs",
"COMPANY_DOCS_API_KEY",
"api-key",
"%s",
host,
name,
api_key_env,
header_name,
value_format,
)
.unwrap();
let text = fs::read_to_string(dir.path.join("company-docs.sbx-mixin.yaml")).unwrap();
let text = fs::read_to_string(dir.path.join(format!("{name}.sbx-mixin.yaml"))).unwrap();
let parsed = serde_yaml::from_str(&text).unwrap();
(text, parsed)
}
fn allow_list(parsed: &serde_yaml::Value) -> Vec<String> {
parsed["permissions"]["network"]["allow"]
.as_sequence()
.expect("a mixin without an allow list whitelists nothing")
.iter()
.map(|v| v.as_str().unwrap().to_string())
.collect()
}
#[test]
fn generated_sbx_mixin_carries_the_schema_envelope() {
let (text, parsed) = render_rag_mixin(
"rag.example.com:6333",
"company-docs",
Some("COMPANY_DOCS_API_KEY"),
"api-key",
"%s",
);
assert!(
text.starts_with("schemaVersion:"),
"envelope must come first:\n{text}"
);
assert!(
text.contains("kind: mixin"),
"kind must be `mixin`, not `sandbox`"
);
assert!(text.contains("name: rag-company-docs"));
assert!(text.contains("description:"));
// It must parse as YAML at all — a broken format! escape is invisible otherwise.
let parsed: serde_yaml::Value = serde_yaml::from_str(&text).unwrap();
assert_eq!(parsed["schemaVersion"].as_str(), Some("2"));
assert_eq!(parsed["kind"].as_str(), Some("mixin"));
// The list entry carries the port; the map key does not.
assert_eq!(parsed["name"].as_str(), Some("rag-company-docs"));
assert!(parsed["description"].as_str().is_some());
assert_eq!(allow_list(&parsed), vec!["rag.example.com:6333"]);
let credential = &parsed["credentials"][0];
assert_eq!(credential["service"].as_str(), Some("company-docs"));
assert_eq!(
parsed["network"]["allowedDomains"][0].as_str(),
Some("rag.example.com:6333")
);
assert_eq!(
parsed["network"]["serviceDomains"]["rag.example.com"].as_str(),
Some("company-docs")
);
// The proxy rewrites this header with the credential it holds.
assert_eq!(
parsed["network"]["serviceAuth"]["company-docs"]["headerName"].as_str(),
Some("api-key")
);
assert_eq!(
parsed["credentials"]["sources"]["company-docs"]["env"][0].as_str(),
credential["apiKey"]["name"].as_str(),
Some("COMPANY_DOCS_API_KEY")
);
assert_eq!(credential["apiKey"]["proxyManaged"].as_bool(), Some(true));
let inject = &credential["apiKey"]["inject"][0];
assert_eq!(inject["domain"].as_str(), Some("rag.example.com:6333"));
assert_eq!(inject["header"].as_str(), Some("api-key"));
assert_eq!(inject["format"].as_str(), Some("%s"));
assert!(
allow_list(&parsed).contains(&inject["domain"].as_str().unwrap().to_string()),
"every inject domain must also appear in allow:\n{text}"
);
for dead in ["allowedDomains", "serviceDomains", "serviceAuth"] {
assert!(!text.contains(dead), "v1 key '{dead}' survived:\n{text}");
}
assert!(
parsed["network"].is_null(),
"v1 top-level `network` survived:\n{text}"
);
assert!(
parsed["environment"].is_null(),
"v1 `environment.proxyManaged` survived:\n{text}"
);
}
#[test]
fn generated_sbx_mixin_allows_the_port_the_client_dials() {
let cases = [
// No scheme means http, and http means port 80 — normalize_base_url
// does not silently append Qdrant's 6333.
("rag.example.com", "rag.example.com:80"),
("rag.example.com:7777", "rag.example.com:7777"),
("http://localhost:6333", "localhost:6333"),
// https on the default port is the one bare-host case.
("https://xyz.cloud.qdrant.io", "xyz.cloud.qdrant.io"),
(
"https://xyz.cloud.qdrant.io:6333",
"xyz.cloud.qdrant.io:6333",
),
];
for (host, expected) in cases {
let (_, parsed) = render_rag_mixin(host, "docs", Some("DOCS_API_KEY"), "api-key", "%s");
assert_eq!(
allow_list(&parsed),
vec![expected.to_string()],
"host {host}"
);
}
}
#[test]
fn generated_sbx_mixin_spells_bearer_as_a_scheme() {
let (_, parsed) = render_rag_mixin(
"https://store.example.com",
"docs",
Some("DOCS_API_KEY"),
"Authorization",
"Bearer %s",
);
let inject = &parsed["credentials"][0]["apiKey"]["inject"][0];
assert_eq!(inject["scheme"].as_str(), Some("bearer"));
assert!(inject["header"].is_null());
assert!(inject["format"].is_null());
}
#[test]
fn generated_sbx_mixin_service_id_matches_the_host_side_bind() {
let (_, parsed) = render_rag_mixin(
"https://store.example.com",
"My_Docs",
Some("MY_DOCS_API_KEY"),
"api-key",
"%s",
);
assert_eq!(
parsed["environment"]["proxyManaged"][0].as_str(),
Some("COMPANY_DOCS_API_KEY")
parsed["credentials"][0]["service"].as_str(),
Some(mcp_credentials::secret_service_id("My_Docs").as_str())
);
assert_eq!(
parsed["credentials"][0]["service"].as_str(),
Some("my-docs")
);
}
#[test]
fn generated_sbx_mixin_omits_credentials_when_there_is_no_api_key() {
let (text, parsed) =
render_rag_mixin("https://store.example.com", "docs", None, "api-key", "%s");
assert_eq!(allow_list(&parsed), vec!["store.example.com"]);
assert!(
parsed["credentials"].is_null(),
"no key means no credential declaration:\n{text}"
);
}
@@ -2331,7 +2371,6 @@ mod tests {
#[test]
fn driver_auth_header_uses_a_bare_api_key_for_qdrant() {
// Qdrant's REST API reads `api-key`, NOT `Authorization: Bearer`.
assert_eq!(driver_auth_header("qdrant"), ("api-key", "%s"));
assert_eq!(
driver_auth_header("something-else"),
@@ -2339,8 +2378,6 @@ mod tests {
);
}
/// An attached RAG has no local `files`, so the citation helpers would
/// otherwise emit "unknown" and an empty source list for every result.
#[test]
fn attached_rag_citation_helpers_do_not_fall_back_to_the_empty_file_index() {
let mut data = RagData {
@@ -2354,7 +2391,6 @@ mod tests {
.insert("collection".into(), "company-kb".into());
assert!(data.files.is_empty());
// The real helper — `resolve_source`/`format_sources` both delegate here.
assert_eq!(
data.attached_source_label(),
"[external collection: company-kb]"
@@ -2488,6 +2524,7 @@ mod tests {
.iter_documents()
.map(|(id, doc)| (id, doc.page_content.as_str()))
.collect();
assert_eq!(
documents,
vec![
@@ -2508,12 +2545,10 @@ mod tests {
None,
GraphRagConfig::default(),
);
assert_eq!(data.iter_documents().count(), 0);
}
/// The document id space is `files`, never `vectors`: `add`'s zip truncates
/// silently, so a vector may exist for an id no file provides. Content lookup
/// and BM25 both key off this iterator and must agree.
#[test]
fn rag_data_iter_documents_ignores_vector_only_ids() {
let mut data = RagData::new(
@@ -2784,14 +2819,17 @@ mod tests {
#[test]
fn merge_vector_results_empty_input() {
let result = super::merge_vector_results(vec![]);
let result = merge_vector_results(vec![]);
assert!(result.is_empty(), "empty input should produce empty output");
}
#[test]
fn merge_vector_results_keeps_best_score_per_document() {
let doc = DocumentId::new(0, 0);
let result = super::merge_vector_results(vec![(doc, 0.2), (doc, 0.9)]);
let result = merge_vector_results(vec![(doc, 0.2), (doc, 0.9)]);
assert_eq!(result.len(), 1, "a document must not be double-counted");
assert_eq!(result[0].0, doc);
assert_eq!(
@@ -2805,10 +2843,10 @@ mod tests {
let doc_a = DocumentId::new(0, 0);
let doc_b = DocumentId::new(1, 0);
let doc_c = DocumentId::new(2, 0);
// Interleaved as two per-chunk hit lists would arrive: concatenating them
// would yield a, c, b — only a global sort produces c, a, b.
let result = super::merge_vector_results(vec![(doc_a, 0.5), (doc_c, 0.9), (doc_b, 0.1)]);
let result = merge_vector_results(vec![(doc_a, 0.5), (doc_c, 0.9), (doc_b, 0.1)]);
let ids: Vec<DocumentId> = result.iter().map(|(id, _)| *id).collect();
assert_eq!(ids, vec![doc_c, doc_a, doc_b]);
}
@@ -2817,7 +2855,9 @@ mod tests {
let input: Vec<(DocumentId, f32)> = (0..10)
.map(|i| (DocumentId::new(i, 0), i as f32 / 10.0))
.collect();
let result = super::merge_vector_results(input);
let result = merge_vector_results(input);
assert_eq!(
result.len(),
10,
@@ -2843,6 +2883,7 @@ mod tests {
#[test]
fn force_reingest_re_embeds_hash_identical_files() {
let (files, to_deleted) = hash_skip_fixture();
assert_eq!(
find_hash_skip(true, &to_deleted, &files, "abc", "test.txt"),
None,
@@ -2853,6 +2894,7 @@ mod tests {
#[test]
fn refresh_without_force_still_hash_skips() {
let (files, to_deleted) = hash_skip_fixture();
assert_eq!(
find_hash_skip(false, &to_deleted, &files, "abc", "test.txt"),
Some((0, 7)),
@@ -2863,6 +2905,7 @@ mod tests {
#[test]
fn find_hash_skip_returns_none_on_path_change() {
let (files, to_deleted) = hash_skip_fixture();
assert_eq!(
find_hash_skip(false, &to_deleted, &files, "abc", "moved.txt"),
None
@@ -2884,6 +2927,7 @@ mod tests {
None,
GraphRagConfig::default(),
);
assert_eq!(data.driver, "yaml");
assert!(!data.attached);
}
@@ -2900,7 +2944,9 @@ document_paths: []
files: {}
vectors: {}
";
let data: RagData = serde_yaml::from_str(yaml).unwrap();
assert_eq!(data.driver, "yaml");
assert!(!data.attached);
}
@@ -2921,6 +2967,7 @@ vectors: {}
let yaml = serde_yaml::to_string(&data).unwrap();
let restored: RagData = serde_yaml::from_str(&yaml).unwrap();
assert_eq!(restored.driver, "qdrant");
assert!(restored.attached);
}
@@ -2937,7 +2984,9 @@ vectors: {}
GraphRagConfig::default(),
);
data.attached = true;
let err = data.validate().unwrap_err().to_string();
assert!(err.contains("cannot be attached"), "got: {err}");
}
@@ -2954,6 +3003,7 @@ vectors: {}
);
data.driver = "qdrant".to_string();
data.attached = true;
assert!(data.validate().is_ok());
}
+7 -18
View File
@@ -4,11 +4,6 @@ use async_trait::async_trait;
/// Abstracts where RAG vector data is stored and queried.
///
/// Implementors:
/// - YamlProvider: HNSW in-memory, state derived from RagData.vectors/files
/// - DuckDbProvider: DuckDB on-disk vector index + document store
/// - QdrantProvider: remote Qdrant collection
///
/// The Rag orchestrator owns: embeddings, chunking, BM25 keyword search, graph RAG,
/// entity extraction, RRF merging. Providers own: vector storage and content retrieval.
#[async_trait]
@@ -26,7 +21,7 @@ pub trait RagProvider: Send + Sync {
///
/// **Ordering contract:** implementations MUST return results in the same
/// relative order as the input `ids` slice. `hybrid_search` passes an
/// RRF-ranked list and feeds the result straight to the LLM — a provider
/// RRF-ranked list and feeds the result straight to the LLM. A provider
/// that returns rows in storage order (e.g. Qdrant `get_points`, DuckDB
/// `WHERE id IN (...)`) would silently discard the ranking. Implementations
/// that query an unordered backend must re-sort by input position before
@@ -43,34 +38,28 @@ pub trait RagProvider: Send + Sync {
/// Called once at the end of every sync_documents pass.
///
/// `full_rebuild` mirrors `sync_documents`' `refresh` parameter:
/// - `true` a full re-index (`.rebuild rag`, `--rebuild-rag`, initial build).
/// - `true`: a full re-index (`.rebuild rag`, `--rebuild-rag`, initial build).
/// Destructive strategies (wipe-then-reindex) are permitted.
/// - `false` an incremental change (`.edit rag-docs` adding/removing a file).
/// - `false`: an incremental change (`.edit rag-docs` adding/removing a file).
/// Implementations MUST NOT wipe existing state; upsert only.
///
/// The parameter is part of the signature from the outset so it is fixed
/// while there is exactly one implementor. Yaml/DuckDb ignore it
/// while there is exactly one implementor. Yaml/DuckDb ignore it,
/// rebuilding their local state wholesale is fast and always correct.
/// Only a remote provider is destructive enough to care.
///
/// YamlProvider: rebuilds HNSW + content map from data.vectors/files.
/// DuckDbProvider: writes new rows to DuckDB, deletes removed rows.
/// QdrantProvider: no-op while attach-only — remote data is unchanged.
async fn rebuild_indexes(&mut self, data: &RagData, full_rebuild: bool) -> Result<()>;
/// Keyword / full-text search. Returns (DocumentId, BM25-style score) sorted desc.
///
/// Default impl returns `Ok(vec![])` — callers fall back to `Rag.bm25` (local in-memory
/// BM25 built from `data.files`). DuckDbProvider overrides this with a native FTS query
/// (DuckDB's `fts` extension, installed once at schema-creation time).
/// Default impl returns `Ok(vec![])`. Callers fall back to `Rag.bm25` (local in-memory
/// BM25 built from `data.files`).
///
/// Callers check `has_native_keyword_search()` before deciding which path to take:
/// - true → call this method; skip `Rag.bm25`
/// - false → call `Rag.keyword_search()` which uses `Rag.bm25` (sync, infallible)
///
/// YamlProvider and QdrantProvider do NOT override this (return empty).
async fn keyword_search(&self, query: &str, top_k: usize) -> Result<Vec<(DocumentId, f32)>> {
let _ = (query, top_k);
Ok(vec![])
}
+106 -138
View File
@@ -1,15 +1,26 @@
use crate::rag::provider::RagProvider;
use crate::rag::{DocumentId, RagData};
use std::collections::HashMap;
use anyhow::{Context, Result, anyhow, bail};
use async_trait::async_trait;
use duckdb::Connection;
use duckdb::types::Value;
use indexmap::IndexMap;
use log::warn;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, MutexGuard};
/// Serializes `INSTALL` across every thread in this process. DuckDB installs an
/// extension by downloading it to a temp file and then MOVING that file into
/// `~/.duckdb/extensions/...`. Two threads installing the same extension at once
/// both perform that move; on Windows the loser's move targets a file the winner
/// already holds open and fails with "Access is denied", where POSIX would let the
/// replacement through. Guards nothing but the install step, so it is never held
/// across a `DuckDbProvider::conn` guard and cannot invert lock order.
static INSTALL_LOCK: Mutex<()> = Mutex::new(());
/// Derive the DuckDB sidecar path from a RAG's YAML path: `docs.yaml` -> `docs.duckdb`.
pub(crate) fn duckdb_path_from_yaml(yaml_path: &Path) -> PathBuf {
yaml_path.with_extension("duckdb")
@@ -44,11 +55,12 @@ impl DuckDbProvider {
// "Setting with name ... is not in the catalog, but it exists in the vss
// extension". Omitting it entirely makes CREATE INDEX ... USING HNSW on a
// file-backed database fail with "HNSW index persistence is not yet supported
// by default". LOAD vss -> LOAD fts -> SET -> CREATE INDEX.
// by default". ensure vss (installing it if missing) -> ensure fts -> SET ->
// CREATE INDEX.
Self::ensure_extension(&conn, "vss")?;
Self::ensure_extension(&conn, "fts")?;
conn.execute_batch(&format!(
"LOAD vss;
LOAD fts;
SET hnsw_enable_experimental_persistence = true;
"SET hnsw_enable_experimental_persistence = true;
CREATE TABLE IF NOT EXISTS vectors (
doc_id UBIGINT PRIMARY KEY,
embedding FLOAT[{dim}]
@@ -73,6 +85,42 @@ impl DuckDbProvider {
})
}
/// Make a DuckDB extension available on `conn`, installing it if this machine does
/// not have it yet. `LOAD` is attempted first so an extension that is already
/// installed costs nothing and never touches the network; `INSTALL` is only reached
/// once, on a machine seeing the extension for the first time.
///
/// `LOAD` is per-connection and so runs on every connection; only `INSTALL` is
/// serialized, and the second `LOAD` under the lock is what keeps it to one
/// install. Without that re-check, every thread that queued behind the winner
/// would still run a redundant `INSTALL` and re-trigger the same file move.
fn ensure_extension(conn: &Connection, name: &str) -> Result<()> {
if conn.execute_batch(&format!("LOAD {name};")).is_ok() {
return Ok(());
}
// A poisoned lock means some other thread panicked mid-install; the lock owns
// no state to corrupt, so recover rather than failing every later open.
let _install_guard = INSTALL_LOCK.lock().unwrap_or_else(|e| e.into_inner());
// Re-check now that we hold the lock: whoever held it before us may already
// have installed the extension, in which case this `LOAD` finds it on disk.
if conn.execute_batch(&format!("LOAD {name};")).is_ok() {
return Ok(());
}
conn.execute_batch(&format!("INSTALL {name};"))
.with_context(|| {
format!(
"Failed to install the DuckDB `{name}` extension. The duckdb RAG driver \
needs it, and downloading it needs network access the first time. If this \
machine is offline, connect once and retry, or run `INSTALL {name};` \
yourself from a DuckDB shell."
)
})?;
conn.execute_batch(&format!("LOAD {name};"))
.with_context(|| {
format!("Failed to load the DuckDB `{name}` extension after installing it.")
})
}
/// Read all `(doc_id, embedding)` pairs so `create()` can hydrate `data.vectors`
/// from disk. This is what makes the next incremental sync non-destructive, and is
/// mandatory rather than an optimization.
@@ -90,9 +138,6 @@ impl DuckDbProvider {
pub(crate) fn read_all_vectors(&self) -> Result<IndexMap<DocumentId, Vec<f32>>> {
let conn = self.lock_conn()?;
let mut stmt = conn.prepare("SELECT doc_id, embedding FROM vectors")?;
// Collect into a Result, NOT a filter_map. `.filter_map(|r| r.ok())` here would
// turn a systematic decode failure (e.g. a schema written by a different duckdb
// version) into a silently short map, indistinguishable from an empty store.
let raw: Vec<(u64, Vec<f32>)> = stmt
.query_map([], |row| {
let id: u64 = row.get(0)?;
@@ -100,12 +145,12 @@ impl DuckDbProvider {
// column is read as a `Value` and destructured. A FLOAT[N] column yields
// `Value::Array`, a FLOAT[] column yields `Value::List`; match both so
// the reader survives a file written under either schema.
let embedding: Vec<f32> = match row.get::<_, duckdb::types::Value>(1)? {
duckdb::types::Value::Array(vals) | duckdb::types::Value::List(vals) => vals
let embedding: Vec<f32> = match row.get::<_, Value>(1)? {
Value::Array(vals) | Value::List(vals) => vals
.into_iter()
.map(|v| match v {
duckdb::types::Value::Float(f) => f,
duckdb::types::Value::Double(d) => d as f32,
Value::Float(f) => f,
Value::Double(d) => d as f32,
_ => f32::NAN,
})
.collect(),
@@ -138,6 +183,7 @@ impl DuckDbProvider {
}
out.insert(DocumentId(id as usize), embedding);
}
Ok(out)
}
@@ -146,11 +192,10 @@ impl DuckDbProvider {
///
/// A schema-existence check alone is NOT sufficient. After
/// `CREATE OR REPLACE TABLE documents`, the `fts_main_documents` schema still
/// exists and `match_bm25` still SUCCEEDS but returns zero rows for every term.
/// exists and `match_bm25` still SUCCEEDS, but returns zero rows for every term.
/// The index is silently dead. The probe therefore asserts that a KNOWN row comes
/// back rather than merely that the call did not error.
fn probe_fts_index(conn: &Connection) -> bool {
// 1. Structural check — cheap, and short-circuits a never-built index.
let schema_exists = conn
.query_row(
"SELECT COUNT(*) FROM duckdb_schemas() WHERE schema_name = 'fts_main_documents'",
@@ -162,11 +207,6 @@ impl DuckDbProvider {
if !schema_exists {
return false;
}
// 2. Liveness check — pull a real token out of a real row and confirm the index
// scores that same row. A live index returns >= 1; a stale one returns 0
// without erroring. An empty `documents` table cannot be probed, and
// reporting false is correct: there is nothing to keyword-search, and the
// next rebuild_indexes sets the flag directly.
conn.query_row(
"SELECT COUNT(*) FROM documents d
WHERE fts_main_documents.match_bm25(
@@ -204,7 +244,6 @@ impl RagProvider for DuckDbProvider {
top_k: usize,
min_score: f32,
) -> Result<Vec<(DocumentId, f32)>> {
// Validate before building the SQL literal — a NaN would produce malformed SQL.
if embedding.iter().any(|f| !f.is_finite()) {
bail!("Query embedding contains a non-finite value (NaN or infinity)");
}
@@ -230,11 +269,11 @@ impl RagProvider for DuckDbProvider {
let id: u64 = row.get(0)?;
// `array_cosine_distance` on a FLOAT[N] column returns FLOAT (f32), NOT
// DOUBLE. Reading it as f64 raises InvalidColumnType INSIDE the closure,
// which a bare `.filter_map(|r| r.ok())` would silently discard
// which a bare `.filter_map(|r| r.ok())` would silently discard,
// yielding ZERO results with no error and no log line.
let distance: f32 = match row.get::<_, duckdb::types::Value>(1)? {
duckdb::types::Value::Float(f) => f,
duckdb::types::Value::Double(d) => d as f32,
let distance: f32 = match row.get::<_, Value>(1)? {
Value::Float(f) => f,
Value::Double(d) => d as f32,
other => {
warn!("unexpected distance type from DuckDB: {other:?}");
return Err(duckdb::Error::InvalidQuery);
@@ -253,6 +292,7 @@ impl RagProvider for DuckDbProvider {
})
.filter(|(_, score)| *score > min_score)
.collect();
Ok(results)
}
@@ -264,10 +304,7 @@ impl RagProvider for DuckDbProvider {
let placeholders = ids.iter().map(|_| "?").collect::<Vec<_>>().join(", ");
let sql =
format!("SELECT doc_id, page_content FROM documents WHERE doc_id IN ({placeholders})");
let params: Vec<duckdb::types::Value> = ids
.iter()
.map(|id| duckdb::types::Value::UBigInt(id.0 as u64))
.collect();
let params: Vec<Value> = ids.iter().map(|id| Value::UBigInt(id.0 as u64)).collect();
let mut stmt = conn.prepare(&sql)?;
let mut rows: Vec<(DocumentId, String)> = stmt
.query_map(duckdb::params_from_iter(params.iter()), |row| {
@@ -283,22 +320,21 @@ impl RagProvider for DuckDbProvider {
}
})
.collect();
// Ordering contract: `WHERE doc_id IN (...)` returns rows in storage order, NOT
// in the order of `ids`. Since `ids` is the RRF-ranked list, returning storage
// order would silently discard the ranking. Re-sort by input position.
let position: std::collections::HashMap<DocumentId, usize> =
let position: HashMap<DocumentId, usize> =
ids.iter().enumerate().map(|(i, id)| (*id, i)).collect();
rows.sort_by_key(|(id, _)| position.get(id).copied().unwrap_or(usize::MAX));
Ok(rows)
}
async fn rebuild_indexes(&mut self, data: &RagData, _full_rebuild: bool) -> Result<()> {
// Local on-disk state — a wholesale table reset plus re-INSERT is always
// Local on-disk state. A wholesale table reset plus re-INSERT is always
// correct, so the incremental/full distinction is ignored.
//
// 🔴 ANTI-WIPE GUARD. The CREATE OR REPLACE TABLE below writes exactly what
// `data.vectors` holds. An empty map on a RAG that HAS indexed files, against a
// store that ALREADY holds vectors, is always a bug a failed/skipped
// store that ALREADY holds vectors, is always a bug; a failed/skipped
// hydration, or a caller that emptied the live map. Refuse rather than commit
// the loss. Every legitimate empty-vector rebuild also has an empty `files`
// (a fresh RAG; the zero-document tests), so this cannot fire on a correct call.
@@ -307,7 +343,7 @@ impl RagProvider for DuckDbProvider {
if data.vectors.is_empty() && !data.files.is_empty() {
let existing: i64 = {
// Scoped: the guard MUST be dropped before `lock_conn()` is taken again
// below. `Mutex` is not reentrant holding both self-deadlocks at
// below. `Mutex` is not reentrant; holding both self-deadlocks at
// runtime, with no compile error.
let conn = self.lock_conn()?;
conn.query_row("SELECT count(*) FROM vectors", [], |r| r.get(0))
@@ -336,7 +372,6 @@ impl RagProvider for DuckDbProvider {
}
}
let dim = self.dim;
// `Connection::transaction()` takes `&mut self`, so this binding must be `mut`.
let mut conn = self.lock_conn()?;
let tx = conn
.transaction()
@@ -362,7 +397,7 @@ impl RagProvider for DuckDbProvider {
{
// Plain INSERT: tables are empty after CREATE OR REPLACE, so no PK conflict.
// The embedding is bound as TEXT and cast in SQL — the duckdb crate's
// The embedding is bound as TEXT and cast in SQL. The duckdb crate's
// `bind_parameter` has no arm for List/Array, so binding a vector directly
// fails at runtime with "binding List parameters is not yet supported".
let mut vstmt = tx.prepare(&format!(
@@ -372,12 +407,12 @@ impl RagProvider for DuckDbProvider {
tx.prepare("INSERT INTO documents (doc_id, page_content) VALUES (?, ?)")?;
// 🔴 TWO INDEPENDENT LOOPS. `documents` is keyed on `files`, `vectors` on
// `vectors` — they are DIFFERENT key sets and neither is a subset of the
// `vectors`. They are DIFFERENT key sets and neither is a subset of the
// other. Do not merge these into one loop over `&data.vectors` gated on a
// lookup into `files`: that produces keys(documents) ⊆ keys(vectors), and
// every id in `files \ vectors` (which `RagData::add`'s zip truncation
// really does produce) becomes a live, rankable id from BM25 and
// graph_search that resolves to nothing no error, no log line.
// graph_search that resolves to nothing; no error, no log line.
for (id, doc) in data.iter_documents() {
dstmt.execute(duckdb::params![id.0 as u64, doc.page_content.as_str()])?;
}
@@ -415,7 +450,7 @@ impl RagProvider for DuckDbProvider {
// Rebuild the FTS index (must be outside the transaction). This rebuild is
// MANDATORY on every pass, not an optimization: CREATE OR REPLACE TABLE above
// leaves the old fts_main_documents schema in place but DEAD match_bm25 keeps
// leaves the old fts_main_documents schema in place but DEAD; match_bm25 keeps
// succeeding while returning zero rows for every term.
// GUARDED on a non-empty table: FTS cannot index an empty table, and
// rebuild_indexes is legitimately called with zero documents (a fresh RAG, and
@@ -434,7 +469,7 @@ impl RagProvider for DuckDbProvider {
// Live only if an index was actually built. With zero documents there is no FTS
// index, so `has_native_keyword_search()` must stay false and hybrid_search must
// fall back to local BM25 which is also empty, and therefore correct.
// fall back to local BM25, which is also empty, and therefore correct.
self.fts_ready.store(doc_count > 0, Ordering::Relaxed);
Ok(())
@@ -455,11 +490,11 @@ impl RagProvider for DuckDbProvider {
let id: u64 = row.get(0)?;
// Same hazard as vector_search's distance column: guessing the width
// wrong raises InvalidColumnType INSIDE the closure, which a bare
// `.filter_map(|r| r.ok())` silently discards yielding ZERO keyword
// `.filter_map(|r| r.ok())` silently discards, yielding ZERO keyword
// hits, indistinguishable from "the query matched nothing".
let score: f32 = match row.get::<_, duckdb::types::Value>(1)? {
duckdb::types::Value::Double(d) => d as f32,
duckdb::types::Value::Float(f) => f,
let score: f32 = match row.get::<_, Value>(1)? {
Value::Double(d) => d as f32,
Value::Float(f) => f,
other => {
warn!("unexpected match_bm25 score type from DuckDB: {other:?}");
return Err(duckdb::Error::InvalidQuery);
@@ -497,7 +532,7 @@ impl RagProvider for DuckDbProvider {
// This means DuckDbProvider clones are NOT independent snapshots: state lives on
// disk and a rebuild through one handle is immediately visible to the other.
// That is unavoidable for any on-disk store and is handled by the discipline
// documented on `Rag`'s Clone impl the pre-clone instance must be discarded.
// documented on `Rag`'s Clone impl, the pre-clone instance must be discarded.
Box::new(DuckDbProvider {
path: self.path.clone(),
conn: Arc::clone(&self.conn),
@@ -510,44 +545,37 @@ impl RagProvider for DuckDbProvider {
#[cfg(test)]
mod tests {
use super::*;
// Trait methods are only callable with the trait in scope.
use crate::rag::provider::RagProvider;
// `RagFile` / `RagDocument` have private fields; struct-literal construction
// compiles only because this module is a descendant of `crate::rag`. They are
// deliberately not in the non-test `use` block — unused there, and `--deny warnings`
// rejects that.
use crate::rag::{RagDocument, RagFile};
use std::time::{SystemTime, UNIX_EPOCH};
use std::{env, fs};
/// Unique temp path per test. `tempfile` is not a dev-dependency; this mirrors the
/// house pattern used elsewhere in the tree.
struct TempDb {
path: PathBuf,
}
impl TempDb {
fn new(tag: &str) -> Self {
let unique = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
let unique = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos();
let path = std::env::temp_dir().join(format!("coyote-duckdb-{tag}-{unique}.duckdb"));
let path = env::temp_dir().join(format!("coyote-duckdb-{tag}-{unique}.duckdb"));
Self { path }
}
}
impl Drop for TempDb {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.path);
// DuckDB writes a `<path>.wal` sidecar; remove it too or /tmp accumulates
// one per test run.
let _ = std::fs::remove_file(self.path.with_extension("duckdb.wal"));
let _ = fs::remove_file(&self.path);
let _ = fs::remove_file(self.path.with_extension("duckdb.wal"));
}
}
/// ⚠️ NOTE THE EMPTY `files`. This fixture describes a RAG with ZERO documents.
/// Since `rebuild_indexes` populates the `documents` table from
/// `data.iter_documents()` (keyed on `files`), any test built on this helper alone
/// exercises the `vectors` table and NOTHING ELSE — no `documents` rows, no FTS
/// exercises the `vectors` table and NOTHING ELSE. No `documents` rows, no FTS
/// index. That is correct for the rebuild tests below, and is exactly why the
/// FTS-flag test and the `fetch_content` tests use `populated_rag_data()` instead.
/// Do not "simplify" them back onto this helper.
@@ -563,7 +591,7 @@ mod tests {
}
}
/// Two files, one document each the minimum fixture that produces `documents`
/// Two files, one document each; the minimum fixture that produces `documents`
/// rows.
fn populated_rag_data() -> RagData {
let mut data = minimal_rag_data();
@@ -598,12 +626,14 @@ mod tests {
let db = TempDb::new("schema");
let provider = DuckDbProvider::open(&db.path, 3).unwrap();
let conn = provider.conn.lock().unwrap();
let v: i64 = conn
.query_row("SELECT COUNT(*) FROM vectors", [], |r| r.get(0))
.unwrap();
let d: i64 = conn
.query_row("SELECT COUNT(*) FROM documents", [], |r| r.get(0))
.unwrap();
assert_eq!(v, 0);
assert_eq!(d, 0);
}
@@ -622,10 +652,12 @@ mod tests {
)
.unwrap();
}
let results = provider
.vector_search(&[0.1, 0.2, 0.3], 5, 0.0)
.await
.unwrap();
assert_eq!(results.len(), 1);
assert_eq!(results[0].0.0, 0);
assert!(results[0].1 > 0.99);
@@ -643,7 +675,9 @@ mod tests {
)
.unwrap();
}
let results = provider.fetch_content(&[DocumentId(42)]).await.unwrap();
assert_eq!(results.len(), 1);
assert_eq!(results[0].1, "hello world");
}
@@ -654,19 +688,15 @@ mod tests {
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
let data = minimal_rag_data();
provider.rebuild_indexes(&data, true).await.unwrap();
let results = provider
.vector_search(&[0.1, 0.2, 0.3], 5, 0.0)
.await
.unwrap();
assert!(results.is_empty()); // no vectors in empty data
assert!(results.is_empty());
}
/// Rebuilding TWICE must succeed.
///
/// The first rebuild always passes because the tables start empty. The bug this
/// guards against — `DELETE FROM` plus re-INSERT of the same PKs inside one
/// transaction — only fires on the SECOND rebuild, with "Duplicate key ... violates
/// primary key constraint". A single-rebuild test cannot catch it.
#[tokio::test]
async fn rebuild_indexes_is_idempotent_across_two_passes() {
let db = TempDb::new("idempotent");
@@ -676,7 +706,7 @@ mod tests {
data.vectors.insert(DocumentId(1), vec![0.4, 0.5, 0.6]);
provider.rebuild_indexes(&data, true).await.unwrap();
// Second pass over the SAME doc_ids — this is the assertion that matters.
provider
.rebuild_indexes(&data, true)
.await
@@ -689,27 +719,16 @@ mod tests {
assert_eq!(count, 2, "rebuild must replace rows, not duplicate them");
}
/// Guards the incremental data-loss bug: `sync_documents` hash-skips unchanged
/// files, so on an incremental pass `data.vectors` holds ONLY the newly embedded
/// chunks. If the live map is ever emptied, the `CREATE OR REPLACE TABLE` in
/// rebuild_indexes writes only those, destroying every previously indexed vector.
/// Hydration via `read_all_vectors()` is what prevents it.
///
/// This simulates the real restart-then-edit sequence: build, drop the in-memory
/// map, rehydrate from disk, add one vector, rebuild incrementally.
#[tokio::test]
async fn rebuild_indexes_preserves_prior_vectors_on_incremental_pass() {
let db = TempDb::new("incremental");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
// Pass 1 — initial full build with two vectors.
let mut data = minimal_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
data.vectors.insert(DocumentId(1), vec![0.4, 0.5, 0.6]);
provider.rebuild_indexes(&data, true).await.unwrap();
// Simulate a restart: the YAML on disk carries NO vectors, so a fresh RagData
// starts empty. create() hydrates it back from the sidecar.
let mut reloaded = minimal_rag_data();
assert!(
reloaded.vectors.is_empty(),
@@ -722,7 +741,6 @@ mod tests {
"hydration must restore both vectors"
);
// Pass 2 — incremental add of ONE new chunk.
reloaded.vectors.insert(DocumentId(2), vec![0.7, 0.8, 0.9]);
provider.rebuild_indexes(&reloaded, false).await.unwrap();
@@ -737,14 +755,6 @@ mod tests {
);
}
/// `has_native_keyword_search()` must be false before any FTS index exists,
/// otherwise hybrid_search routes into keyword_search and the `?` turns a DuckDB
/// catalog error into a total query failure.
///
/// 🔴 THE FIXTURE MUST CARRY DOCUMENTS. `rebuild_indexes` builds the FTS index only
/// when the `documents` table is non-empty, and that table is filled from
/// `data.iter_documents()` (keyed on `files`), never from `vectors`. With `files`
/// empty the pragma is skipped and `fts_ready` stores `false`.
#[tokio::test]
async fn keyword_search_is_not_advertised_before_first_rebuild() {
let db = TempDb::new("ftsflag");
@@ -754,13 +764,10 @@ mod tests {
"a freshly opened DB has no FTS index yet"
);
// 2 files × 1 document → 2 `documents` rows → doc_count > 0 → pragma runs.
let mut data = populated_rag_data();
data.vectors.insert(DocumentId(0), vec![0.1, 0.2, 0.3]);
provider.rebuild_indexes(&data, true).await.unwrap();
{
// Pin the precondition explicitly. If this ever reads 0 the assertion below
// is vacuous and the FTS hazard is unguarded again.
let conn = provider.conn.lock().unwrap();
let docs: i64 = conn
.query_row("SELECT COUNT(*) FROM documents", [], |r| r.get(0))
@@ -773,23 +780,11 @@ mod tests {
);
}
/// The `documents` table is keyed on `files`, NOT on `vectors`.
///
/// `fetch_content` is the sink for BOTH the BM25 and graph_search paths, and both
/// enumerate `files` via `iter_documents()`. If `rebuild_indexes` only inserts a
/// documents row for ids that also appear in `data.vectors`, every id in
/// `files \ vectors` resolves to nothing: healthy BM25 scores, healthy graph hits,
/// zero results, no error.
///
/// The incremental test CANNOT catch this — its fixture has an empty `files`, so the
/// documents table is empty in every assertion either way.
#[tokio::test]
async fn fetch_content_resolves_documents_without_vectors() {
let db = TempDb::new("docsnovec");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
// Two files, one document each — but a vector for ONLY THE FIRST. This is the
// `RagData::add` zip-truncation shape.
let mut data = populated_rag_data();
let id_a = DocumentId::new(0, 0);
let id_b = DocumentId::new(1, 0);
@@ -812,13 +807,6 @@ mod tests {
assert_eq!(got[1].1, "beta keyword");
}
/// `fetch_content` must return rows in INPUT order, not storage order.
///
/// `YamlProvider` satisfies this contract structurally — it maps over `ids` — so the
/// existing yaml test proves nothing about DuckDB. DuckDB's `WHERE doc_id IN (...)`
/// returns storage order and relies on an explicit positional re-sort, which is what
/// can actually regress. `ids` is the RRF-ranked list, so losing the order silently
/// discards the ranking while still returning the right documents.
#[tokio::test]
async fn duckdb_fetch_content_preserves_input_order() {
let db = TempDb::new("order");
@@ -826,11 +814,9 @@ mod tests {
let data = populated_rag_data();
provider.rebuild_indexes(&data, true).await.unwrap();
let id_a = DocumentId::new(0, 0); // inserted first → storage order 0
let id_b = DocumentId::new(1, 0); // inserted second → storage order 1
let id_a = DocumentId::new(0, 0);
let id_b = DocumentId::new(1, 0);
// REVERSED relative to storage order. Without the positional re-sort this
// returns ["alpha keyword", "beta keyword"] and the assertion fails.
let got = provider.fetch_content(&[id_b, id_a]).await.unwrap();
assert_eq!(got.len(), 2);
assert_eq!(got[0].0, id_b, "input order must win over storage order");
@@ -839,19 +825,11 @@ mod tests {
assert_eq!(got[1].1, "alpha keyword");
}
/// The anti-wipe guard.
///
/// `rebuild_indexes` does `CREATE OR REPLACE TABLE` and writes exactly what
/// `data.vectors` holds. An empty map on a RAG that HAS indexed files, against a
/// store that already holds vectors, is always a bug (failed hydration, or a caller
/// that emptied the live map) and must be refused rather than committed. The loss
/// would otherwise be permanent: nothing re-embeds it back.
#[tokio::test]
async fn rebuild_indexes_refuses_to_wipe_when_vectors_are_empty_but_files_are_not() {
let db = TempDb::new("nowipe");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
// A healthy store: two files with documents, two vectors.
let mut data = populated_rag_data();
data.vectors
.insert(DocumentId::new(0, 0), vec![0.1, 0.2, 0.3]);
@@ -859,8 +837,7 @@ mod tests {
.insert(DocumentId::new(1, 0), vec![0.4, 0.5, 0.6]);
provider.rebuild_indexes(&data, true).await.unwrap();
// Now the failure state: vectors lost in memory, files intact.
let broken = populated_rag_data(); // same files, NO vectors
let broken = populated_rag_data();
assert!(
broken.vectors.is_empty() && !broken.files.is_empty(),
"precondition"
@@ -872,7 +849,6 @@ mod tests {
"got: {err}"
);
// The store must be untouched — this is the whole point.
let conn = provider.conn.lock().unwrap();
let count: i64 = conn
.query_row("SELECT COUNT(*) FROM vectors", [], |r| r.get(0))
@@ -883,31 +859,23 @@ mod tests {
);
}
/// The guard must NOT fire on a legitimately empty RAG — otherwise a fresh
/// `Rag::init()` cannot complete. Empty `vectors` AND empty `files` is fine.
#[tokio::test]
async fn rebuild_indexes_allows_empty_vectors_when_files_are_also_empty() {
let db = TempDb::new("emptyok");
let mut provider = DuckDbProvider::open(&db.path, 3).unwrap();
let data = minimal_rag_data(); // no files, no vectors
let data = minimal_rag_data();
provider
.rebuild_indexes(&data, true)
.await
.expect("a fresh RAG with nothing indexed must rebuild cleanly");
}
/// `duplicate()` must clone the Arc, NOT call `open()` again.
///
/// This asserts SHARED state, which is the actual contract. It deliberately does NOT
/// use `fetch_content(&[])` — that early-returns before ever touching the
/// connection, so it would pass even against a broken double-opening implementation.
#[tokio::test]
async fn duplicate_shares_the_same_connection() {
let db = TempDb::new("dup");
let provider = DuckDbProvider::open(&db.path, 3).unwrap();
let dup = provider.duplicate(&minimal_rag_data());
// Write through the ORIGINAL...
{
let conn = provider.conn.lock().unwrap();
conn.execute(
@@ -916,8 +884,9 @@ mod tests {
)
.unwrap();
}
// ...and read it back through the DUPLICATE. Only possible if they share state.
let via_dup = dup.fetch_content(&[DocumentId(7)]).await.unwrap();
assert_eq!(
via_dup.len(),
1,
@@ -926,8 +895,6 @@ mod tests {
assert_eq!(via_dup[0].1, "shared row");
}
/// A decode failure must abort hydration rather than yield a thinned map: a short
/// map is what the next CREATE OR REPLACE commits as the new truth.
#[tokio::test]
async fn read_all_vectors_rejects_non_finite_embeddings() {
let db = TempDb::new("nonfinite");
@@ -959,6 +926,7 @@ mod tests {
#[test]
fn duckdb_path_from_yaml_swaps_extension() {
let p = duckdb_path_from_yaml(Path::new("/tmp/rags/docs.yaml"));
assert_eq!(p, PathBuf::from("/tmp/rags/docs.duckdb"));
}
}
-7
View File
@@ -1,15 +1,8 @@
mod yaml;
// Use `self::` on every re-export in this file. Once a `mod duckdb;` sits here
// alongside a dependency on the `duckdb` CRATE, a bare `pub use duckdb::...`
// is ambiguous (E0659) — `use` paths resolve against both this module's items
// and the extern prelude, and `use` declarations may not shadow.
pub use self::yaml::YamlProvider;
mod duckdb;
pub use self::duckdb::DuckDbProvider;
// `create()` in rag/mod.rs derives the sidecar path through this. It is `pub(crate)`
// in providers/duckdb.rs, and the re-export must be `pub(crate)` too — a `pub use` of
// a `pub(crate)` item is E0364/E0365.
pub(crate) use self::duckdb::duckdb_path_from_yaml;
mod qdrant;
+47 -46
View File
@@ -3,6 +3,9 @@ use crate::rag::{DocumentId, RagData};
use anyhow::{Context, Result, bail};
use async_trait::async_trait;
use reqwest::header::{HeaderMap, HeaderValue};
use reqwest::{Client, Response, StatusCode};
use serde_json::Value;
use std::collections::HashMap;
/// Render Qdrant's error envelope into a human-readable message.
@@ -15,22 +18,18 @@ use std::collections::HashMap;
/// * routing-level 404s (a wrong HTTP verb) return an EMPTY body with no JSON at
/// all, which without the length check surfaces as "EOF while parsing a value"
/// instead of the actual 404.
fn format_error_body(status: reqwest::StatusCode, body: &str) -> String {
fn format_error_body(status: StatusCode, body: &str) -> String {
if body.is_empty() {
return format!("HTTP {status} (empty body — check the HTTP verb and path)");
}
serde_json::from_str::<serde_json::Value>(body)
serde_json::from_str::<Value>(body)
.ok()
.and_then(|v| v["status"]["error"].as_str().map(str::to_string))
.unwrap_or_else(|| format!("HTTP {status}: {body}"))
}
/// Read the vector dimension out of a parsed `GET /collections/{name}` response.
///
/// Unnamed collections put `size` directly under `vectors`; named ones nest it
/// under the vector's name. Both shapes occur in the wild, so try the flat one
/// first and fall back to the first named entry.
fn vector_dimension_from_collection(body: &serde_json::Value) -> Result<u64> {
fn vector_dimension_from_collection(body: &Value) -> Result<u64> {
let params = &body["result"]["config"]["params"];
params["vectors"]["size"]
.as_u64()
@@ -47,12 +46,12 @@ fn vector_dimension_from_collection(body: &serde_json::Value) -> Result<u64> {
/// (multi-vector) collection.
///
/// `vector_search` posts an unnamed vector, which a named-vector collection
/// rejects with HTTP 400 on every query so attaching one yields a RAG that is
/// rejects with HTTP 400 on every query, so attaching one yields a RAG that is
/// silently 100% broken. A named collection holding a SINGLE vector is
/// structurally a map, identical in kind to the multi-named case, and rejects
/// the same way; testing for a numeric `size` directly under `vectors` catches
/// it, whereas counting keys (`len() > 1`) would wrongly accept it.
fn is_multi_vector_config(body: &serde_json::Value) -> bool {
fn is_multi_vector_config(body: &Value) -> bool {
body["result"]["config"]["params"]["vectors"]["size"]
.as_u64()
.is_none()
@@ -63,33 +62,27 @@ fn is_multi_vector_config(body: &serde_json::Value) -> bool {
/// Attach-only: this provider never writes to the remote collection. Coyote does
/// not own the data, and `rebuild_indexes` refuses rather than pretending to.
pub struct QdrantProvider {
/// `reqwest::Client` is Arc-backed, so `clone()` is O(1) and shares both the
/// connection pool and the `api-key` default header injected at build time.
client: reqwest::Client,
/// Includes the scheme, e.g. `http://qdrant.example.com:6333`.
client: Client,
base_url: String,
collection: String,
}
impl QdrantProvider {
/// The resolved API key is injected as a default header here and is
/// deliberately NOT stored on the struct: the plaintext value stays a local
/// of the caller and never outlives it.
fn make_client(api_key: Option<&str>) -> Result<reqwest::Client> {
let mut headers = reqwest::header::HeaderMap::new();
fn make_client(api_key: Option<&str>) -> Result<Client> {
let mut headers = HeaderMap::new();
if let Some(key) = api_key {
let mut value = reqwest::header::HeaderValue::from_str(key)
.context("api-key header value is not valid ASCII")?;
let mut value =
HeaderValue::from_str(key).context("api-key header value is not valid ASCII")?;
value.set_sensitive(true);
headers.insert("api-key", value);
}
reqwest::Client::builder()
Client::builder()
.default_headers(headers)
.build()
.context("Failed to build reqwest client")
}
fn normalize_base_url(host: &str) -> String {
pub(crate) fn normalize_base_url(host: &str) -> String {
if host.starts_with("http://") || host.starts_with("https://") {
host.to_string()
} else {
@@ -97,7 +90,7 @@ impl QdrantProvider {
}
}
async fn error_message(resp: reqwest::Response) -> String {
async fn error_message(resp: Response) -> String {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
format_error_body(status, &body)
@@ -109,7 +102,7 @@ impl QdrantProvider {
host: &str,
collection: &str,
api_key: Option<&str>,
) -> Result<serde_json::Value> {
) -> Result<Value> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(api_key)?;
let resp = client
@@ -123,13 +116,13 @@ impl QdrantProvider {
Self::error_message(resp).await
);
}
Ok(resp.json().await?)
}
pub async fn new(host: &str, collection: &str, api_key: Option<&str>) -> Result<Self> {
let base_url = Self::normalize_base_url(host);
let client = Self::make_client(api_key)?;
// Preflight: confirm the collection exists and we may read it.
let resp = client
.get(format!("{base_url}/collections/{collection}"))
.send()
@@ -141,6 +134,7 @@ impl QdrantProvider {
Self::error_message(resp).await
);
}
Ok(Self {
client,
base_url,
@@ -162,13 +156,15 @@ impl QdrantProvider {
Self::error_message(resp).await
);
}
let body: serde_json::Value = resp.json().await?;
let body: Value = resp.json().await?;
let names = body["result"]["collections"]
.as_array()
.context("Unexpected /collections response shape")?
.iter()
.filter_map(|v| v["name"].as_str().map(str::to_string))
.collect();
Ok(names)
}
@@ -178,6 +174,7 @@ impl QdrantProvider {
api_key: Option<&str>,
) -> Result<u64> {
let body = Self::fetch_collection(host, collection, api_key).await?;
vector_dimension_from_collection(&body)
}
@@ -187,11 +184,10 @@ impl QdrantProvider {
api_key: Option<&str>,
) -> Result<bool> {
let body = Self::fetch_collection(host, collection, api_key).await?;
Ok(is_multi_vector_config(&body))
}
/// Peek at one point to learn how its ID is typed. Returns the raw JSON
/// rendering, so a string ID comes back quoted and an integer one bare.
pub async fn sample_point_id(
host: &str,
collection: &str,
@@ -201,23 +197,27 @@ impl QdrantProvider {
let client = Self::make_client(api_key)?;
let url = format!("{base_url}/collections/{collection}/points/scroll");
let body = serde_json::json!({ "limit": 1, "with_payload": false });
let resp = client
.post(&url)
.json(&body)
.send()
.await
.with_context(|| format!("Failed to connect to {host}"))?;
if !resp.status().is_success() {
bail!(
"Failed to sample a point from '{collection}': {}",
Self::error_message(resp).await
);
}
let data: serde_json::Value = resp.json().await?;
let data: Value = resp.json().await?;
let id_val = data["result"]["points"]
.as_array()
.and_then(|pts| pts.first())
.map(|pt| pt["id"].to_string());
Ok(id_val)
}
}
@@ -251,7 +251,7 @@ impl RagProvider for QdrantProvider {
Self::error_message(resp).await
);
}
let data: serde_json::Value = resp.json().await?;
let data: Value = resp.json().await?;
let results = data["result"]
.as_array()
.context("Unexpected /points/search response shape")?
@@ -266,6 +266,7 @@ impl RagProvider for QdrantProvider {
})
.filter(|(_, score)| *score > min_score)
.collect();
Ok(results)
}
@@ -279,7 +280,9 @@ impl RagProvider for QdrantProvider {
"ids": id_list,
"with_payload": true,
});
let resp = self.client.post(&url).json(&body).send().await?;
if !resp.status().is_success() {
bail!(
"Qdrant point fetch on '{}' failed: {}",
@@ -287,7 +290,7 @@ impl RagProvider for QdrantProvider {
Self::error_message(resp).await
);
}
let data: serde_json::Value = resp.json().await?;
let data: Value = resp.json().await?;
let mut rows: Vec<(DocumentId, String)> = data["result"]
.as_array()
.context("Unexpected /points response shape")?
@@ -303,13 +306,14 @@ impl RagProvider for QdrantProvider {
let position: HashMap<DocumentId, usize> =
ids.iter().enumerate().map(|(i, id)| (*id, i)).collect();
rows.sort_by_key(|(id, _)| position.get(id).copied().unwrap_or(usize::MAX));
Ok(rows)
}
async fn rebuild_indexes(&mut self, data: &RagData, _full_rebuild: bool) -> Result<()> {
// Both arms refuse. A silent `Ok(())` would make `.rebuild rag` and
// `.edit rag-docs` look like they worked while writing nothing to the
// remote leaving the user believing the collection was updated.
// remote, leaving the user believing the collection was updated.
if data.attached {
bail!(
"This RAG is attached to an external Qdrant collection. Coyote does not own \
@@ -340,7 +344,7 @@ mod tests {
fn error_message_reads_the_object_status_envelope() {
let body =
r#"{"status": {"error": "Wrong input: Not existing vector name error:"}, "time": 0.0}"#;
let msg = format_error_body(reqwest::StatusCode::BAD_REQUEST, body);
let msg = format_error_body(StatusCode::BAD_REQUEST, body);
assert!(msg.contains("Not existing vector name"), "got: {msg}");
assert!(
!msg.contains("EOF"),
@@ -350,15 +354,14 @@ mod tests {
#[test]
fn error_message_survives_the_string_status_and_the_empty_body() {
// Success envelope: `status` is a bare string, so the object lookup misses
// and we must fall back rather than panic or invent an error text.
let ok = format_error_body(reqwest::StatusCode::OK, r#"{"status": "ok", "time": 0.0}"#);
let ok = format_error_body(StatusCode::OK, r#"{"status": "ok", "time": 0.0}"#);
assert!(
ok.contains("200"),
"no `status.error` present → fall back to status+body: {ok}"
);
// Routing-level 404 from a wrong HTTP verb: empty body, no JSON at all.
let empty = format_error_body(reqwest::StatusCode::NOT_FOUND, "");
let empty = format_error_body(StatusCode::NOT_FOUND, "");
assert!(empty.contains("empty body"), "got: {empty}");
assert!(
empty.contains("verb"),
@@ -384,15 +387,11 @@ mod tests {
#[test]
fn is_multi_vector_rejects_the_named_single_collection() {
// The only supported shape: a single unnamed vector.
let unnamed = serde_json::json!({
"result": {"config": {"params": {"vectors": {"size": 1536, "distance": "Cosine"}}}}
});
assert!(!is_multi_vector_config(&unnamed));
// Named but SINGLE — structurally a map, and writes to it fail with
// `400 "Wrong input: Not existing vector name error:"`. A `len() > 1` check
// would wrongly accept this one; that is the bug this case exists to catch.
let named_single = serde_json::json!({
"result": {"config": {"params": {"vectors": {"text": {"size": 1536}}}}}
});
@@ -426,7 +425,7 @@ mod tests {
#[tokio::test]
async fn rebuild_indexes_refuses_for_attached_and_unattached_alike() {
let mut provider = QdrantProvider {
client: reqwest::Client::new(),
client: Client::new(),
base_url: "http://localhost:6333".to_string(),
collection: "c".to_string(),
};
@@ -459,13 +458,12 @@ mod tests {
#[tokio::test]
async fn fetch_content_short_circuits_on_an_empty_id_list() {
// No network is touched: the early return happens before any request, which
// is why this can assert against an unreachable host.
let provider = QdrantProvider {
client: reqwest::Client::new(),
client: Client::new(),
base_url: "http://127.0.0.1:1".to_string(),
collection: "c".to_string(),
};
assert!(provider.fetch_content(&[]).await.unwrap().is_empty());
}
@@ -475,6 +473,7 @@ mod tests {
let collections = QdrantProvider::list_collections("http://localhost:6333", None)
.await
.unwrap();
assert!(!collections.is_empty());
}
@@ -485,7 +484,9 @@ mod tests {
.await
.unwrap();
let embedding = vec![0.0f32; 1536];
let results = provider.vector_search(&embedding, 5, 0.0).await.unwrap();
assert!(results.len() <= 5);
}
}
+9 -34
View File
@@ -20,9 +20,6 @@ impl YamlProvider {
}
fn build_content_map(data: &RagData) -> IndexMap<DocumentId, String> {
// Keyed on `files`, NOT `vectors`: this is the exact replacement for the
// per-id document lookup it supersedes, and it must resolve every id that
// BM25 or graph_search can produce — both of which enumerate `files`.
data.iter_documents()
.map(|(id, doc)| (id, doc.page_content.clone()))
.collect()
@@ -52,12 +49,11 @@ impl RagProvider for YamlProvider {
})
})
.collect();
Ok(results)
}
async fn fetch_content(&self, ids: &[DocumentId]) -> Result<Vec<(DocumentId, String)>> {
// Iterating `ids` (not `content_map`) satisfies the trait's ordering
// contract for free — output order mirrors input order.
Ok(ids
.iter()
.filter_map(|id| self.content_map.get(id).map(|text| (*id, text.clone())))
@@ -65,10 +61,10 @@ impl RagProvider for YamlProvider {
}
async fn rebuild_indexes(&mut self, data: &RagData, _full_rebuild: bool) -> Result<()> {
// Local in-memory state — a wholesale rebuild is fast and always correct,
// so the incremental/full distinction is irrelevant here.
self.hnsw = data.build_hnsw();
self.content_map = Self::build_content_map(data);
Ok(())
}
@@ -80,14 +76,9 @@ impl RagProvider for YamlProvider {
#[cfg(test)]
mod provider_tests {
use super::*;
// `RagFile` and `RagDocument` are not used by the impl above, so they are
// imported here rather than at module scope. Both have private fields, which
// is why these tests must live in-crate rather than under `tests/`.
use crate::rag::{RagDocument, RagFile};
fn minimal_rag_data() -> RagData {
// `..Default::default()` rather than an exhaustive struct literal so that
// later additions to `RagData` do not break this helper.
RagData {
embedding_model: "text-embedding-3-small".to_string(),
chunk_size: 1024,
@@ -99,7 +90,7 @@ mod provider_tests {
}
}
/// Two files, one chunk each, with vectors the minimum needed to exercise
/// Two files, one chunk each, with vectors, the minimum needed to exercise
/// `build_content_map` and the `fetch_content` ordering contract.
/// `DocumentId::new(f, d)` packs (file_index, document_index); `RagData::add`
/// is the real insertion path but a direct literal is sufficient and avoids
@@ -143,13 +134,12 @@ mod provider_tests {
async fn yaml_provider_empty_data_returns_nothing() {
let data = minimal_rag_data();
let provider = YamlProvider::from_data(&data);
let results = provider.fetch_content(&[]).await.unwrap();
assert!(results.is_empty());
}
/// `fetch_content` MUST return results in the same relative order as the input
/// ids. The reversed-input case is the one that fails if an implementation ever
/// iterates its own map instead of `ids`.
#[tokio::test]
async fn yaml_provider_fetch_content_preserves_input_order() {
let data = populated_rag_data();
@@ -163,8 +153,8 @@ mod provider_tests {
assert_eq!(forward[0].1, "alpha");
assert_eq!(forward[1].1, "beta");
// Reversed input must produce reversed output — NOT storage order.
let reversed = provider.fetch_content(&[b, a]).await.unwrap();
assert_eq!(
reversed[0].1, "beta",
"fetch_content must honor input order"
@@ -172,8 +162,6 @@ mod provider_tests {
assert_eq!(reversed[1].1, "alpha");
}
/// A missing id is skipped, not an error, and does not disturb the order of
/// the ids that DO resolve.
#[tokio::test]
async fn yaml_provider_fetch_content_skips_missing_ids() {
let data = populated_rag_data();
@@ -189,23 +177,16 @@ mod provider_tests {
assert_eq!(out[1].1, "beta");
}
/// `YamlProvider::duplicate()` rebuilds from `data`, so the clone is a genuine
/// independent snapshot. Providers backed by a shared store deliberately are not.
#[tokio::test]
async fn yaml_provider_duplicate_returns_equivalent_content() {
// MUST be populated_rag_data(): on minimal_rag_data() both providers hold an
// EMPTY content map, so `assert_eq!(r1, r2)` compares two empty vectors and
// passes against a duplicate() that returns nothing at all.
let data = populated_rag_data();
let provider = YamlProvider::from_data(&data);
let dup = provider.duplicate(&data);
let ids = [DocumentId::new(0, 0), DocumentId::new(1, 0)];
// Query with REAL ids, not `&[]` — an empty slice is answered without ever
// touching the content map, so it would pass against a broken duplicate().
let r1 = provider.fetch_content(&ids).await.unwrap();
let r2 = dup.fetch_content(&ids).await.unwrap();
// Guard against the vacuous case: if both sides resolved nothing, the equality
// below proves nothing. Assert the fixture actually produced content first.
assert_eq!(r1.len(), 2, "fixture must resolve both documents");
assert_eq!(
r1, r2,
@@ -213,11 +194,6 @@ mod provider_tests {
);
}
/// The content store is keyed on `files`, never on `vectors`. A vector may exist
/// for an id with no backing file (a stale entry, or a file dropped mid-sync);
/// keying on `vectors` would surface such an id with empty text instead of
/// dropping it. The shared fixture only ever inserts vectors for ids that also
/// have files, so this case has to be constructed here.
#[tokio::test]
async fn yaml_provider_content_is_keyed_on_files_not_vectors() {
let mut data = populated_rag_data();
@@ -232,7 +208,6 @@ mod provider_tests {
"an id present only in `vectors` must not resolve to content"
);
// The file-backed ids still resolve, so the assertion above is not vacuous.
let real = provider
.fetch_content(&[DocumentId::new(0, 0), DocumentId::new(1, 0)])
.await
-11
View File
@@ -1749,11 +1749,6 @@ std::error::Error>> {
);
}
/// Removes CSI escape sequences so only printable content is measured.
///
/// Deliberately tolerant of malformed input: a sequence that was sliced
/// mid-escape swallows the following characters, which is precisely the
/// corruption `render_table_pads_columns_by_display_width` exists to catch.
fn strip_ansi(text: &str) -> String {
let mut out = String::with_capacity(text.len());
let mut chars = text.chars();
@@ -1780,12 +1775,6 @@ std::error::Error>> {
assert_eq!(strip_ansi("plain"), "plain");
}
/// `render_table` hands comfy-table pre-styled cells that already contain
/// ANSI escapes, and `colorize_box_chars` adds more afterwards. Column
/// widths are therefore only correct if the escapes are excluded from the
/// width calculation. When they are not, the table still renders and every
/// other assertion in this file still passes -- only the alignment silently
/// degrades -- so this is the sole guard over that behaviour.
#[test]
fn render_table_pads_columns_by_display_width() {
use unicode_width::UnicodeWidthStr;
+10 -15
View File
@@ -219,7 +219,7 @@ static REPL_COMMANDS: LazyLock<[ReplCommand; 60]> = LazyLock::new(|| {
),
ReplCommand::new(
".rag attach",
"Attach to a pre-existing external RAG (Qdrant)",
"Attach to a pre-existing external RAG",
AssertState::False(StateFlags::AGENT),
),
ReplCommand::new(
@@ -889,22 +889,17 @@ pub async fn run_repl_command(
let version = args.map(|s| s.trim().to_string());
task::spawn_blocking(move || config::run_self_update(version, false)).await??;
}
".rag" => {
// `split_first_arg` rather than `starts_with("attach ")`: the latter
// misses a bare `.rag attach`, which would silently create a RAG
// literally named "attach".
match split_first_arg(args) {
Some(("attach", rest)) => match rest {
Some(name) if !name.trim().is_empty() => {
ctx.attach_rag(name.trim()).await?;
}
_ => println!("Usage: .rag attach <name>"),
},
_ => {
ctx.use_rag(args, abort_signal.clone()).await?;
".rag" => match split_first_arg(args) {
Some(("attach", rest)) => match rest {
Some(name) if !name.trim().is_empty() => {
ctx.attach_rag(name.trim()).await?;
}
_ => println!("Usage: .rag attach <name>"),
},
_ => {
ctx.use_rag(args, abort_signal.clone()).await?;
}
}
},
".agent" => match split_first_arg(args) {
Some((agent_name, args)) => {
let (new_args, _) = split_args_text(args.unwrap_or_default(), cfg!(windows));
File diff suppressed because it is too large Load Diff
+55 -21
View File
@@ -34,8 +34,12 @@ impl DiscoveredMixin {
pub fn wrap_mixin_as_kit(mixin_path: &Path) -> Result<PathBuf> {
let bytes = fs::read(mixin_path)
.with_context(|| format!("Failed to read sbx mixin {}", mixin_path.display()))?;
wrap_mixin_bytes_as_kit(&bytes, &mixin_path.display().to_string())
}
pub fn wrap_mixin_bytes_as_kit(bytes: &[u8], label: &str) -> Result<PathBuf> {
let mut hasher = Sha256::new();
hasher.update(&bytes);
hasher.update(bytes);
let hash = format!("{:x}", hasher.finalize());
let kit_dir = paths::sbx_mixin_kits_dir().join(&hash);
@@ -49,14 +53,10 @@ pub fn wrap_mixin_as_kit(mixin_path: &Path) -> Result<PathBuf> {
fs::create_dir_all(&kit_dir)
.with_context(|| format!("Failed to create mixin kit dir {}", kit_dir.display()))?;
fs::write(&spec_path, &bytes)
fs::write(&spec_path, bytes)
.with_context(|| format!("Failed to write {}", spec_path.display()))?;
debug!(
"Wrapped mixin {} as kit at {}",
mixin_path.display(),
kit_dir.display()
);
debug!("Wrapped mixin {label} as kit at {}", kit_dir.display());
Ok(kit_dir)
}
@@ -73,10 +73,6 @@ pub fn discover() -> Result<Vec<DiscoveredMixin>> {
for path in collect_subdir_mixins(&paths::agents_data_dir()) {
out.push(read_mixin(path)?);
}
// RAG sidecars are FLAT files named `<rag>.sbx-mixin.yaml` inside rags/, not
// the `<subdir>/sbx-mixin.yaml` shape the two scans above walk. Loaded
// unconditionally, mirroring agents/*: a RAG mixin only adds an outbound
// allowlist entry for that RAG's host and opens no inbound rules.
for path in collect_flat_mixins(&paths::rags_dir()) {
out.push(read_mixin(path)?);
}
@@ -97,15 +93,18 @@ pub fn summarize(path: &Path) -> Result<(usize, usize)> {
.with_context(|| format!("Failed to parse sbx mixin {}", path.display()))?;
let installs = value
.get("commands")
.and_then(|c| c.get("install"))
.get("setup")
.and_then(|s| s.get("install"))
.or_else(|| value.get("commands").and_then(|c| c.get("install")))
.and_then(|i| i.as_sequence())
.map(|s| s.len())
.unwrap_or(0);
let domains = value
.get("network")
.and_then(|n| n.get("allowedDomains"))
.get("permissions")
.and_then(|p| p.get("network"))
.and_then(|n| n.get("allow"))
.or_else(|| value.get("network").and_then(|n| n.get("allowedDomains")))
.and_then(|d| d.as_sequence())
.map(|s| s.len())
.unwrap_or(0);
@@ -181,8 +180,6 @@ fn collect_subdir_mixins(dir: &Path) -> Vec<PathBuf> {
result
}
/// Mixins stored as flat `<name>.sbx-mixin.yaml` files directly inside `dir`,
/// matched by suffix rather than by exact filename.
fn collect_flat_mixins(dir: &Path) -> Vec<PathBuf> {
let mut result = Vec::new();
let Ok(rd) = read_dir(dir) else { return result };
@@ -228,6 +225,34 @@ mod tests {
fs::write(
&path,
r#"
schemaVersion: "2"
kind: mixin
setup:
install:
- command: "echo hi"
- command: "echo bye"
permissions:
network:
allow:
- "a.example.com:443"
- "b.example.com:443"
- "c.example.com:443"
"#,
)
.unwrap();
assert_eq!(summarize(&path).unwrap(), (2, 3));
let _ = fs::remove_dir_all(&root);
}
#[test]
fn summarize_falls_back_to_v1_field_paths() {
let root = unique_root("sbx-mixin-counts-v1");
let path = root.join("sbx-mixin.yaml");
fs::write(
&path,
r#"
schemaVersion: "1"
kind: mixin
commands:
@@ -375,6 +400,19 @@ network:
assert_eq!(fs::read_to_string(&spec).unwrap(), content);
}
#[test]
#[serial]
fn wrap_mixin_bytes_as_kit_writes_spec_yaml() {
let _guard = TestCacheDirGuard::new();
let content = b"schemaVersion: '2'\nkind: mixin\nname: generated\n";
let kit_dir = wrap_mixin_bytes_as_kit(content, "generated").unwrap();
let spec = kit_dir.join("spec.yaml");
assert!(spec.exists(), "spec.yaml must exist in wrapped kit dir");
assert_eq!(fs::read(&spec).unwrap(), content);
}
#[test]
#[serial]
fn wrap_mixin_as_kit_is_deterministic_for_identical_content() {
@@ -472,16 +510,13 @@ network:
}
}
/// RAG sidecars are flat `<name>.sbx-mixin.yaml` files, matched by SUFFIX.
#[test]
fn collect_flat_mixins_matches_rag_sidecars_by_suffix() {
let root = unique_root("flat-mixins");
fs::write(root.join("company-docs.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
fs::write(root.join("alpha.sbx-mixin.yaml"), "kind: mixin\n").unwrap();
// The RAGs themselves must not be picked up, only their sidecars.
fs::write(root.join("company-docs.yaml"), "driver: qdrant\n").unwrap();
fs::write(root.join("notes.yaml"), "driver: yaml\n").unwrap();
// A directory whose name ends in the suffix is not a mixin file.
fs::create_dir_all(root.join("decoy.sbx-mixin.yaml")).unwrap();
let found = collect_flat_mixins(&root);
@@ -489,7 +524,6 @@ network:
.iter()
.map(|p| p.file_name().unwrap().to_str().unwrap())
.collect();
// Sorted by file name, so the order is deterministic.
assert_eq!(
names,
vec!["alpha.sbx-mixin.yaml", "company-docs.sbx-mixin.yaml"]
+162 -57
View File
@@ -10,13 +10,17 @@ use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use which::which;
pub(crate) mod mcp_credentials;
mod mixins;
pub(crate) use mcp_credentials::sandbox_secret_env_var;
use crate::config::AppConfig;
use crate::config::Config;
use crate::config::VAULT_DATA_FILE_NAME;
use crate::config::paths;
use crate::rag::RagData;
use crate::sandbox::mcp_credentials::MCP_MIXIN_NAME;
use crate::sandbox::mixins::DiscoveredMixin;
use crate::utils::run_command_with_output;
use crate::vault::SECRET_RE;
@@ -51,17 +55,22 @@ pub fn launch(name: Option<String>, fresh: bool) -> Result<()> {
let registered = sbx_registered_services()?;
inject_llm_secret(&config_content, &vault, &registered)?;
if !fresh {
inject_mcp_secrets(&vault, &registered)?;
inject_rag_secrets(&vault, &registered)?;
}
let credentials_mixin = if fresh {
None
} else {
inject_mcp_secrets(&vault, &registered)?
};
let discovered = mixins::discover()?;
if sandbox_exists(&name)? {
info!("Re-attaching to existing sandbox '{name}'");
} else {
mixins::log_discovery(&discovered, false);
create_sandbox(&name, &kit_path, &discovered)?;
create_sandbox(&name, &kit_path, &discovered, credentials_mixin.as_deref())?;
if !fresh {
copy_host_files(&name)?;
}
@@ -234,7 +243,7 @@ fn inject_llm_secret(
if registered.contains(&service) {
eprintln!(
"Secret for '{service}' already registered with sbx. \
To update it, run: sbx secret set -g --force {service}"
To update it, run: sbx secret set --force {service}"
);
continue;
}
@@ -249,23 +258,14 @@ fn inject_llm_secret(
Ok(())
}
fn find_secret_placeholder(value: &Value) -> Option<String> {
match value {
Value::String(s) => SECRET_RE
.captures(s)
.ok()
.flatten()
.map(|caps| caps[1].to_string()),
Value::Object(map) => map.values().find_map(find_secret_placeholder),
Value::Array(arr) => arr.iter().find_map(find_secret_placeholder),
_ => None,
}
}
fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()> {
/// Registers one sbx secret per distinct `{{placeholder}}` in the MCP config
/// and returns the generated schema-v2 `coyote-mcp` mixin (network egress for
/// every remote MCP server + credential declarations), or `None` when the MCP
/// config references no remote servers and no secrets.
fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<Option<String>> {
let mcp_path = paths::mcp_config_file();
if !mcp_path.exists() {
return Ok(());
return Ok(None);
}
let content = fs::read_to_string(&mcp_path)
@@ -274,40 +274,46 @@ fn inject_mcp_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()>
.with_context(|| format!("Failed to parse {}", mcp_path.display()))?;
let Some(servers) = mcp.get("mcpServers").and_then(|v| v.as_object()) else {
return Ok(());
return Ok(None);
};
for (server_name, server_config) in servers {
let Some(secret_name) = find_secret_placeholder(server_config) else {
continue;
};
let credentials = mcp_credentials::collect_credentials(servers)?;
let allow_entries = mcp_credentials::collect_server_allow_entries(servers);
if credentials.is_empty() && allow_entries.is_empty() {
return Ok(None);
}
if registered.contains(server_name.as_str()) {
for credential in &credentials {
if registered.contains(credential.service_id.as_str()) {
eprintln!(
"Secret for '{server_name}' already registered with sbx. \
To update it, run: sbx secret set -g --force {server_name}"
"Secret for '{}' already registered with sbx. \
To update it, run: sbx secret set --force {}",
credential.service_id, credential.service_id
);
continue;
}
let secret_value = vault.get_secret(&secret_name, false).with_context(|| {
format!(
"Secret '{secret_name}' referenced by MCP server '{server_name}' not found \
in vault. Add it with: coyote --add-secret {secret_name}"
)
})?;
let secret_value = vault
.get_secret(&credential.secret_name, false)
.with_context(|| {
format!(
"Secret '{}' referenced by MCP server(s) {} not found \
in vault. Add it with: coyote --add-secret {}",
credential.secret_name,
mcp_credentials::quoted_list(&credential.servers),
credential.secret_name
)
})?;
sbx_secret_set(server_name, &secret_value)?;
sbx_secret_set(&credential.service_id, &secret_value)?;
}
Ok(())
Ok(Some(mcp_credentials::render_mixin_yaml(
&credentials,
&allow_entries,
)?))
}
/// Registers the API key of every attached RAG with the sbx proxy.
///
/// `launch()` has no notion of an active RAG — that is runtime state set by
/// `--rag` / `.rag` and never persisted — so every attached RAG is scanned
/// unconditionally, exactly as `inject_mcp_secrets` does for MCP servers.
fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()> {
let rags_dir = paths::rags_dir();
if !rags_dir.exists() {
@@ -319,7 +325,6 @@ fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()>
continue;
}
let stem = match path.file_stem().and_then(|s| s.to_str()) {
// Skip sidecars ("myrag.sbx-mixin.yaml" has stem "myrag.sbx-mixin").
Some(s) if !paths::is_rag_sidecar_name(s) => s.to_string(),
_ => continue,
};
@@ -335,19 +340,18 @@ fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()>
let Some(placeholder) = data.driver_config.get("api_key") else {
continue;
};
if registered.contains(&stem) {
let service_id = mcp_credentials::secret_service_id(&stem);
if service_id.is_empty() || registered.contains(&service_id) {
continue;
}
let secret_name = placeholder
.trim_start_matches("{{")
.trim_end_matches("}}")
.trim();
// Degrade rather than abort: one stale RAG key must not block the whole
// sandbox launch. Queries to that RAG fail with a 401 at runtime, which
// is recoverable without a restart.
match vault.get_secret(secret_name, false) {
Ok(secret_value) => {
sbx_secret_set(&stem, &secret_value)
sbx_secret_set(&service_id, &secret_value)
.context("Failed to register RAG secret with sbx")?;
}
Err(e) => {
@@ -359,6 +363,7 @@ fn inject_rag_secrets(vault: &Vault, registered: &HashSet<String>) -> Result<()>
}
}
}
Ok(())
}
@@ -366,7 +371,7 @@ fn provider_to_sbx_service(provider_type: &str, client_name: Option<&str>) -> St
match provider_type {
"claude" => "anthropic".to_string(),
"openai" => "openai".to_string(),
"gemini" | "vertexai" => "google".to_string(),
"gemini" | "vertexai" => "gemini".to_string(),
"openai-compatible" => client_name.unwrap_or("openai-compatible").to_string(),
other => client_name.unwrap_or(other).to_string(),
}
@@ -399,25 +404,29 @@ fn sbx_registered_services() -> Result<HashSet<String>> {
fn sbx_secret_set(service: &str, secret_value: &str) -> Result<()> {
let mut child = Command::new(SBX_BINARY)
.args(["secret", "set", "-g", service])
.args(["secret", "set", service])
.stdin(Stdio::piped())
.stdout(Stdio::inherit())
.stderr(Stdio::inherit())
.spawn()
.context("Failed to spawn `sbx secret set -g`")?;
.context("Failed to spawn `sbx secret set`")?;
if let Some(mut stdin_handle) = child.stdin.take() {
stdin_handle
.write_all(secret_value.as_bytes())
.context("Failed to write secret to `sbx secret set -g` stdin")?;
.context("Failed to write secret to `sbx secret set` stdin")?;
}
let status = child
.wait()
.context("Failed to wait for `sbx secret set -g`")?;
.context("Failed to wait for `sbx secret set`")?;
if !status.success() {
bail!("`sbx secret set -g {service}` exited with {status}");
eprintln!(
"Warning: failed to register sbx secret '{service}' \
(`sbx secret set {service}` exited with {status}). \
Set it manually with: echo '<value>' | sbx secret set {service}"
);
}
Ok(())
@@ -436,9 +445,17 @@ fn sandbox_exists(name: &str) -> Result<bool> {
.any(|line| line.split_whitespace().next() == Some(name)))
}
fn create_sandbox(name: &str, kit_path: &Path, mixins: &[DiscoveredMixin]) -> Result<()> {
fn create_sandbox(
name: &str,
kit_path: &Path,
mixins: &[DiscoveredMixin],
credentials_mixin: Option<&str>,
) -> Result<()> {
info!("Creating sandbox '{name}'");
let args = build_create_args(name, kit_path, mixins)?;
let credentials_kit = credentials_mixin
.map(|yaml| mixins::wrap_mixin_bytes_as_kit(yaml.as_bytes(), MCP_MIXIN_NAME))
.transpose()?;
let args = build_create_args(name, kit_path, mixins, credentials_kit.as_deref())?;
debug!("sbx {}", args.join(" "));
let status = Command::new(SBX_BINARY)
.args(&args)
@@ -459,6 +476,7 @@ fn build_create_args(
name: &str,
kit_path: &Path,
mixins: &[DiscoveredMixin],
credentials_kit: Option<&Path>,
) -> Result<Vec<String>> {
let kit_str = kit_path
.to_str()
@@ -482,6 +500,15 @@ fn build_create_args(
args.push(mixin_str);
}
if let Some(kit) = credentials_kit {
let cred_str = kit
.to_str()
.ok_or_else(|| anyhow!("Credentials kit path is not valid UTF-8: {}", kit.display()))?
.to_string();
args.push("--kit".to_string());
args.push(cred_str);
}
args.push(SANDBOX_AGENT.to_string());
args.push(".".to_string());
@@ -619,6 +646,7 @@ fn chown_agent_recursive(sandbox: &str, path: &str) -> Result<()> {
#[cfg(test)]
mod tests {
use super::*;
use std::time::{SystemTime, UNIX_EPOCH};
#[test]
fn sanitize_name_lowercases() {
@@ -687,8 +715,8 @@ mod tests {
#[test]
fn build_create_args_emits_base_kit_before_mixins() {
let kit = PathBuf::from("/cache/sbx-kit");
let unique = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
let unique = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos();
let dir_a = env::temp_dir().join(format!("coyote-mixin-a-{unique}"));
@@ -711,7 +739,7 @@ mod tests {
},
];
let args = build_create_args("my-box", &kit, &mixins).unwrap();
let args = build_create_args("my-box", &kit, &mixins, None).unwrap();
assert_eq!(
args,
@@ -737,7 +765,9 @@ mod tests {
#[test]
fn build_create_args_with_no_mixins_omits_mixin_kits() {
let kit = PathBuf::from("/cache/sbx-kit");
let args = build_create_args("box", &kit, &[]).unwrap();
let args = build_create_args("box", &kit, &[], None).unwrap();
assert_eq!(
args,
vec![
@@ -751,4 +781,79 @@ mod tests {
]
);
}
#[test]
fn build_create_args_appends_credentials_kit_after_mixins() {
let kit = PathBuf::from("/cache/sbx-kit");
let credentials_kit = PathBuf::from("/cache/sbx-mixin-kits/abc123");
let args = build_create_args("box", &kit, &[], Some(&credentials_kit)).unwrap();
assert_eq!(
args,
vec![
"create".to_string(),
"--name".to_string(),
"box".to_string(),
"--kit".to_string(),
"/cache/sbx-kit".to_string(),
"--kit".to_string(),
"/cache/sbx-mixin-kits/abc123".to_string(),
"coyote".to_string(),
".".to_string(),
]
);
}
#[test]
fn build_create_args_orders_base_kit_then_mixins_then_credentials_kit() {
let kit = PathBuf::from("/cache/sbx-kit");
let credentials_kit = PathBuf::from("/cache/sbx-mixin-kits/abc123");
let unique = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_nanos();
let dir = env::temp_dir().join(format!("coyote-mixin-cred-{unique}"));
fs::create_dir_all(&dir).unwrap();
let mixins = vec![DiscoveredMixin {
path: dir.clone(),
label: "user".into(),
install_count: 0,
domain_count: 0,
}];
let args = build_create_args("box", &kit, &mixins, Some(&credentials_kit)).unwrap();
assert_eq!(
args,
vec![
"create".to_string(),
"--name".to_string(),
"box".to_string(),
"--kit".to_string(),
"/cache/sbx-kit".to_string(),
"--kit".to_string(),
dir.display().to_string(),
"--kit".to_string(),
"/cache/sbx-mixin-kits/abc123".to_string(),
"coyote".to_string(),
".".to_string(),
]
);
let _ = fs::remove_dir_all(&dir);
}
#[test]
fn provider_to_sbx_service_maps_gemini_family_to_gemini() {
assert_eq!(provider_to_sbx_service("gemini", None), "gemini");
assert_eq!(provider_to_sbx_service("vertexai", None), "gemini");
}
#[test]
fn provider_to_sbx_service_maps_known_providers() {
assert_eq!(provider_to_sbx_service("claude", None), "anthropic");
assert_eq!(provider_to_sbx_service("openai", None), "openai");
}
}
+27 -2
View File
@@ -1,5 +1,5 @@
use crate::config::ensure_parent_exists;
use crate::sandbox::SANDBOX_ENV_FLAG;
use crate::sandbox::{SANDBOX_ENV_FLAG, sandbox_secret_env_var};
use crate::vault::{SECRET_RE, Vault};
use anyhow::Result;
use anyhow::anyhow;
@@ -358,7 +358,32 @@ fn required_cli_preflight(label: &str, cli: &str, install_url: &str) {
pub fn interpolate_secrets(content: &str, vault: &Vault) -> Result<(String, Vec<String>)> {
if env::var_os(SANDBOX_ENV_FLAG).is_some() {
return Ok((content.to_string(), vec![]));
let (parsed, missing) = interpolate_secrets_with(content, None, |name| {
env::var(sandbox_secret_env_var(name)).map_err(|_| {
anyhow!(SecretError::NotFound {
key: name.to_string(),
provider: "sandbox environment",
})
})
})?;
if !missing.is_empty() {
let mut env_vars: Vec<String> = missing
.iter()
.map(|name| sandbox_secret_env_var(name))
.collect();
env_vars.sort();
env_vars.dedup();
eprintln!(
"Config references secrets that are not available inside this sandbox \
(expected env vars: {}). Sandbox secrets are provisioned at creation \
from the host; add the missing secrets on the host, then re-create \
the sandbox.",
env_vars.join(", ")
);
}
return Ok((parsed, missing));
}
interpolate_secrets_with(content, vault.auth_hint(), |name| {
vault.get_secret(name, false)