docs: prompt caching (prompt_cache, default on), token-usage prompt variables, cache pricing catalog fields
3 files changed
+50
-4
No files matched your search
+26
@@ -62,6 +62,8 @@ The `models` array lists the available models from the model client. Each one ha
|
|||||||
| `type` | * | `all` | The type of model. Coyote supports only 3 types of models: <ul><li>`chat`</li><li>`embedding`</li><li>`reranker`</li></ul> |
|
| `type` | * | `all` | The type of model. Coyote supports only 3 types of models: <ul><li>`chat`</li><li>`embedding`</li><li>`reranker`</li></ul> |
|
||||||
| `input_price` | | `all` | The cost in USD per 1M tokens for each input sequence; Coyote will keep track of usage costs if this is defined |
|
| `input_price` | | `all` | The cost in USD per 1M tokens for each input sequence; Coyote will keep track of usage costs if this is defined |
|
||||||
| `output_price` | | `all` | The cost in USD per 1M tokens of the model output; Coyote will keep track of usage costs if this is defined |
|
| `output_price` | | `all` | The cost in USD per 1M tokens of the model output; Coyote will keep track of usage costs if this is defined |
|
||||||
|
| `cache_read_price` | | `all` | The cost in USD per 1M tokens for reading prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it |
|
||||||
|
| `cache_write_price` | | `all` | The cost in USD per 1M tokens for writing prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it |
|
||||||
| `patch` | | `all` | See the [model-specific patch configuration](Patches#model-specific-patches) documentation |
|
| `patch` | | `all` | See the [model-specific patch configuration](Patches#model-specific-patches) documentation |
|
||||||
| `max_input_tokens` | | `all` | The maximum number of input tokens for the model |
|
| `max_input_tokens` | | `all` | The maximum number of input tokens for the model |
|
||||||
| `max_output_tokens` | | `chat` | The maximum number of output tokens for the model |
|
| `max_output_tokens` | | `chat` | The maximum number of output tokens for the model |
|
||||||
@@ -77,6 +79,30 @@ The `models` array lists the available models from the model client. Each one ha
|
|||||||
| `default_chunk_size` | | `embedding` | The default chunk size to use with the given model |
|
| `default_chunk_size` | | `embedding` | The default chunk size to use with the given model |
|
||||||
| `max_batch_size` | | `embedding` | The maximum batch size that the given embedding model supports |
|
| `max_batch_size` | | `embedding` | The maximum batch size that the given embedding model supports |
|
||||||
|
|
||||||
|
# Prompt Caching
|
||||||
|
|
||||||
|
The Anthropic-protocol clients (i.e. `claude`, `bedrock` (Anthropic models only), and `vertexai` (Claude models)) support
|
||||||
|
[Anthropic prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching), and Coyote enables it
|
||||||
|
**by default**. Requests automatically place cache breakpoints on the system prompt, the tool definitions, and the two
|
||||||
|
most recent user messages, so long sessions and tool-heavy loops re-read the cached prefix each turn instead of paying
|
||||||
|
the full input-token price.
|
||||||
|
|
||||||
|
Set `prompt_cache: false` on the client to opt out:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
clients:
|
||||||
|
- type: claude
|
||||||
|
prompt_cache: false # Optional - Anthropic prompt caching (default: true)
|
||||||
|
```
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> * On Bedrock, cache points are only added for Anthropic models (`anthropic.*` / `*claude*` model IDs); requests for
|
||||||
|
> all other Bedrock models are unaffected.
|
||||||
|
> * Cache activity is visible in the REPL: the default right prompt shows a ⚡ segment with the tokens read from cache
|
||||||
|
> on the most recent call (see the [token usage prompt variables](REPL-Prompt#variables)), and `.info session` shows
|
||||||
|
> cumulative cache reads/writes for the session.
|
||||||
|
> * Setting `prompt_cache: false` produces request bodies byte-identical to Coyote versions that predate prompt caching.
|
||||||
|
|
||||||
# Authentication
|
# Authentication
|
||||||
|
|
||||||
Coyote clients support two authentication methods: **API keys** and **OAuth**. Each client entry in your configuration
|
Coyote clients support two authentication methods: **API keys** and **OAuth**. Each client entry in your configuration
|
||||||
|
|||||||
+5
-3
@@ -1,9 +1,11 @@
|
|||||||
# Model Catalog
|
# Model Catalog
|
||||||
|
|
||||||
Coyote ships with a built-in catalog of models for every supported provider: context window
|
Coyote ships with a built-in catalog of models for every supported provider: context window
|
||||||
sizes, output limits, pricing, capabilities (vision, function calling, reasoning), wire-name
|
sizes, output limits, pricing (including prompt-cache read/write rates (`cache_read_price` /
|
||||||
aliases, and per-model request patches. The catalog is what powers the `.model` tab-completion
|
`cache_write_price`) where the sources publish them), capabilities (vision, function calling,
|
||||||
table in the REPL, `--list-models`, token-budget accounting, and cost display.
|
reasoning), wire-name aliases, and per-model request patches. The catalog is what powers the
|
||||||
|
`.model` tab-completion table in the REPL, `--list-models`, token-budget accounting, and cost
|
||||||
|
display.
|
||||||
|
|
||||||
This page explains where that data comes from, how it stays fresh automatically, and how to
|
This page explains where that data comes from, how it stays fresh automatically, and how to
|
||||||
control it.
|
control it.
|
||||||
|
|||||||
+19
-1
@@ -3,9 +3,14 @@ and `right_prompt` settings in the global Coyote configuration file:
|
|||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
|
left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
|
||||||
right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
|
right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> The default `right_prompt` includes a `⚡{last_cache_read_tokens}` segment that appears whenever the most recent
|
||||||
|
> API call read tokens from Anthropic's prompt cache (see the [token usage variables](#variables) below). Because
|
||||||
|
> `{?variable ...}` treats `0` as falsy, the indicator stays hidden on cold-cache calls.
|
||||||
|
|
||||||
The location of the global configuration file differs between systems, so you can use the following command to find your
|
The location of the global configuration file differs between systems, so you can use the following command to find your
|
||||||
global configuration file's location:
|
global configuration file's location:
|
||||||
|
|
||||||
@@ -83,6 +88,19 @@ consume_tokens: 200 # The number of tokens consumed
|
|||||||
consume_percent: 1% # The percentage of tokens consumed to the maximum input tokens
|
consume_percent: 1% # The percentage of tokens consumed to the maximum input tokens
|
||||||
user_messages_len: 0 # The total number of sent user messages
|
user_messages_len: 0 # The total number of sent user messages
|
||||||
|
|
||||||
|
# Token Usage Variables
|
||||||
|
# Real, API-reported token usage. Currently populated by the Anthropic-protocol clients (claude, bedrock with
|
||||||
|
# Anthropic models, and vertexai with Claude models); other providers leave these unset, so always guard them
|
||||||
|
# with {?variable ...}. Reminder: {?variable} treats 0 as falsy, so a cold cache (0 tokens read) hides the segment.
|
||||||
|
last_input_tokens: 1200 # Input tokens billed for the most recent API call
|
||||||
|
last_output_tokens: 340 # Output tokens generated by the most recent API call
|
||||||
|
last_cache_read_tokens: 900 # Prompt-cache tokens read (cache hits) on the most recent API call
|
||||||
|
last_cache_creation_tokens: 0 # Prompt-cache tokens written on the most recent API call
|
||||||
|
total_input_tokens: 5400 # Cumulative input tokens for the active session (requires an active session)
|
||||||
|
total_output_tokens: 1800 # Cumulative output tokens for the active session
|
||||||
|
total_cache_read_tokens: 4100 # Cumulative cache-read tokens for the active session
|
||||||
|
total_cache_creation_tokens: 1200 # Cumulative cache-write tokens for the active session
|
||||||
|
|
||||||
# RAG Variables
|
# RAG Variables
|
||||||
rag: temp # The name of the active RAG
|
rag: temp # The name of the active RAG
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user