docs: prompt caching (prompt_cache, default on), token-usage prompt variables, cache pricing catalog fields

Dark-Alex-17 committed 2026-09-14 14:50:41 -06:00
1 parent 5a07931d67
commit 15d0346b42
3 files changed
+50 -4

No files matched your search

+26
@@ -62,6 +62,8 @@ The `models` array lists the available models from the model client. Each one ha
| `type` | * | `all` | The type of model. Coyote supports only 3 types of models: <ul><li>`chat`</li><li>`embedding`</li><li>`reranker`</li></ul> | | `type` | * | `all` | The type of model. Coyote supports only 3 types of models: <ul><li>`chat`</li><li>`embedding`</li><li>`reranker`</li></ul> |
| `input_price` | | `all` | The cost in USD per 1M tokens for each input sequence; Coyote will keep track of usage costs if this is defined | | `input_price` | | `all` | The cost in USD per 1M tokens for each input sequence; Coyote will keep track of usage costs if this is defined |
| `output_price` | | `all` | The cost in USD per 1M tokens of the model output; Coyote will keep track of usage costs if this is defined | | `output_price` | | `all` | The cost in USD per 1M tokens of the model output; Coyote will keep track of usage costs if this is defined |
| `cache_read_price` | | `all` | The cost in USD per 1M tokens for reading prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it |
| `cache_write_price` | | `all` | The cost in USD per 1M tokens for writing prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it |
| `patch` | | `all` | See the [model-specific patch configuration](Patches#model-specific-patches) documentation | | `patch` | | `all` | See the [model-specific patch configuration](Patches#model-specific-patches) documentation |
| `max_input_tokens` | | `all` | The maximum number of input tokens for the model | | `max_input_tokens` | | `all` | The maximum number of input tokens for the model |
| `max_output_tokens` | | `chat` | The maximum number of output tokens for the model | | `max_output_tokens` | | `chat` | The maximum number of output tokens for the model |
@@ -77,6 +79,30 @@ The `models` array lists the available models from the model client. Each one ha
| `default_chunk_size` | | `embedding` | The default chunk size to use with the given model | | `default_chunk_size` | | `embedding` | The default chunk size to use with the given model |
| `max_batch_size` | | `embedding` | The maximum batch size that the given embedding model supports | | `max_batch_size` | | `embedding` | The maximum batch size that the given embedding model supports |
# Prompt Caching
The Anthropic-protocol clients (i.e. `claude`, `bedrock` (Anthropic models only), and `vertexai` (Claude models)) support
[Anthropic prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching), and Coyote enables it
**by default**. Requests automatically place cache breakpoints on the system prompt, the tool definitions, and the two
most recent user messages, so long sessions and tool-heavy loops re-read the cached prefix each turn instead of paying
the full input-token price.
Set `prompt_cache: false` on the client to opt out:
```yaml
clients:
- type: claude
prompt_cache: false # Optional - Anthropic prompt caching (default: true)
```
> [!NOTE]
> * On Bedrock, cache points are only added for Anthropic models (`anthropic.*` / `*claude*` model IDs); requests for
> all other Bedrock models are unaffected.
> * Cache activity is visible in the REPL: the default right prompt shows a ⚡ segment with the tokens read from cache
> on the most recent call (see the [token usage prompt variables](REPL-Prompt#variables)), and `.info session` shows
> cumulative cache reads/writes for the session.
> * Setting `prompt_cache: false` produces request bodies byte-identical to Coyote versions that predate prompt caching.
# Authentication # Authentication
Coyote clients support two authentication methods: **API keys** and **OAuth**. Each client entry in your configuration Coyote clients support two authentication methods: **API keys** and **OAuth**. Each client entry in your configuration
+5 -3
@@ -1,9 +1,11 @@
# Model Catalog # Model Catalog
Coyote ships with a built-in catalog of models for every supported provider: context window Coyote ships with a built-in catalog of models for every supported provider: context window
sizes, output limits, pricing, capabilities (vision, function calling, reasoning), wire-name sizes, output limits, pricing (including prompt-cache read/write rates (`cache_read_price` /
aliases, and per-model request patches. The catalog is what powers the `.model` tab-completion `cache_write_price`) where the sources publish them), capabilities (vision, function calling,
table in the REPL, `--list-models`, token-budget accounting, and cost display. reasoning), wire-name aliases, and per-model request patches. The catalog is what powers the
`.model` tab-completion table in the REPL, `--list-models`, token-budget accounting, and cost
display.
This page explains where that data comes from, how it stays fresh automatically, and how to This page explains where that data comes from, how it stays fresh automatically, and how to
control it. control it.
+19 -1
@@ -3,9 +3,14 @@ and `right_prompt` settings in the global Coyote configuration file:
```yaml ```yaml
left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} ' left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} '
right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}' right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}'
``` ```
> [!NOTE]
> The default `right_prompt` includes a `⚡{last_cache_read_tokens}` segment that appears whenever the most recent
> API call read tokens from Anthropic's prompt cache (see the [token usage variables](#variables) below). Because
> `{?variable ...}` treats `0` as falsy, the indicator stays hidden on cold-cache calls.
The location of the global configuration file differs between systems, so you can use the following command to find your The location of the global configuration file differs between systems, so you can use the following command to find your
global configuration file's location: global configuration file's location:
@@ -83,6 +88,19 @@ consume_tokens: 200 # The number of tokens consumed
consume_percent: 1% # The percentage of tokens consumed to the maximum input tokens consume_percent: 1% # The percentage of tokens consumed to the maximum input tokens
user_messages_len: 0 # The total number of sent user messages user_messages_len: 0 # The total number of sent user messages
# Token Usage Variables
# Real, API-reported token usage. Currently populated by the Anthropic-protocol clients (claude, bedrock with
# Anthropic models, and vertexai with Claude models); other providers leave these unset, so always guard them
# with {?variable ...}. Reminder: {?variable} treats 0 as falsy, so a cold cache (0 tokens read) hides the segment.
last_input_tokens: 1200 # Input tokens billed for the most recent API call
last_output_tokens: 340 # Output tokens generated by the most recent API call
last_cache_read_tokens: 900 # Prompt-cache tokens read (cache hits) on the most recent API call
last_cache_creation_tokens: 0 # Prompt-cache tokens written on the most recent API call
total_input_tokens: 5400 # Cumulative input tokens for the active session (requires an active session)
total_output_tokens: 1800 # Cumulative output tokens for the active session
total_cache_read_tokens: 4100 # Cumulative cache-read tokens for the active session
total_cache_creation_tokens: 1200 # Cumulative cache-write tokens for the active session
# RAG Variables # RAG Variables
rag: temp # The name of the active RAG rag: temp # The name of the active RAG