diff --git a/Clients.md b/Clients.md index b162951..bbacc65 100644 --- a/Clients.md +++ b/Clients.md @@ -62,6 +62,8 @@ The `models` array lists the available models from the model client. Each one ha | `type` | * | `all` | The type of model. Coyote supports only 3 types of models: | | `input_price` | | `all` | The cost in USD per 1M tokens for each input sequence; Coyote will keep track of usage costs if this is defined | | `output_price` | | `all` | The cost in USD per 1M tokens of the model output; Coyote will keep track of usage costs if this is defined | +| `cache_read_price` | | `all` | The cost in USD per 1M tokens for reading prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it | +| `cache_write_price` | | `all` | The cost in USD per 1M tokens for writing prompt-cache tokens; populated automatically from the [model catalog](Model-Catalog) where the sources publish it | | `patch` | | `all` | See the [model-specific patch configuration](Patches#model-specific-patches) documentation | | `max_input_tokens` | | `all` | The maximum number of input tokens for the model | | `max_output_tokens` | | `chat` | The maximum number of output tokens for the model | @@ -77,6 +79,30 @@ The `models` array lists the available models from the model client. Each one ha | `default_chunk_size` | | `embedding` | The default chunk size to use with the given model | | `max_batch_size` | | `embedding` | The maximum batch size that the given embedding model supports | +# Prompt Caching + +The Anthropic-protocol clients (i.e. `claude`, `bedrock` (Anthropic models only), and `vertexai` (Claude models)) support +[Anthropic prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching), and Coyote enables it +**by default**. Requests automatically place cache breakpoints on the system prompt, the tool definitions, and the two +most recent user messages, so long sessions and tool-heavy loops re-read the cached prefix each turn instead of paying +the full input-token price. + +Set `prompt_cache: false` on the client to opt out: + +```yaml +clients: + - type: claude + prompt_cache: false # Optional - Anthropic prompt caching (default: true) +``` + +> [!NOTE] +> * On Bedrock, cache points are only added for Anthropic models (`anthropic.*` / `*claude*` model IDs); requests for +> all other Bedrock models are unaffected. +> * Cache activity is visible in the REPL: the default right prompt shows a ⚡ segment with the tokens read from cache +> on the most recent call (see the [token usage prompt variables](REPL-Prompt#variables)), and `.info session` shows +> cumulative cache reads/writes for the session. +> * Setting `prompt_cache: false` produces request bodies byte-identical to Coyote versions that predate prompt caching. + # Authentication Coyote clients support two authentication methods: **API keys** and **OAuth**. Each client entry in your configuration diff --git a/Model-Catalog.md b/Model-Catalog.md index 616b2cd..3443619 100644 --- a/Model-Catalog.md +++ b/Model-Catalog.md @@ -1,9 +1,11 @@ # Model Catalog Coyote ships with a built-in catalog of models for every supported provider: context window -sizes, output limits, pricing, capabilities (vision, function calling, reasoning), wire-name -aliases, and per-model request patches. The catalog is what powers the `.model` tab-completion -table in the REPL, `--list-models`, token-budget accounting, and cost display. +sizes, output limits, pricing (including prompt-cache read/write rates (`cache_read_price` / +`cache_write_price`) where the sources publish them), capabilities (vision, function calling, +reasoning), wire-name aliases, and per-model request patches. The catalog is what powers the +`.model` tab-completion table in the REPL, `--list-models`, token-budget accounting, and cost +display. This page explains where that data comes from, how it stays fresh automatically, and how to control it. diff --git a/REPL-Prompt.md b/REPL-Prompt.md index 6c72fc9..4650e3b 100644 --- a/REPL-Prompt.md +++ b/REPL-Prompt.md @@ -3,9 +3,14 @@ and `right_prompt` settings in the global Coyote configuration file: ```yaml left_prompt: '{color.red}{model}){color.green}{?session {?agent {agent}>}{session}{?role /}}{!session {?agent {agent}>}}{role}{?rag @{rag}}{color.cyan}{?session )}{!session >}{color.reset} ' -right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}' +right_prompt: '{color.cyan}{?reasoning_effort [{reasoning_effort}] }{color.purple}{?last_cache_read_tokens ⚡{last_cache_read_tokens} }{?session {?consume_tokens {consume_tokens}({consume_percent}%)}{!consume_tokens {consume_tokens}}}{color.reset}' ``` +> [!NOTE] +> The default `right_prompt` includes a `⚡{last_cache_read_tokens}` segment that appears whenever the most recent +> API call read tokens from Anthropic's prompt cache (see the [token usage variables](#variables) below). Because +> `{?variable ...}` treats `0` as falsy, the indicator stays hidden on cold-cache calls. + The location of the global configuration file differs between systems, so you can use the following command to find your global configuration file's location: @@ -83,6 +88,19 @@ consume_tokens: 200 # The number of tokens consumed consume_percent: 1% # The percentage of tokens consumed to the maximum input tokens user_messages_len: 0 # The total number of sent user messages +# Token Usage Variables +# Real, API-reported token usage. Currently populated by the Anthropic-protocol clients (claude, bedrock with +# Anthropic models, and vertexai with Claude models); other providers leave these unset, so always guard them +# with {?variable ...}. Reminder: {?variable} treats 0 as falsy, so a cold cache (0 tokens read) hides the segment. +last_input_tokens: 1200 # Input tokens billed for the most recent API call +last_output_tokens: 340 # Output tokens generated by the most recent API call +last_cache_read_tokens: 900 # Prompt-cache tokens read (cache hits) on the most recent API call +last_cache_creation_tokens: 0 # Prompt-cache tokens written on the most recent API call +total_input_tokens: 5400 # Cumulative input tokens for the active session (requires an active session) +total_output_tokens: 1800 # Cumulative output tokens for the active session +total_cache_read_tokens: 4100 # Cumulative cache-read tokens for the active session +total_cache_creation_tokens: 1200 # Cumulative cache-write tokens for the active session + # RAG Variables rag: temp # The name of the active RAG