You are likely wasting tokens by reprocessing the same system instructions, tool definitions, and reference documents on every single API call. Prompt caching is an optimization that lets the model store the computed state of frequently used context. By caching these prefixes, you stop paying to recompute the same attention weights over and over.
Processing tokens is a compute-heavy operation centered on the "attention" mechanism. In this stage, every token's vector looks backward at previous vectors to adjust its value in high-dimensional space. Prompt caching lets the LLM skip this expensive compute for the static parts of your prompt, resuming from a specific prefix rather than starting from zero.
When implemented correctly, prompt caching cuts input costs by up to 90% and reduces first-token latency by up to 85%.
For production workloads, that is the difference between a high-margin agent and one that steadily runs up your costs.
What is prompt caching?
Prompt caching is a technique for storing the unchanged portions of a prompt: long instructional blocks, few-shot examples, or large knowledge bases. Unlike conventional web caching that stores static HTML or database results, prompt caching targets the LLM's internal attention mechanism. It captures the KV (key-value) cache of the transformer, storing the computed representations of the prefix so the model doesn't have to re-tokenize and re-process the same data.
A key distinction for engineers is that the prefix hash is cumulative. Because the attention mechanism lets each token look back at all previous tokens, any change at the beginning of the prompt (even a single character) invalidates the entire cache for that segment and everything after it. Prompt caching also requires exact parameter alignment: changing the temperature, top-p, or even the reasoning level on certain models results in a cache miss.
How does prompt caching work?
The LLM pipeline tokenizes text into IDs, converts those IDs into high-dimensional vectors, and performs the "compute attention" step. During attention computation, vectors are adjusted based on their relationship to previous tokens. Prompt caching stores the result of this math for a specific prefix, which follows a strict hierarchy: tools first, then system instructions, and finally message history.
When you send a request, the system uses a 20-block lookback window. It checks at most 20 positions per breakpoint, walking backward from your designated cache point to find a matching prefix that was previously written. If the prefix diverged more than 20 blocks before your current breakpoint, the lookback fails. This is why manual breakpoints matter for growing conversations that exceed the 20-block limit.
You set this at the block level using the cache_control parameter to mark exactly where the static prefix ends.
"content": [
{
"type": "text",
"text": "Extensive system context and few-shot examples...",
"cache_control": { "type": "ephemeral" }
}
]