Prompt caching is the highest-return optimisation available to most LLM applications, and it is frequently either unused or used in a way that never produces a hit. The mechanism is simple once you see it, and it dictates exactly how a prompt has to be laid out.
The rule that determines everything
Caching works on an exact prefix. Everything static goes at the start, everything variable at the end. One changed character near the beginning invalidates the entire cache for that request.
What is being cached
Before generating anything, a model processes your entire prompt and builds an internal key-value representation for every token — the prefill stage. On a long prompt this dominates time-to-first-token and a large share of the cost.
That state is deterministic: the same prefix always produces the same KV entries. So it can be stored and reused.
Because each token's representation depends on every token before it, reuse is only possible for a contiguous prefix. Change something in the middle and everything after it must be recomputed — which is why the ordering rule is absolute rather than a preference.
What it saves
| Uncached input | Cache write | Cache read | |
|---|---|---|---|
| Typical rate | 100% | ~125% | ~10% |
| Latency | Full prefill | Full prefill | Skipped |
Note the write premium. The first request costs slightly more than an uncached one, so caching a prefix used once loses money. It pays from the second hit onwards, and pays enormously by the tenth.
Structuring the prompt
Conversation history is the interesting case. Because turns are appended rather than inserted, each request's history is a prefix of the next one's — so a growing conversation caches naturally, with only the new turn needing computation. This is why caching suits chat and agent loops so well.
🚨 The things that silently break caching
- A timestamp in the system prompt. "Current date and time: 2026-08-02 14:31:07" changes every second. Use the date alone, or move it to the end.
- A user name or session ID injected near the top.
- Randomised example order. Shuffling few-shot examples guarantees a miss.
- Non-deterministic JSON serialisation. Key order must be stable — sort them.
- Tools built from a set or dict with unstable iteration order.
- Trailing whitespace differences between requests.
Each produces a 0% hit rate with no error. The application works perfectly and costs several times what it should.
Cache lifetime
Cached prefixes usually expire after a few minutes of inactivity, with each hit refreshing the window. That maps well onto conversations and agent runs, and poorly onto sporadic single requests.
| Workload | Caching value |
|---|---|
| Multi-turn conversation | Very high |
| Agent loop with tool calls | Very high — many turns, identical tools |
| Batch of similar requests | High if run back to back |
| High-traffic shared prompt | High — traffic keeps it warm |
| Occasional one-off requests | Negative — you pay the write premium |
One practical consequence: batch similar work together. Processing a thousand documents with the same instructions in one run keeps the prefix warm throughout. Spreading the same thousand across a day means paying the write premium repeatedly.
Verifying it works
Do not assume. API responses report cache usage explicitly:
A hit ratio near zero on a repetitive workload means something varies in the prefix. Diff two consecutive serialised requests — the difference is usually obvious once you look, and usually a timestamp.
✅ Reordering retrieval helps too
If several requests share the same retrieved documents — a user asking follow-up questions about one contract — put the documents before the question rather than after. The document block then becomes part of the cacheable prefix across the whole exchange.
This conflicts slightly with the lost-in-the-middle guidance about placing instructions last. In practice both are satisfiable: static documents early, instruction and question last, nothing variable in between.
Minimum sizes and breakpoints
Providers impose a minimum cacheable length — commonly around 1,000 tokens — because caching a short prefix costs more to manage than it saves. Prompts below the threshold simply are not cached, silently.
Some APIs require you to mark cache breakpoints explicitly rather than caching automatically. Place them at the boundary between static and variable content:
What else it enables
Caching changes some architectural decisions, not just the bill:
- Longer system prompts become affordable. A detailed 4,000-token instruction block costs almost nothing after the first request, so the usual pressure to trim it disappears.
- More few-shot examples. Twenty examples improve reliability and were previously too expensive to send every time.
- Document-resident workflows. Loading a whole contract once and asking many questions becomes cheap enough to compete with retrieval.
- Lower latency. Skipping prefill on a large prefix is often the largest single improvement to time-to-first-token.
Checking what your prompts contain?
Count characters and words to size prompt components, or format the JSON payloads you send — all in your browser.
Open the Word Counter →Summary
- Caching reuses the KV state for an exact prefix. Prefix only, always.
- Static content first, variable content last. No exceptions.
- Cache reads cost ~10%; writes cost ~125%. It pays from the second hit.
- A timestamp in the system prompt is the classic silent killer.
- Serialise deterministically — sort keys, fix tool order.
- Conversation history caches naturally because turns are appended.
- Batch similar work to keep the prefix warm.
- Log the hit ratio. A silent 0% is the common failure.
Frequently Asked Questions
What is prompt caching?
Storing the model's internal key-value state for a prefix of your prompt, so repeat requests that begin with the same text skip recomputing it. You are billed a reduced rate for cached input, and time-to-first-token drops because the expensive prefill work is skipped.
Why must the cached part be at the beginning?
Because each token's internal representation depends on everything before it. Change one token near the start and every subsequent representation differs, so nothing after the change is reusable. Only an exact matching prefix can be cached.
How much does prompt caching save?
Providers typically bill cached input at 10 to 25% of the normal rate. On an agent where a large system prompt and tool definitions dominate input, total cost commonly falls by half or more, and time-to-first-token improves noticeably.
Why is my cache never hitting?
Almost always because something varies near the start of the prompt — a timestamp, a session ID, a randomised example order, or a user name injected into the system prompt. Any variation invalidates everything after it, so move all dynamic content to the end.
How long does a cached prefix last?
Typically a few minutes of inactivity, with each hit refreshing the window. That suits conversations and agent loops well and suits infrequent one-off requests poorly. Some providers offer longer retention at a higher write cost.