Context windows are marketed as capacity — a million tokens sounds strictly better than two hundred thousand. In practice the window is a shared budget with several competing claimants, the model does not use all of it equally well, and filling it is frequently the wrong move. This page covers what is really happening and how to budget one.
The three things to know
The window holds everything at once — system prompt, tools, history, retrieved documents and the response. Attention cost scales roughly with the square of its length. And models attend well to the start and end of a long context and poorly to the middle.
What competes for the space
| Consumer | Typical size | Sent every request? |
|---|---|---|
| System prompt | 500–2,000 | Yes |
| Tool definitions | 200–400 per tool | Yes |
| Conversation history | Grows without bound | Yes |
| Retrieved documents | 2,000–20,000 | Per request |
| Attached files or images | 1,000+ each | Per request |
| The user's message | 10–500 | Once |
| Reserved for output | 1,000–8,000 | Every request |
Two of these are easy to overlook. Output shares the window — a model with a 128,000-token context and 120,000 tokens of input has only 8,000 left to answer in, and will truncate. And tool definitions are resent constantly: fifteen documented tools can consume 5,000 tokens before anything else.
Why long contexts are expensive and slow
Transformer attention compares every token against every other token. For n tokens that is n² comparisons, so the cost of processing context does not scale linearly:
| Context length | Relative attention work |
|---|---|
| 1,000 | 1× |
| 10,000 | 100× |
| 100,000 | 10,000× |
| 1,000,000 | 1,000,000× |
Real implementations use optimisations — flash attention, sliding windows, sparse patterns — that improve the constants considerably, and the underlying scaling remains superlinear. This is why time-to-first-token climbs sharply with prompt length while generation speed stays roughly constant: the model must process the entire input before producing anything.
The KV cache
During generation, the model caches the key and value vectors for every token it has processed, so each new token does not require recomputing the whole sequence. That cache is what makes generation fast — and it grows linearly with context.
This is why long-context requests are disproportionately expensive to serve, why providers price them accordingly, and why a self-hosted model that runs comfortably at 4k context may fail outright at 100k on the same hardware.
Lost in the middle
The most important practical finding about long contexts is that position matters.
Research on retrieval within long contexts consistently shows a U-shaped performance curve: models retrieve information placed at the beginning or end of the context reliably, and accuracy degrades markedly for information in the middle. In some experiments, performance on middle-positioned facts falls below what the model achieves with no context at all.
The practical implications are direct:
- Put the instruction last. A task description after a long document is followed more reliably than one before it.
- Put critical constraints at both ends. Repetition at the start and end is not waste.
- Rank retrieved passages deliberately. The most relevant chunk should not land in the middle of twenty others.
- Fewer passages beat more. Adding marginal chunks pushes good ones into the weak zone.
⚠️ Needle-in-a-haystack tests overstate the case
Providers demonstrate long-context capability with tests that insert a distinctive sentence into a long document and ask the model to find it. Models score very well on these.
The test is easier than real work. The needle is lexically distinctive — it stands out from surrounding text — and there is exactly one. Real retrieval involves several relevant passages that resemble the surrounding material, require synthesis, and sometimes contradict each other. A high needle score does not predict good performance on that.
Advertised versus effective context
Two different numbers, and only one is published.
| Advertised context | Effective context | |
|---|---|---|
| Definition | Maximum accepted without error | Length where quality holds up |
| Determined by | Architecture and training | Measurement on your task |
| Published? | Prominently | Rarely |
| Relationship | Effective is often a fraction of advertised | |
A model that accepts a million tokens will accept them. Whether it reasons over them as well as it reasons over ten thousand is a separate question, and one you have to answer empirically for your own workload.
✅ Measure it on your data
Build a small evaluation set of questions with known answers drawn from your documents. Run it at several context lengths — 4k, 16k, 64k, 128k — with the answer placed at the start, middle and end each time.
The point where accuracy drops is your effective context for that task. It is usually shorter than expected, it differs by model, and it is the number that should drive your architecture rather than the marketing figure.
Long conversations degrade
A related effect appears in extended agent runs and long chats. As history accumulates, several things compound:
- Early instructions drift into the middle — the weak zone — as the conversation grows around them.
- Superseded information persists. A corrected fact and its correction are both present, and the model may use either.
- Failed attempts accumulate. An agent's unsuccessful tool calls stay in context and can bias later attempts toward repeating them.
- Signal dilutes. The relevant fraction of the context falls steadily.
The counter is compaction: periodically replace older history with a structured summary that preserves decisions, established facts and current state while discarding the transcript that produced them.
Keeping recent turns verbatim matters — summarising everything loses the immediate conversational thread. The usual shape is a summary of old history plus the last several turns in full.
Retrieval versus stuffing
| Put everything in context | Retrieve what is relevant | |
|---|---|---|
| Cost per request | High — you pay for all of it | Low |
| Latency | High | Low |
| Accuracy | Degrades with length | Usually better |
| Scale limit | The window | Effectively unbounded |
| Setup effort | Trivial | Real |
| Whole-document reasoning | Possible | Difficult |
Stuffing wins in one specific case: when the question genuinely requires reasoning across an entire document and cannot be answered from extracts. "Summarise this contract" or "find every inconsistency in this specification" are real examples — no retrieval strategy substitutes for having the whole text.
For everything else — the large majority of question-answering over a corpus — retrieval is cheaper, faster and more accurate. The right architecture for most systems is retrieval by default, with long context reserved for the cases that need it.
💡 A hybrid worth knowing
Retrieve to identify which documents are relevant, then load those documents whole rather than passing only the matched chunks. You get retrieval's efficiency in narrowing the search and long context's advantage in preserving surrounding meaning.
This works well when documents are moderately sized — contracts, reports, specifications — and poorly when they are books.
Budgeting a window
- Reserve output first. Decide the maximum response length and subtract it before anything else.
- Measure your fixed overhead. System prompt plus tool definitions is a constant tax — know the number.
- Cap retrieval. Set a token budget for retrieved content and rank into it, rather than taking the top k regardless of size.
- Bound history. Compact when it exceeds a threshold. Unbounded growth is not a strategy.
- Position deliberately. Instructions last, most relevant retrieval near the ends.
- Cache the static prefix. System prompt and tool definitions are identical every request — prompt caching can cut their cost dramatically.
- Measure quality against length on your own evaluation set, not the advertised figure.
Working with structured data for prompts?
Format and validate JSON, or convert CSV to JSON and back — in your browser, with nothing uploaded.
Open the JSON Formatter →Summary
- The window holds everything — including the response you have not generated yet.
- Attention scales quadratically, so long prompts raise latency sharply.
- The KV cache grows linearly and can exceed the model weights in memory.
- Models attend well at the start and end, poorly in the middle. Position matters.
- Needle tests overstate long-context ability — real retrieval is harder.
- Effective context is shorter than advertised. Measure it yourself.
- Compact long histories rather than letting them grow without bound.
- Retrieve by default; use long context for genuine whole-document reasoning.
Frequently Asked Questions
What is a context window?
The maximum number of tokens a model can consider at once. It holds everything — the system prompt, tool definitions, the full conversation history, any retrieved documents, and the response being generated. All of it competes for the same budget, so the space available for your actual content is always less than the advertised figure.
Does a model use its whole context window equally well?
No. Performance follows a U-shaped curve: models reliably attend to information at the beginning and end of a long context and are measurably weaker in the middle. This is known as the lost-in-the-middle effect and it means where you place information matters, not just whether you included it.
Why does a long prompt make responses slower?
Attention compares every token against every other token, so compute scales with roughly the square of the context length. Doubling the context roughly quadruples the attention work. Memory also scales, because the key-value cache holds an entry for every token processed.
What is the difference between advertised and effective context?
Advertised context is the maximum the model will accept without erroring. Effective context is the length at which it still performs reliably, which is often considerably shorter. A model advertising a million tokens may show noticeably degraded retrieval well before that limit.
Should I use retrieval or just put everything in the context?
Retrieval, in almost every case. Filling a large window costs more on every request, adds latency, and typically produces worse answers than sending a few well-chosen passages. Long context is a capability for problems that genuinely need whole-document reasoning, not a substitute for finding the right information.