Long-Context Processing: Stuff, Retrieve, or Compact?

Long-context processing is the technical management of high-token inputs in large language models (LLMs)

— sequences that typically range from 32,000 to over one million tokens. There is no single best method for handling them. Stuffing, where you drop the whole document into the prompt and let the model sort it out, frequently fails because internal reasoning degrades as the sequence grows, while retrieval-augmented generation (RAG) stays limited by document relevance and noise. Automated compaction, or summarization, is what holds performance up in long-running agentic systems.

Treat the context window as a constrained attention budget rather than a container. Tokens are not equal once they are in there, because the value of each one has to be weighed against the model's shrinking ability to hold focus. Vendors advertise the window in millions, but your infrastructure has to account for a performance gradient that starts long before that number.

That gradient comes from the architecture, not from a bug anyone can patch: as you scale token counts, the transformer's self-attention mechanism becomes a bottleneck for reasoning. Your job is to maximize signal while aggressively pruning noise, so the model's reasoning stays precise and cost-effective across extended horizons.

A long context stream feeding a language model, with the important signal thinning out as the token count grows

Why do models get worse as the context grows?

The main driver of that degradation is context rot, and it comes straight from the transformer's architecture, which relies on n² pairwise relationships for n tokens. Push the context length up and the model's ability to capture those dependencies gets stretched thin, which is what people mean by attention scarcity: the budget for focus is finite, and every new token you introduce spends some of it, until the model drifts off the primary task.

A model's attention budget spread thinner as token count grows, with pairwise relationships expanding quadratically

Two kinds of noise do the damage, and they are worth separating. Distractors sit close to the needle of information topically but do not answer the query, whereas irrelevant content has nothing to do with it at all. Models learn their attention patterns from training data where shorter sequences dominate, so they carry fewer specialized parameters for extreme sequence lengths, which leaves them open to interference from both distractors and general context noise.

The shape of the input matters too, and this is where the research turned counterintuitive for me: logical coherence in the haystack — the irrelevant background text — can hurt performance more than shuffled, non-continuous text does. I had assumed well-formed filler would be the easier kind to skip. The reading that fits the data is that the model is oversensitive to the logical flow of irrelevant information, so it spends its attention budget parsing the story of the noise instead of isolating the high-signal tokens.

Model families also fail differently under the same pressure. Claude Sonnet 4 and Opus 4 are the conservative ones: they abstain when uncertain and say outright that no answer can be found. GPT models, in the same tests, show the highest hallucination rates and return confident but incorrect responses when distractors are present. That difference should decide your guardrails, because catching an abstention is a retry and catching a confident fabrication is a whole evaluation pipeline.

How does effective context length differ from the advertised number?

In systematic testing, reasoning accuracy degraded by 13.9% to 85% as input length increased, and those inputs stayed well within the models' claimed limits. So you cannot assume that a model capable of ingesting 128K or 1M tokens can reason across them with the precision it applies to a short sequence. The advertised window measures ingestion capacity, while reasoning reliability is a second number nobody prints on the spec sheet.

Advertised context window compared with effective context length, where accuracy starts falling long before the limit

Take Llama-3.1-8B Instruct, which claims a 128K window. Extend its input to 30,000 tokens and its accuracy on MMLU, a broad multiple-choice knowledge benchmark, falls by 24.2 percentage points against the short-context baseline. The same 30,000-token condition costs it 85.0 points on variable summation and 47.6 on HumanEval, while Mistral-v0.3-7B loses 34.2 points on GSM8K and 34.8 on HumanEval. These are drops measured in percentage points from a zero-context baseline, not relative percentages, and none of these inputs came close to the models' stated ceilings.

The damage comes from sheer length rather than just the presence of distracting information. Experiments using whitespace and token masking show that widening the distance between the evidence and the question is enough on its own to make models fail: padding the gap with thousands of semantically empty whitespace tokens dragged Llama's variable-summation score from 96.0 down to 48.0. Nothing in that padding carries meaning, so it is the distance itself, and not the content of the distance, that impairs the model's ability to link the query to the data it needs.

None of this arrives as a cliff, though, because it follows a gradient, and accuracy begins to erode long before the model hits its technical token limit. What you actually want to measure is your system's effective reasoning length: the point where accuracy falls below your acceptable threshold for production. In practice that point sits well below the number printed on the model's spec sheet.

Is more accurate retrieval enough?

The common engineering assumption is that if you solve retrieval — the needle-in-a-haystack problem — you have solved long-context performance. The perfect-retrieval result kills that idea: at 30,000 tokens, Llama-3.1-8B Instruct recited the correct evidence verbatim for 970 of 1,000 MMLU problems, and its answer accuracy still fell 24.2 points. The model found the sentence, quoted the sentence, and then answered wrong. Retrieval is a necessary condition for long-context reasoning and nowhere near a sufficient one.

A model reciting the correct evidence yet still answering wrongly, showing retrieval is necessary but not sufficient

Standard benchmarks overestimate model progress for exactly this reason, because they isolate retrieval as a standalone capability. In real applications, identifying the information is only the first step, and synthesizing it into a response is the step that fails. When you provide a model with the correct context but wrap it in 100,000 irrelevant tokens, its working memory is effectively diluted, so the reasoning mechanism never links the evidence even though it can see it sitting there in the prompt.

The lost-in-the-middle effect complicates this further: models favor information at the extreme beginning or end of the context. But placing evidence in those best positions does not eliminate the degradation caused by overall input length. If your task requires the model to aggregate three pieces of evidence scattered across a long document, the cost of maintaining those references across a massive attention span frequently produces hallucinations or logic errors.

So read these failures as a breakdown in the model's ability to use information, not to find it. If your system requires complex reasoning over multiple points of evidence, stuffing the window will give you worse answers, and improving your vector database retrieval scores will not rescue a model that is struggling with the sheer volume of its own attention weights. The fix sits upstream of retrieval quality: deliver less context to the reasoning engine, with more signal in it.

When does stuffing the whole document win, and when does retrieval win?

The choice between stuffing and RAG depends heavily on your task. Stuffing generally outperforms RAG in scenarios requiring holistic document understanding, and the advantage is clearest on Wikipedia-based question answering. When the task requires reasoning across an entire text where any detail might matter, providing the full context is often superior, because the model has to see how sections relate to each other and a retriever hands it those sections in isolation.

Stuffing the whole document compared with retrieval, and the task types where each one wins

RAG suits the opposite shape: dialogue-based and general queries, where knowledge is fragmented rather than continuous. Filtering out irrelevant documents prevents context pollution, which means the model's limited attention budget is not spent on noise before it reaches the part that matters. Against a massive, static dataset where only a fraction of the information is relevant to any single query, RAG is your best defense, and it keeps the context window tight enough that reasoning precision holds up.

When you do stuff the window, a retrieve-then-solve strategy closes some of the gap: prompt the model to explicitly recite the retrieved evidence before attempting to answer. On RULER's QA2 task this bought GPT-4o up to 4 percentage points at 32K, on top of a baseline that was already strong. Forcing the model to output the evidence first converts a long-context problem into a high-precision, short-context one inside the model's immediate generation window, and it is a cheap trick for the accuracy it buys.

Let the nature of the information govern the decision. If the signal is a needle in a haystack, use RAG; if the signal is the haystack itself — a legal contract whose clauses must be cross-referenced, a large source file — lean toward stuffing or compaction. Note that chunking is where RAG loses coherence, so the more the answer depends on connections across sections, the worse a chunked pipeline serves you.

The real cost of each strategy: price, latency, and caching

Accuracy is not the only thing at stake when you manage long contexts. Compaction requires an additional sampling step to generate summaries, and that step counts toward your rate limits and your bill. You can monitor it through the iterations array in API responses, which distinguishes the compaction iteration from the final message iteration. Because compaction is an extra model call, the request that triggers it pays for it in both raw cost and latency.

The real cost of long context: time to first token, the API bill, and where to place cache breakpoints

Prompt caching is where you get some of that back, if you place the breakpoints deliberately. By putting cache_control breakpoints at the end of your system prompt, you can keep cache hits even across compaction events; leave the breakpoint out and every new dynamic compaction summary invalidates everything cached before it. Caching the static instructions separately means you only pay to write the new summary while still hitting cache for the heavy system instructions.

Latency trade-offs also exist between just-in-time and pre-inference retrieval. Progressive disclosure, where an agent dynamically loads context through tools as it needs it, is slower than loading everything upfront, but it stops the model from drowning in data it never asked for. This mirrors how people actually research: you hold a tight working memory and keep a broader index of where things live. For massive codebases or document stores, just-in-time loading is the more scalable default.

All of this depends on understanding your caching behavior in detail. If you frequently update the compacted state of a conversation, caching costs can spike when breakpoints are placed carelessly. Isolate high-churn content — recent messages, fresh summaries — from low-churn content like system instructions and tool definitions, so the bulk of your context stays warm.

How do summarization and compaction work in long-running tasks?

Server-side compaction is the recommended mechanic for agents that outrun their reasoning limits. You configure a trigger parameter to define when compaction begins: the default is 150,000 input tokens, and the lowest value you may set is 50,000. The lifecycle after that is fixed, in that the API detects the threshold, generates a summary, creates a compaction block, and drops the older messages. To preserve the compacted state you must append the entire assistant response containing that block to your message history.

The four-step compaction lifecycle: detect the token threshold, generate a summary, create a compaction block, continue

Custom instructions completely replace the default summarizer rather than supplementing it. This detail bites hard: if you supply instructions to focus on code, you must also restate the need to preserve general state and next steps, or you lose the continuity guidance baked into the default prompt. Compaction can also fail outright when tools are defined, because the model sometimes calls a tool during the internal summarization step instead of writing the summary, and you get a compaction block with content: null. The fix is to tell it explicitly, in instructions, not to call tools during that step.

On Claude Fable 5.1 and Claude Mythos 5.1, thinking blocks from before a compaction block are not carried forward, so the summary is all the model retains of that earlier reasoning. On Claude Fable 5.1 specifically, if you re-insert older assistant turns after the compaction block, strip their thinking and redacted_thinking blocks first. The pause_after_compaction parameter lets your system stop right after the summary is generated, which is the hook you want if you intend to intervene before the agent continues.

Structured note-taking sits alongside automated compaction rather than competing with it. Here the agent maintains its own state in a persistent file — a NOTES.md, a to-do list — that gets passed back into the context window, so it can track progress across tasks spanning thousands of tool calls. By reading its own notes after a context reset, it holds a multi-hour migration or research project together even though the window has already been wiped once.

Which strategy should you choose for your system?

Two things decide it: the nature of the dataset and the duration of the workflow. A vast, static dataset wants RAG, which keeps the context window tight and focused on relevant segments. An agentic workflow or a long-running dialogue, where the history itself is the context, wants server-side compaction to hold state without letting context rot set in. And for anything long enough to survive a reset, add structured note-taking so the agent can rebuild its own state, then push heavy reading into a sub-agent that returns a summary rather than a transcript.

The whole of context engineering comes down to finding the smallest possible set of high-signal tokens the task can run on, and the levers are dull ones: minimize the distance between evidence and query, prune what the answer does not need, and stop treating the window size as a target to fill. The 970-of-1,000 result is the one worth remembering, because the model quoted the right evidence almost every time and still lost 24.2 points on the answer.