What Is Context Compaction and When Should You Use It?

Context compaction is a server-side process that manages the context window of long-running LLM sessions by distilling history into high-fidelity summaries.

When a conversation approaches its token limit, the system packs the state that matters, the decisions already made, and the next steps into one condensed block, and then restarts the window with that block standing in for the raw transcript. The model keeps a working memory that still functions, and it does so without the slow, expensive decay that a massive unmanaged history drags along with it.

You want context compaction in agentic workflows and multi-turn sessions, the ones where a task has to run on its own for a long stretch. Information snowballs fast when an agent iterates through tool calls and sub-tasks, because every result it reads becomes part of the prompt for everything it does afterwards. Without compaction your system eventually hits a hard context limit or, far more often, a soft pricing cliff where the cost and latency of each new turn are worth more than whatever that turn produces.

Compaction shifts the context window from a growing pile of logs into an actively managed engineering handoff.

That matters if you are building agents that have to stay fast over hundreds of turns, like autonomous coders or deep-research tutors, where keeping technical continuity beats preserving every half-finished thought the model had along the way.

A long raw conversation history distilled into one structured summary block inside the context window

What is context compaction?

Context compaction is the automated distillation of conversation history so a session does not overflow its context window. Instead of letting the context grow until it slams into a hard cap, the system watches for a trigger threshold — usually a specific volume of tokens — and writes a structured account of where the session stands. That summary goes into the prompt as a "compaction block," and the raw messages that came before it are dropped from the active window.

Compaction versus truncation: compaction keeps constraints and state, truncation mechanically slices the front of the sequence

This is not naive truncation, which slices tokens at a boundary and cheerfully cuts off system instructions or the architectural decision someone made in the first five minutes. It is also more than prose summarization, which is usually too lossy to survive engineering work. Good compaction behaves like a handoff note to the next person on shift: it keeps the current system state, the specific technical decisions, and the issues still open, while it throws away exploratory dead ends and the tool output nobody needs to read twice.

In systems that run the compact_20260112 strategy, compaction happens server-side, and that is the part I like most about it. You do not have to write client-side logic that summarizes history and re-injects it by hand. The API watches the token count, generates the <summary> block, and rebuilds the next prompt for you, so the model keeps working from a lean, high-signal context.

The payoff is continuity: hundreds of thousands of tokens of raw logs get replaced by a few thousand tokens of state. That is context engineering in one sentence, because the goal is to give the model the smallest set of tokens that still gives it a good chance of finishing the job.

Why long context becomes an engineering problem

As context grows, LLMs suffer from context rot: recall accuracy, the thing the "needle-in-a-haystack" tests measure, degrades badly. Accuracy starts around 75% and sags into the mid-forties for facts sitting in the middle of a massive window. This happens even when the model technically "supports" a window that large, because the attention budget is finite, and the competition for it means buried constraints and prior failures get missed.

The three costs of long context: falling recall accuracy, rising latency, and a growing token bill

The money side is just as bad. Without optimization, every token in a growing window is re-processed and re-billed on every single API call, so in a multi-turn agentic loop the cost of the session scales quadratically. Ignore context management and you will find the pricing cliff on your own: the value of a single turn gets eclipsed by the cost of re-sending a history that is 90% exploratory noise.

Latency is the third bottleneck, and it is the one your users actually feel. Time to first token (TTFT) climbs linearly as the input prefix balloons, because the model has to read the entire window before it writes a single token back. Median TTFT sits at roughly 22 seconds below 100,000 input tokens and around 76 seconds above 800,000. That gap is the difference between a pause a person will sit through and an autonomous loop that crawls.

Finally, unmanaged context turns into a signal-to-noise failure. When an agent reads dozens of documents or produces 300-line stack traces, those tool results start to dominate the input, and the actual task instructions and user intent shrink to a tiny fraction of the total tokens. Without compaction to clear out those payloads, the model spreads its reasoning across too many irrelevant tokens, which is how you end up with logic mistakes and mission drift.

How compaction works

The mechanism starts with trigger-threshold detection. The API watches input tokens against a pre-set value, which defaults to 150,000 tokens and can be configured down to a minimum of 50,000. When that limit is breached, the system pauses normal generation and runs a compaction cycle: the model reads the visible transcript and produces a distilled summary wrapped in <summary> tags, capturing the plot so far, the current state, and the immediate next steps.

The four steps of the server-side compaction cycle: threshold detection, summarization, compaction block, and response

Once the compaction block exists, the API drops the message history that came before it. To keep continuity you have to append the assistant's response, compaction block included, to your message array, and every request after that uses the new, shortened prompt. There is a warning here that deserves more attention than it usually gets: on reasoning models like Claude Fable 5.1 and Claude Mythos 5.1, thinking blocks from before the compaction event are not carried forward. The summary is all the model has left of its own prior reasoning, so your summary instructions need to be explicit about what state to preserve.

You can tune the behavior with the pause_after_compaction parameter. With it enabled, the API stops as soon as it has generated the summary and returns a stop_reason of compaction, which gives you a window to inspect the state, inject custom instructions, or hand-pick a few recent messages to keep before continuing. If you care whether the agent still knows what it was doing after a reset, this is the control you want.

You can also give the summarizer custom instructions. Instead of the default prompt, you might tell it to "focus on preserving code snippets, variable names, and architectural decisions," so that the lossy part of the summary eats conversational noise rather than the technical detail the agent needs in order to keep working in a real codebase.

Common context compaction strategies

The basic approach is the sliding window: keep the last N turns of history plus the core system instructions, and drop the oldest data entirely. It is computationally free and it is fast, but it forgets, and an agent that forgets a constraint someone set an hour ago will break that constraint without ever noticing. Treat it as a trivial-tier intervention, fine for stateless tasks and not much else.

Four compaction strategies: sliding window, structured state, tool output offloading, and staged compaction

Structured state management is the more advanced pattern, where raw transcripts give way to a JSON scratchpad. The agent maintains an object tracking goals, facts, and variable states, and at each turn the raw history is discarded so the model receives only its instructions and the updated JSON. This is very token-efficient, though it leans hard on your ability to define a schema that captures every variable that turns out to matter, which is rarely something you know up front.

Tool output offloading is the reversible one. Large payloads, say a 400-line log file, are moved to an external store or a memory tool, and the agent is left holding a small reference pointer or summary in its context. If the model needs the raw data later, it re-fetches it with a tool call. The active window stays lean while no technical signal is permanently destroyed, which is why this is usually the first thing I would reach for.

Staged compaction is a ladder of interventions keyed to how much pressure the window is under. It starts with observation truncation, cheaply cutting a tool's output down to its head and tail with no LLM call at all. If context is still high, it moves to tool-result clearing, which surgically removes old payloads while keeping the record that the call happened. Only when those non-destructive steps run out does the system fall back to full lossy summarization of the entire transcript.

Reversible compaction versus lossy summarization

Reversible compaction removes information from the context that still exists somewhere else, in a file system or a vector database. Because the agent can re-fetch it with tools like grep or read_file, the operation is safe: the active window shrinks, the underlying signal survives, and the agent goes back for a specific needle only when a sub-task genuinely needs it.

Reversible compaction versus lossy summarization: data that can be re-fetched versus data destroyed for good

Lossy summarization is the opposite trade. Anything not captured in the summary block is destroyed for good, and once a raw transcript has been flattened into prose, the exact phrasings, the exploratory dead ends, and the specific numbers are gone with it. If the summarizer decides a detail is unimportant and that detail turns out to matter three turns later, the agent has no way to get it back. That is why lossy summarization belongs at the end of your list of options rather than the start.

There is also a real distinction between tool-result clearing and whole-transcript compaction, and it is worth keeping straight. Clearing is a surgical sub-transcript operation that targets only the bulky payloads of tool messages. Compaction is a whole-transcript operation that flattens everything, user messages, assistant reasoning, and prior summaries alike, into a single block. Clearing shrinks the context without rewriting the history, whereas compaction restarts the history from a new baseline.

The tradeoff underneath all of this is precision versus capacity. Reversible methods win when the agent might need to look again at raw data or a precise API response later in the session. Lossy summarization is what you fall back on when the context window is physically or economically full and the agent only needs the gist of its own progress to finish what is left.

Compaction, bigger context windows, and RAG

It is a mistake to assume that million-token context windows make compaction obsolete. Larger windows raise the capacity ceiling while making context rot and latency worse at the same time, which is not the trade most people think they are getting. Even if a whole session fits inside a 1M-token window, re-processing that window on every turn is slow enough and expensive enough to be indefensible. Compaction is what lets you use that capacity instead of merely paying for it.

A layered architecture: RAG for the corpus, a large context window for capacity, and compaction for session state

If you are choosing between stuffing the whole document, retrieving, and compacting for a particular workload, the head-to-head comparison lives in long-context processing.

Compaction and retrieval-augmented generation (RAG) do related but different jobs. RAG pulls external knowledge out of a corpus far too large to ever fit in a window, while compaction manages the agent's working memory inside the window. You use RAG to find the information, and compaction to stop the record of all that searching from crowding out the model's ability to reason over what it found.

A serious architecture layers all three: RAG for the corpus, a large context window for capacity during heavy reasoning bursts, and compaction to prune the session state as it goes. Otherwise the agent slowly buries itself under the weight of its own earlier thoughts.

Advanced agents also use direct corpus interaction, where the model runs terminal tools like grep or ls against a raw corpus. This often beats vector-only RAG, because the agent can find a specific variable name or code symbol that embeddings would miss. Compaction then clears the results of those terminal searches, so the exploratory logs do not clutter the window once the fact has been found and written into the agent's notes.

What breaks when compaction goes wrong

Over-compaction is one of the hardest context failure modes to spot. The primary failure mode is digital amnesia. If the summary block is too aggressive, the agent loses the context of its own previous failures, and you get infinite loops where it repeats the same mistaken tool call because it has forgotten that the approach already failed five turns ago. Without a record of past errors the agent cannot learn anything inside the session, which defeats most of the reason you gave it a long session in the first place.

The blind judge failure: answer-quality scores stay high while memory recall of session facts drops

Compaction also puts turn 1 rules at risk. A user establishes a strict constraint at the start of a session, something like "never use library X." After several compactions, that turn 1 instruction can be summarized away or buried so deep in the compaction block that the model breaks it at turn 45 while still sounding perfectly on-topic about it.

The nastiest problem is the blind judge. Evaluations show that while answer quality scores often stay high, at 97% to 99%, memory recall of session facts in that same screen sat at 58%. The agent hands you a well-formatted, confident, plausible answer that happens to be factually wrong, because it quietly ignored information you provided earlier. Since the failure is not a visible crash but a confident hallucination, you will not catch it without rigorous fact-probing evals, and that combination of high confidence and low recall is the one I would worry about most.

Successive compactions compound the loss, because summarizing a summary is a recursive lossy operation. By the fifth or sixth compaction event, the original mission constraints can be distorted or gone entirely, and what you are left with is an agent hallucinating its own progress from a vague, degraded history.

When should compaction trigger?

The standard technical trigger is a token-count threshold, with a default of 150,000. That number balances the need for enough raw history to keep nuance against the latency and cost of a massive prefix. It is a reasonable default, but I would not stop there, because you can pair it with a budget trigger built on pause_after_compaction.

In a budget trigger, you catch the stop_reason of compaction, increment a counter, and check whether the number of compactions multiplied by the trigger threshold has reached your total token budget. Your system can then decide on its own whether to continue the task or to instruct the agent to wrap up its current work and provide a final summary before the session becomes prohibitively expensive.

The $0.55 per 1M token threshold is the economic crossover point, and it is the number most people skip straight past. Because compaction rewrites the prompt prefix, it invalidates the KV-cache, the stored attention state that lets a provider skip re-processing a prompt prefix it has already seen. On providers with deep prompt caching discounts, like DeepSeek, where the cached rate drops from $0.14 to $0.0028 per million tokens, or Gemini at roughly 90% off, compacting too early can multiply your costs instead of cutting them. You compact when the cost of re-processing a new, smaller prefix is lower than the cost of continuing to send the full, cached history, and not one turn before that.

Triggers should also be context-aware. If a session is dominated by tool results, say the agent has just read a 100k-token documentation file, you should trigger tool clearing or compaction as soon as the model has extracted the relevant notes. The exploratory data then leaves the window before it can cause context rot or add unnecessary billing to every reasoning turn that follows.

When compaction is not worth it

Compaction is frequently an economic trap, and prompt caching is the reason why. Because a summary creates a brand-new prefix, it invalidates the existing KV-cache and forces the provider to re-process the summary and the messages after it at the full raw price. On providers like DeepSeek or Gemini, where cache hits are up to 90% cheaper, a summary has to shrink the context substantially just to break even on cost. Compact 200k tokens down to 50k and lose your 90% discount, and you have effectively paid more for a dumber model.

So the rule is: don't compact by default. Only trigger compaction when you hit a hard architectural wall, meaning the context window limit itself, when the cost of cached input finally crosses the $0.55/1M token threshold, or when you have empirical evidence in front of you that context rot is causing task failure. In most production environments, keeping the full history and using prompt caching is faster, cheaper, and holds reasoning fidelity better than any summary you could write in its place.