# What Is Context Compaction and When Should You Use It?

> Understand context compaction to optimize agentic workflows. Learn to balance memory recall, latency, and costs across long-running LLM sessions.

Source: https://camnangai.com/en/engineering/what-is-context-compaction  
Published: 2026-09-06

**Context compaction** is a server-side process that manages the [context
  window](https://camnangai.com/en/knowledge/context-window) of long-running [LLM](https://camnangai.com/en/knowledge/what-is-an-llm) sessions
  by distilling history into high-fidelity summaries.

When a conversation approaches its token limit, the system packs the state that matters, the
decisions already made, and the next steps into one condensed block, and then restarts the window
with that block standing in for the raw transcript. The model keeps a working memory that still
functions, and it does so without the slow, expensive decay that a massive unmanaged history drags
along with it.

You want context compaction in agentic workflows and multi-turn sessions, the ones where a task has
to run on its own for a long stretch. Information snowballs fast when an
[agent](https://camnangai.com/en/knowledge/what-is-an-ai-agent) iterates through tool calls and sub-tasks, because every
result it reads becomes part of the prompt for everything it does afterwards. Without compaction your
system eventually hits a hard context limit or, far more often, a soft pricing cliff where the cost
and latency of each new turn are worth more than whatever that turn produces.


  Compaction shifts the context window from a growing pile of logs into an actively managed
  engineering handoff.

That matters if you are building agents that have to stay fast over hundreds of turns, like
autonomous coders or deep-research tutors, where keeping technical continuity beats preserving every
half-finished thought the model had along the way.

![A long raw conversation history distilled into one structured summary block inside the context window](https://camnangai.com/articles/nen-ngu-canh-la-gi/01-infographic-en.webp)

## What is context compaction?

**Context compaction** is the automated distillation of conversation history so a session does not
overflow its context window. Instead of letting the context grow until it slams into a hard cap, the
system watches for a trigger threshold — usually a specific volume of
[tokens](https://camnangai.com/en/knowledge/what-are-ai-tokens) — and writes a structured account of where the session
stands. That summary goes into the prompt as a "compaction block," and the raw messages that came
before it are dropped from the active window.

![Compaction versus truncation: compaction keeps constraints and state, truncation mechanically slices the front of the sequence](https://camnangai.com/articles/nen-ngu-canh-la-gi/02-en.webp)

This is not naive truncation, which slices tokens at a boundary and cheerfully cuts off system
instructions or the architectural decision someone made in the first five minutes. It is also more
than prose summarization, which is usually too lossy to survive engineering work. Good compaction
behaves like a handoff note to the next person on shift: it keeps the current system state, the
specific technical decisions, and the issues still open, while it throws away exploratory dead ends
and the tool output nobody needs to read twice.

In systems that run the `compact_20260112` strategy, compaction happens server-side, and that is the
part I like most about it. You do not have to write client-side logic that summarizes history and
re-injects it by hand. The API watches the token count, generates the `<summary>` block, and rebuilds
the next prompt for you, so the model keeps working from a lean, high-signal context.

The payoff is continuity: hundreds of thousands of tokens of raw logs get replaced by a few thousand
tokens of state. That is context engineering in one sentence, because the goal is to give the model
the smallest set of tokens that still gives it a good chance of finishing the job.

## Why long context becomes an engineering problem

As context grows, LLMs suffer from [context rot](https://camnangai.com/en/for/developers/compact-command-context-rot):
recall accuracy, the thing the "needle-in-a-haystack" tests measure, degrades badly. Accuracy starts
around 75% and sags into the mid-forties for facts sitting in the middle of a massive window. This
happens even when the model technically "supports" a window that large, because the attention budget
is finite, and the competition for it means buried constraints and prior failures get missed.

![The three costs of long context: falling recall accuracy, rising latency, and a growing token bill](https://camnangai.com/articles/nen-ngu-canh-la-gi/03-en.webp)

The money side is just as bad. Without optimization, every token in a growing window is re-processed
and re-billed on every single API call, so in a multi-turn agentic loop the cost of the session
scales quadratically. Ignore context management and you will find the pricing cliff on your own: the
value of a single turn gets eclipsed by the cost of re-sending a history that is 90% exploratory
noise.

Latency is the third bottleneck, and it is the one your users actually feel. Time to first token
(TTFT) climbs linearly as the input prefix balloons, because the model has to read the entire window
before it writes a single token back. Median TTFT sits at roughly 22 seconds below 100,000 input
tokens and around 76 seconds above 800,000. That gap is the difference between a pause a person will
sit through and an autonomous loop that crawls.

Finally, unmanaged context turns into a signal-to-noise failure. When an agent reads dozens of
documents or produces 300-line stack traces, those tool results start to dominate the input, and the
actual task instructions and user intent shrink to a tiny fraction of the total tokens. Without
compaction to clear out those payloads, the model spreads its reasoning across too many irrelevant
tokens, which is how you end up with logic mistakes and mission drift.

## How compaction works

The mechanism starts with trigger-threshold detection. The API watches input tokens against a pre-set
value, which defaults to 150,000 tokens and can be configured down to a minimum of 50,000. When that
limit is breached, the system pauses normal generation and runs a compaction cycle: the model reads
the visible transcript and produces a distilled summary wrapped in `<summary>` tags, capturing the
plot so far, the current state, and the immediate next steps.

![The four steps of the server-side compaction cycle: threshold detection, summarization, compaction block, and response](https://camnangai.com/articles/nen-ngu-canh-la-gi/04-en.webp)

Once the compaction block exists, the API drops the message history that came before it. To keep
continuity you have to append the assistant's response, compaction block included, to your message
array, and every request after that uses the new, shortened prompt. There is a warning here that
deserves more attention than it usually gets: on reasoning models like Claude Fable 5.1 and Claude
Mythos 5.1, thinking blocks from before the compaction event are not carried forward. The summary is
all the model has left of its own prior reasoning, so your summary instructions need to be explicit
about what state to preserve.

You can tune the behavior with the `pause_after_compaction` parameter. With it enabled, the API stops
as soon as it has generated the summary and returns a `stop_reason` of `compaction`, which gives you
a window to inspect the state, inject custom instructions, or hand-pick a few recent messages to keep
before continuing. If you care whether the agent still knows what it was doing after a reset, this is
the control you want.

You can also give the summarizer custom instructions. Instead of the default prompt, you might tell
it to "focus on preserving code snippets, variable names, and architectural decisions," so that the
lossy part of the summary eats conversational noise rather than the technical detail the agent needs
in order to keep working in a real codebase.

## Common context compaction strategies

The basic approach is the sliding window: keep the last N turns of history plus the core system
instructions, and drop the oldest data entirely. It is computationally free and it is fast, but it
forgets, and an agent that forgets a constraint someone set an hour ago will break that constraint
without ever noticing. Treat it as a trivial-tier intervention, fine for stateless tasks and not much
else.

![Four compaction strategies: sliding window, structured state, tool output offloading, and staged compaction](https://camnangai.com/articles/nen-ngu-canh-la-gi/05-en.webp)

Structured state management is the more advanced pattern, where raw transcripts give way to a JSON
scratchpad. The agent maintains an object tracking goals, facts, and variable states, and at each
turn the raw history is discarded so the model receives only its instructions and the updated JSON.
This is very token-efficient, though it leans hard on your ability to define a schema that captures
every variable that turns out to matter, which is rarely something you know up front.

Tool output offloading is the reversible one. Large payloads, say a 400-line log file, are moved to
an external store or a memory tool, and the agent is left holding a small reference pointer or
summary in its context. If the model needs the raw data later, it re-fetches it with a tool call. The
active window stays lean while no technical signal is permanently destroyed, which is why this is
usually the first thing I would reach for.

Staged compaction is a ladder of interventions keyed to how much pressure the window is under. It
starts with observation truncation, cheaply cutting a tool's output down to its head and tail with no
LLM call at all. If context is still high, it moves to tool-result clearing, which surgically removes
old payloads while keeping the record that the call happened. Only when those non-destructive steps
run out does the system fall back to full lossy summarization of the entire transcript.

## Reversible compaction versus lossy summarization

Reversible compaction removes information from the context that still exists somewhere else, in a
file system or a vector database. Because the agent can re-fetch it with tools like `grep` or
`read_file`, the operation is safe: the active window shrinks, the underlying signal survives, and
the agent goes back for a specific needle only when a sub-task genuinely needs it.

![Reversible compaction versus lossy summarization: data that can be re-fetched versus data destroyed for good](https://camnangai.com/articles/nen-ngu-canh-la-gi/06-en.webp)

Lossy summarization is the opposite trade. Anything not captured in the summary block is destroyed
for good, and once a raw transcript has been flattened into prose, the exact phrasings, the
exploratory dead ends, and the specific numbers are gone with it. If the summarizer decides a detail
is unimportant and that detail turns out to matter three turns later, the agent has no way to get it
back. That is why lossy summarization belongs at the end of your list of options rather than the
start.

There is also a real distinction between tool-result clearing and whole-transcript compaction, and it
is worth keeping straight. Clearing is a surgical sub-transcript operation that targets only the
bulky payloads of tool messages. Compaction is a whole-transcript operation that flattens everything,
user messages, assistant reasoning, and prior summaries alike, into a single block. Clearing shrinks
the context without rewriting the history, whereas compaction restarts the history from a new
baseline.

The tradeoff underneath all of this is precision versus capacity. Reversible methods win when the
agent might need to look again at raw data or a precise API response later in the session. Lossy
summarization is what you fall back on when the context window is physically or economically full and
the agent only needs the gist of its own progress to finish what is left.

## Compaction, bigger context windows, and RAG

It is a mistake to assume that million-token context windows make compaction obsolete. Larger windows
raise the capacity ceiling while making context rot and latency worse at the same time, which is not
the trade most people think they are getting. Even if a whole session fits inside a 1M-token window,
re-processing that window on every turn is slow enough and expensive enough to be indefensible.
Compaction is what lets you use that capacity instead of merely paying for it.

![A layered architecture: RAG for the corpus, a large context window for capacity, and compaction for session state](https://camnangai.com/articles/nen-ngu-canh-la-gi/07-en.webp)

If you are choosing between stuffing the whole document, retrieving, and compacting for a particular
workload, the head-to-head comparison lives in
[long-context processing](https://camnangai.com/en/engineering/long-context-processing).

Compaction and [retrieval-augmented generation (RAG)](https://camnangai.com/en/knowledge/what-is-rag) do related but
different jobs. RAG pulls external knowledge out of a corpus far too large to ever fit in a window,
while compaction manages the agent's
[working memory](https://camnangai.com/en/engineering/what-is-agent-state) inside the window. You use RAG to find the
information, and compaction to stop the record of all that searching from crowding out the model's
ability to reason over what it found.

A serious architecture layers all three: RAG for the corpus, a large context window for capacity
during heavy reasoning bursts, and compaction to prune the session state as it goes. Otherwise the
agent slowly buries itself under the weight of its own earlier thoughts.

Advanced agents also use direct corpus interaction, where the model runs terminal tools like `grep`
or `ls` against a raw corpus. This often beats vector-only RAG, because the agent can find a specific
variable name or code symbol that embeddings would miss. Compaction then clears the results of those
terminal searches, so the exploratory logs do not clutter the window once the fact has been found and
written into the agent's notes.

## What breaks when compaction goes wrong

Over-compaction is one of the hardest
[context failure modes](https://camnangai.com/en/engineering/context-failure-modes) to spot. The primary failure mode is
digital amnesia. If the summary block is too aggressive, the agent loses
the context of its own previous failures, and you get infinite loops where it repeats the same
mistaken tool call because it has forgotten that the approach already failed five turns ago. Without
a record of past errors the agent cannot learn anything inside the session, which defeats most of the
reason you gave it a long session in the first place.

![The blind judge failure: answer-quality scores stay high while memory recall of session facts drops](https://camnangai.com/articles/nen-ngu-canh-la-gi/08-en.webp)

Compaction also puts turn 1 rules at risk. A user establishes a strict constraint at the start of a
session, something like "never use library X." After several compactions, that turn 1 instruction can
be summarized away or buried so deep in the compaction block that the model breaks it at turn 45
while still sounding perfectly on-topic about it.

The nastiest problem is the blind judge. Evaluations show that while answer quality scores often stay
high, at 97% to 99%, memory recall of session facts in that same screen sat at 58%. The agent hands
you a well-formatted, confident, plausible answer that happens to be factually wrong, because it
quietly ignored information you provided earlier. Since the failure is not a visible crash but a
confident hallucination, you will not catch it without rigorous fact-probing evals, and that
combination of high confidence and low recall is the one I would worry about most.

Successive compactions compound the loss, because summarizing a summary is a recursive lossy
operation. By the fifth or sixth compaction event, the original mission constraints can be distorted
or gone entirely, and what you are left with is an agent hallucinating its own progress from a vague,
degraded history.

## When should compaction trigger?

The standard technical trigger is a token-count threshold, with a default of 150,000. That number
balances the need for enough raw history to keep nuance against the latency and cost of a massive
prefix. It is a reasonable default, but I would not stop there, because you can pair it with a budget
trigger built on `pause_after_compaction`.

In a budget trigger, you catch the `stop_reason` of `compaction`, increment a counter, and check
whether the number of compactions multiplied by the trigger threshold has reached your total token
budget. Your system can then decide on its own whether to continue the task or to instruct the agent
to wrap up its current work and provide a final summary before the session becomes prohibitively
expensive.

The $0.55 per 1M token threshold is the economic crossover point, and it is the number most people
skip straight past. Because compaction rewrites the prompt prefix, it invalidates the KV-cache, the
stored attention state that lets a provider skip re-processing a prompt prefix it has already seen.
On providers with deep [prompt caching](https://camnangai.com/en/knowledge/prompt-caching-explained) discounts, like
DeepSeek, where the cached rate drops from $0.14 to $0.0028 per million tokens, or Gemini at roughly
90% off, compacting too early can multiply your costs instead of cutting them. You compact when the
cost of re-processing a new, smaller prefix is lower than the cost of continuing to send the full,
cached history, and not one turn before that.

Triggers should also be context-aware. If a session is dominated by tool results, say the agent has
just read a 100k-token documentation file, you should trigger tool clearing or compaction as soon as
the model has extracted the relevant notes. The exploratory data then leaves the window before it can
cause context rot or add unnecessary billing to every reasoning turn that follows.

## When compaction is not worth it

Compaction is frequently an economic trap, and prompt caching is the reason why. Because a summary
creates a brand-new prefix, it invalidates the existing KV-cache and forces the provider to
re-process the summary and the messages after it at the full raw price. On providers like DeepSeek or
Gemini, where cache hits are up to 90% cheaper, a summary has to shrink the context substantially
just to break even on cost. Compact 200k tokens down to 50k and lose your 90% discount, and you have
effectively paid more for a dumber model.

So the rule is: don't compact by default. Only trigger compaction when you hit a hard architectural
wall, meaning the context window limit itself, when the cost of cached input finally crosses the
$0.55/1M token threshold, or when you have empirical evidence in front of you that context rot is
causing task failure. In most production environments, keeping the full history and using prompt
caching is faster, cheaper, and holds reasoning fidelity better than any summary you could write in
its place.

## References

- [Effective context engineering for AI agents — Anthropic](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
- [Compaction — Claude Platform Docs](https://platform.claude.com/docs/en/build-with-claude/compaction)
- [Context engineering: memory, compaction, and tool clearing — Claude Cookbook](https://platform.claude.com/cookbook/tool-use-context-engineering-context-engineering-tools)
- [Context Compaction for AI Agents: A Complete Guide — Redis](https://redis.io/blog/context-compaction/)
- [Context Window Management for Long-Running Agents: Strategies and Tradeoffs — MachineLearningMastery](https://machinelearningmastery.com/context-window-management-for-long-running-agents-strategies-and-tradeoffs/)
- [Context engineering in agents — LangChain Docs](https://docs.langchain.com/oss/python/langchain/context-engineering)
- [Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents](https://arxiv.org/abs/2606.22528)
- [Self-Compacting Language Model Agents](https://arxiv.org/abs/2606.23525)
- [Context Engineering in 2026: Why We Stopped Compacting Our Agent's Context — Louis Bouchard](https://www.louisbouchard.ai/context-engineering-2026/)