Agent State: How AI Agents Track What Already Happened

Agent state is your system's internal belief model of a partially observable world. It is the bridge between raw model inference and reliable task execution, and it is what lets an AI agent hold a persistent understanding of an environment that keeps changing underneath it. The performance gap between systems that manage state well and systems that do not is often wider than the gap between the model backbones they run on, which is not the ranking most teams expect.

State is the mechanism you use to track history, manage token budgets, and keep answers coherent across multi-turn interactions. Without it, you are running disconnected, stateless functions and hoping the prompt carries the rest.

An AI agent holding a persistent record of what has already happened across a multi-step task

What is agent state and historical context?

In the framework of a Partially Observable Markov Decision Process (POMDP), agent state is the "belief state." An agent cannot see its entire environment at once — whether that environment is a 50,000-line repository or a week-long customer support thread — so it has to build a persistent internal model instead. The state record is that model in data form, and it is what keeps every downstream decision grounded in a verified record of what has already occurred.

Comparison of state as a curated belief model against history as a raw event log

The move worth making here is from a model-centric view of infrastructure to a memory-centric one. Most of the effort in a team goes into model selection or prompt tuning, while memory architecture is the thing that actually separates the production systems that hold up from the ones that do not. The transition from prompt engineering (static instructions) to context engineering (the dynamic management of state and tokens across time) is what this field maturing looks like. And in plenty of cases, swapping a frontier model for a smaller one with a better state management system buys you more reliability than upscaling the backbone does.

All of this management is necessary for one unglamorous reason: context is a finite resource. Large language models (LLMs) operate on a strict "attention budget" dictated by the transformer architecture, and that architecture creates n² pairwise relationships for n tokens. So as the context window fills with raw, unmanaged history, the model's ability to capture those dependencies is stretched thin, and you end up in "context rot," where the model's precision for information retrieval and long-range reasoning degrades.

Good context engineering, then, is a hunt for the smallest possible set of high-signal tokens that still satisfies the attention budget. That means curating actively: deciding which tokens are worth the n² computational cost, and which ones are noise you are paying to carry around. If you treat state as a managed asset rather than an append-only log, you keep the agent steerable even as the interaction horizon expands.

What does a state record actually contain?

A structured state trace is the foundation of observability, and it rests on four pillars that together let you reconstruct an agent's semantic behavior:

Tool Calls: Records the tool name, arguments, return values, and metadata such as latency or retry counts.
Reasoning Steps: Captures the intermediate chain-of-thought, planning phases, and the plan-act-observe transitions that define the agent's logic.
State Transitions: Documents the before and after snapshots of working memory, tracking how context was edited or compressed.
Memory Operations: Logs interactions with long-term storage, including semantic search hits, retrieval scores, and the freshness of the data retrieved.

Your infrastructure needs nested spans so that parent-child relationships survive, and that matters most in multi-agent handoffs, where the failure usually begins because Agent A passes incomplete context to Agent B. Without deep tracing of those transitions, Agent B acts on drifted assumptions and the root cause becomes impossible to diagnose, because there is no unified trace ID tying the two halves of the run together.

json

{
  "span_type": "state_transition",
  "span_id": "st-9982",
  "parent_id": "run-443",
  "inputs": {
    "prior_state": "User requested refund for duplicate charge.",
    "new_observation": "Transaction ID #554 found in database via lookupTransactions."
  },
  "outputs": {
    "updated_state": "Found duplicate ID #554; verified timestamp; proceeding to createRefund tool."
  },
  "timing": {
    "start": "2026-06-21T10:00:01Z",
    "duration_ms": 45
  }
}

What happens when an agent doesn't track state?

When agents fail to manage state, they suffer from the "Lost in Conversation" phenomenon. Performance drops by an average of 39% across six generation tasks — Code, Math, Database, Actions, Data-to-text, and Summary — in multi-turn settings. An agent may pass a fully-specified single-turn evaluation at 90% accuracy, yet its ability to solve that same problem collapses when the information is revealed gradually over several turns, which is exactly how every real user talks to it.

Performance falling 39% from single-turn to multi-turn, with unreliability rising 112%

Stateless agents typically fail through three specific modes:

Premature Assumptions: Agents often guess at missing details in early turns and then over-rely on those incorrect initial attempts, even after being provided with the correct data.
Answer Bloat: Without state curation, solutions grow longer and noisier because the agent cannot invalidate its earlier wrong assumptions. Reasoning models like o3 and DeepSeek-R1 already generate responses around 33% longer than non-reasoning LLMs, which compounds the problem.
Loss-of-Middle-Turns: Agents prioritize the first and last turns of an interaction, effectively ignoring instructions or data introduced in the center of the conversation.

These failures push unreliability, the performance gap between best-case and worst-case runs, up by an average of 112%, while raw aptitude falls only 16%. That split is the finding I keep coming back to: the model barely got less capable, it got far less predictable. And in these settings even a high-aptitude model becomes useless, because it cannot produce a successful outcome consistently across multiple attempts.

The write-manage-read loop, and the step most systems skip

Reliable state management runs as a continuous cycle: Write (logging observations), Manage (curating the record), and Read (retrieving relevant context). Nearly every system builds Write and Read, and nearly none of them builds Manage, which is how you arrive at attentional dilution: the model has the information sitting right there in the window and still cannot focus on it.

The three-stage write, manage and read cycle, with manage marked as the skipped step

The fix is a heuristic control policy, a small set of rules that decides how memory gets groomed. A policy might say: escalate an observation to semantic memory only after it has been referenced successfully in three turns, or age out tool results from the context window after five turns and replace them with a single-sentence summary. None of it is clever; it is bookkeeping, and the bookkeeping is what keeps everything above it honest.

Without the Manage step, memory becomes a junk drawer of stale data — and a junk drawer is exactly what erodes the attention budget you were trying to protect.

Layers of state: working, episodic, semantic, and procedural

Organize state into distinct temporal layers and both retrieval and cost get easier to reason about:

Four stacked agent memory layers: working, episodic, semantic and procedural

Working Memory: The high-bandwidth, ephemeral context window. It is the most prone to context rot and must be limited to immediate tasks.
Episodic Memory: A chronological record of concrete experiences, such as daily standup logs or searchable timelines. This tracks what happened and when.
Semantic Memory: Distilled, curated facts. This is the lasting truth an agent preserves, often stored in a MEMORY.md file or a long-term vector database.
Procedural Memory: Encoded skills, persona constraints, and behavioral patterns. These are often loaded via AGENTS.md or SOUL.md.

markdown

# AGENTS.md (Procedural Memory)
 
- Always verify transaction IDs before initiating refunds.
- Version: 2.1.0 (2026-06-21)
 
# MEMORY.md (Semantic Memory)
 
- User preferred currency: USD (Confirmed turn 4).
- Last successful build: v2.4.1 (2026-06-15).

The split between the layers matters more than the naming does, and two of them live in plain files you can open and read. That is a feature rather than a shortcut, because a memory system you cannot inspect is a memory system you cannot debug.

How systems store and restore state

Modern infrastructure uses a tiered storage model, so engineering state comes down to balancing four families of mechanism:

Context-Resident Compression: Sliding windows or rolling summaries. These are clean but risk summarization drift, where nuances are lost.
Retrieval-Augmented Generation (RAG): Embedding past interactions. This is powerful for deep history but fails when semantic intent (vector similarity) doesn't match causal intent — searching for "what happened last Monday" is the classic miss.
Hierarchical Virtual Context: A RAM, disk, and cold-storage model where the agent manages its own paging.
Just-in-Time (JIT) Context: Tools like Claude Code use lightweight identifiers such as file paths to load specific data only when needed, rather than stuffing everything into the attention budget upfront.

Each execution step saved as a state snapshot, with the agent resuming from the latest checkpoint

Underneath all four sits checkpointing, which is the least fashionable item on this list and the one I would build first: each execution step is saved as a state snapshot, scoped to a thread identifier so that separate conversations keep separate histories. After a crash or a context reset, the agent resumes from the most recent checkpoint rather than replaying the task from the beginning. Anthropic's Claude Developer Platform also offers a file-based memory tool in public beta, released alongside Sonnet 4.5, for storing and consulting information outside the context window.

When history grows: compaction, external notes, and the cost of summarizing

As a conversation nears the context limit, systems reach for compaction: summarizing the history and re-initiating the window. Do that often enough, however, and you are back in context rot and drift, because each round of summarizing throws away slightly more than you meant it to. To combat this, better-built agents use structured note-taking.

Compaction losing detail versus external notes keeping the context lean

By maintaining an external file such as NOTES.md, an agent can persist state across context resets. In Anthropic's "Claude plays Pokémon" test, an agent maintained coherence over a multi-hour session by tracking objectives like "for the last 1,234 steps I've been training my Pokémon in Route 1, Pikachu has gained 8 levels toward the target of 10." By reading its own structured notes after a context purge, the agent avoided the unreliability that plagues standard chat interfaces.

Keep the raw episodic record alongside every summary. Summaries let the model reason cheaply, but only the raw log lets you recover when a summary has drifted away from what actually happened, and it will drift without ever announcing that it has.

Designing state for an agent: where to start

Prioritize reliability and state architecture over model selection, because a smarter model will still get lost if its state management is poor. Start by defining explicit temporal scopes: build episodic memory first to track what happened, then move to semantic memory for distilled facts only as the use case actually demands it, and not a sprint earlier.

Treat procedural memory (persona and rules) as code: review it, and keep it under source control. And keep those raw episodic records even when the summaries look fine, because summarization drift is silent, and the raw log is the only ground truth you can return to.