AI Agent Memory Systems: How Agents Remember Across Sessions

Standard large language models (LLMs) are stateless in the strictest sense: they reset their internal state with every API request. So you bolt a memory layer onto the model to hold context between those requests, and that layer is what turns a transient completion function into an agent that can learn over time and carry one task across several sessions.

In production, that layer is more than a conversation log, because it has to work as a structured context engineering layer: it is what lets an agent keep a consistent identity, remember what a specific user prefers, and build on what it did three weeks ago. It is also the least glamorous part of an agent stack, and the part that decides whether the thing feels like a product or a demo. Without a memory system, an agent treats every interaction as a first-turn encounter.

Why does an LLM forget everything between sessions?

LLM inference is stateless because the underlying transformer architecture treats every sequence of tokens as an independent computational event. Once the model generates an output, the activations are purged and the model retains no inherent memory of the transaction. For an agent to "remember" a previous turn, the entire history must be re-inserted into the context window on every subsequent request, and that costs real infrastructure, because recomputing the key-value (KV) cache, the intermediate attention state the model keeps for the tokens already in the window, is memory-bandwidth bound and grows as dialogues lengthen.

As token counts increase toward the edge of the context window, models experience "context rot", a degradation where reasoning precision and retrieval accuracy drop. The transformer's attention mechanism relies on pairwise relationships between tokens, so cost grows quadratically with sequence length. At extreme scales the model's attention budget is stretched thin, which makes it harder to pick the relevant signal out of the surrounding noise. High token counts lead to attention errors regardless of the window size a provider advertises, and that is the part people skip past when a new model ships with a bigger number on the box.

Growing the window, then, defers the problem instead of solving it. Managing memory through an external retrieval layer is what lets you hit sub-second response targets while keeping high-fidelity recall over long durations, because you feed the model only what the current turn needs instead of everything that has ever been said.

What separates short-term from long-term memory?

Short-term memory (thread-scoped, or working memory) is the immediate context required for a single session. It is managed through checkpointers that persist the state of one conversation thread, so if a network error hits or a process restarts, the agent resumes from the last saved state. In-memory stores like Redis are the usual home for it because the access pattern is hot and latency-sensitive. This memory is essentially discardable once the task reaches its logical conclusion.

Long-term memory is designed for cross-session persistence and survives system restarts over months. Unlike thread-scoped context, it is global and shared across conversation namespaces, and it holds the agent's evolving understanding of a user or an environment. This is the layer that enables cross-session learning, where an agent uses an insight from a conversation three weeks ago to inform a decision today.

The two layers also differ in when they are written, and that is the design decision people underestimate. Short-term state is updated on the hot path, during the primary execution loop. Long-term memory is usually written in the background, because writing on the hot path makes new facts immediately available but adds latency and complexity to every single turn, while writing asynchronously keeps the response loop clean at the cost of deciding how often to trigger memory formation.

The three kinds of long-term memory: facts, experiences, procedures

A serious agent architecture borrows its taxonomy from cognitive science: semantic, episodic, and procedural memory. Semantic memory stores factual knowledge and concepts — the tribal knowledge of user preferences, product specifications, or organizational hierarchies. You can hold it as a single continuously-updated profile document, or as a collection of narrowly-scoped documents. The collection approach gives higher recall, because it is easier for a model to generate a new object for new information than to reconcile it against a growing profile, while the profile approach is easier to read back but becomes error-prone as it grows. Either way, strict decoding, which constrains the model to emit only output matching the declared structure, is what keeps those documents valid.

Episodic memory captures temporal sequences and specific past experiences. It records what happened in a particular event — the exact steps taken during a troubleshooting session last month. In production it is implemented with event logs or time-stamped vector entries, which is what makes temporal reasoning possible: the system can track how a fact changed over time and let recent updates take precedence over historical ones.

Procedural memory holds the operational logic and behavioral constraints of the agent, and it is increasingly managed through system prompts and runtime constraints rather than hardcoded. Using reflection and meta-prompting, an agent can refine its own instructions based on past feedback, updating its how-to manual without retraining any weights. Representing these memories as a graph rather than flat text helps with multi-hop relationships, though the gain is smaller than the architecture diagrams imply: in the Mem0 team's own evaluation, the graph variant scored roughly 2% higher overall than the base configuration.

How does an agent store and retrieve a memory?

The memory lifecycle runs through four stages: encoding, storage, retrieval, and integration. During encoding, conversational data is transformed into vector embeddings using transformer-based models. Those embeddings give a mathematical representation of semantic meaning, so the system can calculate similarity between the current query and stored history instead of matching keywords.

Storage needs an indexing structure that balances latency, recall, and resource use. The usual choice is between Hierarchical Navigable Small World (HNSW) and Inverted File Index (IVF) in a vector database. HNSW gives high recall and fast approximate search, but its memory footprint is larger because the navigable graph lives in RAM. IVF is more efficient at billion-vector scale because it clusters vectors and searches only the relevant buckets, trading some precision for a smaller footprint. Retrieval itself is fast, roughly 200 milliseconds for a semantic search, which is why the expensive work in a memory system is never the lookup.

Vector search is not the only shape memory takes, and I think the file-based option is underrated. Agents also use memory tools that operate on a /memories directory, where the model creates, reads, updates, and deletes files directly. That is how you store facts that must come back exactly as written rather than by semantic similarity — a policy document, a set of project conventions. It carries its own obligations, though: cap how large a memory file can grow, reject any path that escapes the memory directory, and delete files that have gone a long time without being read.

Retrieved context is then integrated into the prompt just in time. Rather than loading the whole knowledge base up front, the agent holds lightweight identifiers and pulls the specific record into context only when the current step needs it, which keeps the active window lean and high-signal.

What to keep, what to summarize, what to throw away

Maintaining high-fidelity memory requires a consolidation pipeline built on four operations: ADD, UPDATE, DELETE, and NO-OP. When new information arrives, the system compares it against existing memories. If a customer's budget moves from $500 to $750, the new value becomes the active memory and the previous one is marked inactive. If the new statement merely restates what is already stored, the correct action is NO-OP. That last one is what keeps the agent from carrying two contradictory versions of the same fact.

Extraction comes first, and it is where most of the judgment lives. Instead of storing raw chat logs, the system uses a model to separate meaningful information — preferences, decisions, constraints — from routine conversational chatter. Getting this wrong in either direction is costly: extract too little and the agent forgets what mattered, extract too much and you have filled long-term storage with noise you will later retrieve and pay for.

Background maintenance, sometimes called "dreaming", continuously curates what has accumulated. Running asynchronously, it reduces duplication, evicts stale memories, forms new associations between related records, and derives new knowledge from them. This is the part of the system that handles "how long to keep it" — nothing else in the pipeline is watching for a memory that was true last quarter and is now merely old.

Compaction handles the other half of the problem, inside a single long session. When the conversation approaches the context limit, the system summarizes the history and reinitiates the window with a distilled version, keeping architectural decisions, unresolved bugs, and current goals, and dropping redundant tool outputs and superseded messages. Done well, the agent continues a long-horizon task without carrying the raw transcript that produced it.

How agent memory breaks in production

The first failure mode is latency, and it comes from putting the expensive work in the wrong place. Extraction and consolidation take roughly 20 to 40 seconds for a standard conversation once triggered, against about 200 milliseconds for a semantic search. Run that synchronously inside the response loop and the user sits there waiting, which is why the pipeline belongs in the background, with the response path only reading memory that is already built.

Context pollution is the second. Retrieval returns records that are semantically similar but contextually irrelevant, and those distract the model rather than helping it. Pure vector similarity with no filtering or reranking is the usual cause. A prompt padded with stale or off-topic memories loses precision in exactly the way a bloated context window does — you have rebuilt the problem the memory layer was supposed to solve, only now with extra infrastructure to maintain.

The third one is subtler: a memory that was correct when it was written and is wrong now. Conflicting facts are the hard case, because both values were true at some point and neither one looks malformed. That budget that moved from $500 to $750 is only handled correctly if the consolidation logic tracks timestamps and marks the superseded record inactive, since a system that simply appends both will retrieve whichever one happens to rank higher.

What all three have in common is that the agent itself will not tell you. In one published walkthrough, an assistant was asked in session one to research free weather APIs for a project called Beacon, and it recommended Open-Meteo. In session two it had no record of the project or the API, because there was no memory layer holding either of them. Nothing errored. The agent simply answered as though the first conversation had never happened.

When do you actually need a memory system?

You need one when the task requires state across days, personalization tied to a specific user, or cross-session learning, and not before. The cost argument is concrete: against stuffing the full history into context, a memory system cut p95 latency by about 91%, from 17.1 seconds to 1.44 seconds, and saved over 90% of the token cost. For a coding agent tracking repository structure and past debugging attempts across thousands of steps, a stateless approach simply will not hold.

If your system answers short, self-contained requests, a simple store is enough and a memory layer is operational overhead you will not earn back. The dividing line is not how much data you have — it is whether the agent needs to know anything at all after the session ends.

References

• Memory overview — LangChain
• Memory tool — Claude Platform Docs
• Effective context engineering for AI agents — Anthropic
• AI agent memory: types, architecture & implementation — Redis
• Building smarter AI agents: AgentCore long-term memory deep dive — AWS
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — arXiv
• Mem0 vs Letta (MemGPT): AI Agent Memory Compared — Vectorize
• From context to dreams: architecting memory for AI agents — Red Hat