Have you ever been in a project meeting that feels like you’re stuck in a loop? You’re trying to build on last week’s decisions, but a key collaborator seems to have a fresh start every morning. You spend the first twenty minutes just re-explaining the context, the trade-offs, and the conclusions you all agreed to yesterday. It’s frustrating. You’re not just collaborating; you’re constantly performing a manual “context reload” for a human.

This experience, surprisingly, gives us a perfect window into one of the biggest challenges in building intelligent AI. An agent without memory treats every interaction as its first. It’s the difference between a conversation with a seasoned collaborator and a goldfish. It can’t recall your preferences, the context of your last request, or the results of an action it just took. This digital amnesia is the single biggest barrier that separates a clever tool from a true partner. To make that leap, our agent needs to remember. But what does that even mean for a machine?

Before we break down the layers of an agent’s memory, it’s crucial to understand a fundamental constraint of most large language models today: their APIs are stateless. This means that with each new request, the model has no inherent recollection of your previous interactions. It’s like having a conversation with a brilliant expert who has no short-term memory; every time you speak, you have to reintroduce yourself and painstakingly recount the entire conversation up to that point. The only information the model has to work with is what you provide in the current API call. This is why the entire history of a conversation, plus any other relevant documents or data, must be bundled together and sent back to the model with every single turn.

While this might sound like a significant limitation, it’s also a source of incredible control. Because we, as the system’s architects, are responsible for curating the agent’s memory for each interaction, we can be highly intentional about what it remembers. This statelessness is precisely what makes techniques for managing memory not just possible, but powerful. It forces us to build an external memory system, effectively turning a bug into a feature.

As we’ll see, an agent’s memory is not a single thing, but a layered system of recall, much like our own. We can break it down into three distinct forms: what the agent is thinking about right now (working memory), what it remembers happening (episodic memory), and what it knows about the world (semantic memory). Each of these layers comes with a fundamental trade-off that every engineer must navigate: the tension between cost, latency, and fidelity.

The Agent’s Mental Scratchpad

At the heart of any agent’s ability to “pay attention” is the context window. But before we dive in, let’s clarify a fundamental unit of measurement in this world: the token. A token isn’t quite a word; it’s the basic unit of text or code that a large language model processes. Think of them as the atoms of language for an LLM. For example, the word “unforgettable” might be split into three tokens: “un,” “forget,” and “table.” A good rule of thumb is that one token is roughly equivalent to about four characters of text. So, when we talk about a model’s “context window,” we’re talking about the total number of these tokens it can hold in its attention at any given time.

This window is the agent’s mental scratchpad, its working memory—the space where the current conversation, immediate instructions, and relevant data live. It’s what allows the agent to follow the thread of a conversation and connect one turn to the next.

With the advent of models like Gemini that boast million-token context windows, it’s tempting to see this as the ultimate solution to the memory problem. And for certain tasks, it’s a superpower. You can drop an entire codebase into the context to find a bug, analyze the full script of a movie to discuss character arcs, or sift through a massive legal document to find a specific clause. It provides a vast, temporary workspace for a single, complex analysis.

But it’s not a true long-term memory. Using a massive context window for an ongoing agentic task is like trying to have a focused conversation in a room where every previous discussion is still echoing. Stuffing it with a long, rambling chat history or slightly different versions of the same file doesn’t just increase cost and latency; it introduces noise. As researchers discovered in the “Lost in the Middle” paper, models suffer from a peculiar form of inattention—they reliably recall information at the very beginning and very end of their context window, but their performance degrades significantly when trying to access information buried in the middle. A bloated context, therefore, doesn’t just cost more; it can actively make the agent less effective by hiding the signal in the noise.

A more elegant approach is to treat the context window not as a bucket to be filled, but as a workspace to be managed. A fantastic, real-world example of this is the GEMINI.md file used by the Gemini CLI. It’s a simple markdown file that acts as a running log, a set of instructions, and a summary of the project’s state. Before starting a session, the agent can load this curated file into its context. It isn’t a raw transcript; it’s a human-and-machine-readable summary that grounds the agent in the specific task at hand, turning the context window into a persistent, but session-specific, memory space. This architectural takeaway is key: the agent’s immediate attention is a precious resource, and managing it sets the stage for more sophisticated memory systems.

The Agent’s Diary of Interactions

While working memory handles the here and now, an agent needs a way to remember the narrative of its interactions over time. What did the user ask for ten minutes ago? What was the result of that API call I made? This is the agent’s episodic memory—its personal story.

The simplest approach is a sliding window of conversation history, which leads to a kind of abrupt amnesia. A much smarter solution is the use of summarization buffers. The agent essentially keeps a diary of its interactions. As the conversation grows, a separate process recursively summarizes older parts of the dialogue. It’s like creating a “Previously on…” segment for a TV show. You don’t need every line of dialogue, just the key plot points. This can be made even more dynamic through progressive summarization, where the agent periodically re-summarizes its existing summaries to consolidate knowledge and identify higher-level themes.

But a truly intelligent agent doesn’t just record its past; it reflects on it. This is where the concept of salience comes in—the agent’s ability to determine what’s important. After an interaction, a more advanced agent can perform a self-reflection step, asking itself: “What were the key takeaways from that conversation? What new facts did I learn? What was the most important user preference revealed?” By scoring memories based on their importance, the agent can prioritize what to keep in its more detailed memory stores.

A powerful architectural pattern for this is explored in the paper “Entities as Experts,” which proposes a model with a dedicated memory for specific entities. A practical application of this is Entity Memory, where an agent is specifically tuned to extract and remember key entities—like people, project names, or locations—and their context, creating a quick-reference cache of the most important nouns in its world.

Just as important as remembering is the ability to forget. In human intelligence, forgetting is a feature, not a bug. It’s what prevents us from being overwhelmed by a lifetime of trivial details. For an agent, this is a crucial design principle. The process of recision—identifying and removing outdated or irrelevant information—is what keeps an agent’s memory relevant. If a project’s goals change, the agent doesn’t just add a new memory; it revises its understanding of the past. Designing how an agent forgets is as important as designing how it remembers.

Of course, each of these techniques lives on a spectrum of trade-offs. A highly detailed, reflective memory with entity extraction provides a rich, high-fidelity context, but every summarization and reflection step adds latency and computational cost. Deciding where to land on that spectrum—a fast agent or a thorough one—is a core design decision in building any stateful system.

Giving the Agent a Library to Read

Episodic memory gives the agent a personal history, but it doesn’t give it knowledge about the outside world. For that, we need to give it a library. This is where Retrieval-Augmented Generation, or RAG, comes in.

The technique was formally introduced in a 2020 paper, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” and it represented a major breakthrough. The core idea was elegant: combine a pre-trained generative model with an external information retriever. This hybrid approach gives the agent an open-book exam. Instead of relying solely on the information baked into its parameters during training, it can look things up in an external knowledge base at the moment it’s needed. This makes the agent’s responses more factual, verifiable, and up-to-date.

In practice, this is how it works today: as I explored in my previous series on embeddings, the process involves taking your documents, chopping them into chunks, and using an embedding model to create “meaning vectors” for each. These vectors are stored in a specialized vector database. When a user asks a question, the question itself is turned into a vector, and the database finds the chunks of text with the most similar meaning. Those chunks are then “augmented” into the context window along with the original question, giving the model the source material it needs to generate a factually grounded answer. This is the very technique I used to build my podcast RAG project, which lets me query the transcripts of my own recorded conversations.

The next evolution of this is what’s being called Agentic RAG. Where a simple RAG is a one-shot lookup, an agentic RAG system behaves more like a real researcher. It’s the engine behind the new wave of deep research tools. An agent using this pattern doesn’t just pull one document. It can perform an initial search, synthesize the results, identify gaps in its knowledge, and then autonomously formulate new, more refined queries to dig deeper. This iterative process provides far greater depth, but again, introduces that fundamental trade-off: a multi-step agentic query is significantly more expensive and slower than a single lookup.

The Immutable Laws of the Agent’s World

Some information, however, isn’t meant for semantic interpretation; it’s factual ground truth. An agent needs a place to get hard facts: “What is the user’s ID?” “What’s the current inventory level for this product?” This is where structured memory—the world of traditional SQL and NoSQL databases—comes in. They are the agent’s physics engine, representing the immutable laws of its world.

Accessing this kind of memory requires the agent to act. And this brings us to one of the most important concepts in agentic architecture, which we’ll explore fully in the next part of this series: tool use. An agent doesn’t need to know the data in a database; it just needs to know how to ask for it by using a tool. This creates a safe and powerful boundary between the agent’s fluid, probabilistic reasoning engine and the rigid, factual data store. When a question requires structured data, the agent can generate and execute a SQL query, not as a direct thought, but as a deliberate action.

For example:

• User: “How many users signed up in the last 24 hours?”
• Agent Thought: The user is asking for a specific count from the user database. I need to use the query_user_database tool. I will formulate a SQL query to get this information.
• Agent Action: tool.query_user_database(sql=”SELECT COUNT(*) FROM users WHERE signup_date >= NOW() – INTERVAL ’24 hours’;”)
• Tool Response: (count: 1,234)
• Agent Response: “There were 1,234 new sign-ups in the last 24 hours.”

This “Think, Act, Observe” loop is the fundamental pattern for how agents interact with the outside world. While we’re introducing it here in the context of memory, it’s the key to unlocking an agent’s ability to do just about anything—from sending an email, to calling a weather API, to interacting with another agent. It’s the core of the “Action” in our “Perceive, Reason, Act” model, and it’s the subject of our next post.

The Expanding Architecture of Memory

The categories we’ve explored form the foundation of agent memory, but the field is rapidly evolving. Advanced architectures are becoming increasingly important for building more capable systems.

While vector databases excel at finding semantically similar information and SQL databases provide rigid facts, a third powerful structure is emerging as a cornerstone of advanced agent memory: the knowledge graph. A knowledge graph stores information not as documents or rows, but as a network of entities and the explicit relationships between them. Think of it less like a library and more like a detailed mind map. An agent equipped with a knowledge graph doesn’t just know that a document mentions “Project X” and “Alice”; it knows that “Alice” is the owner of “Project X,” and “Project X” is dependent on “Component Y.” This allows the agent to perform complex, multi-hop reasoning, answering questions like, “Who are the owners of all the projects that depend on Component Y?”—a query that would be incredibly difficult for a standard RAG system.

Furthermore, our discussion has been very text-centric. But the world an agent perceives is rich and multi-modal, and its memory must be as well. Multi-modal memory is the frontier where agents learn to recall not just what they’ve been told, but what they’ve seen. This could mean remembering the specific UI element on a screen to complete a complex navigation task, recalling the contents of a chart from a presentation slide, or identifying a product in a user-uploaded photograph. Instead of just text embeddings, the agent’s memory systems must store image embeddings, graphical data, and representations of spatial layouts, creating a far more holistic understanding of its environment.

When One Memory Serves Many

Finally, we must consider the “who” of memory. Most of our examples implicitly assume an agent serving a single user. But in real-world applications, agents will increasingly serve teams, departments, or entire companies. This introduces the complex challenge of shared memory and multi-tenancy. How does an agent partition its knowledge? What information becomes part of a shared team memory that everyone can access, and what must remain private to an individual’s episodic history? Designing the permissions, boundaries, and a “common ground” for a shared agent memory is as much a challenge of security and product design as it is of technical architecture. It’s about building a memory that respects context, privacy, and collaboration.

Weaving the Threads of Memory

So which architecture is right for you? The answer, as always, depends on the job to be done. For a conversational support agent that needs to recall user history and preferences, a robust episodic memory is paramount. For an agent designed to perform deep research and synthesis, a rich semantic memory powered by Agentic RAG is the core of its intelligence. And for an agent tasked with managing a business process or interacting with internal systems, a reliable structured memory is non-negotiable.

But the most sophisticated applications won’t be defined by a single memory type. Just as we concluded in Part 2 that an agent might employ an ensemble of different reasoning patterns, a truly capable agent will feature a hybrid memory system. The most powerful agents will use episodic memory to recall your preferences, query a semantic knowledge base to answer your questions, and access a structured database to execute a task on your behalf, all within a single, coherent workflow.

These memory systems are also deeply intertwined with how an agent thinks. A simple ReAct agent, as we explored in Part 2, lives almost entirely in its working memory, using its scratchpad to reason step-by-step. A more advanced planning agent, by contrast, must constantly query its episodic and semantic memory to build and validate its complex plans before acting.

Ultimately, an agent’s memory isn’t one thing, but a sophisticated, layered system. Building it is how we transform a forgetful tool into a context-aware partner. It doesn’t remove the need for human oversight; it elevates it. Our role shifts from operator to strategist, collaborating with an agent that understands our shared history and goals. The frontier is already pushing further, towards agents that can learn and adapt their own memory structures and, eventually, achieve true persistence not just through external databases, but through continuous fine-tuning of the model itself.

We’ve given our agent a brain, hands, and now, a memory. But to truly interact with the world, it needs to be able to use those hands. In our next post, we’ll dive into the “Action” part of the agent: a deep dive into the tools it uses to interact with these memory systems and the digital world at large.

This is a rapidly evolving space, and these patterns are just the beginning. I’m curious to hear from those of you on the front lines: What are the most interesting memory challenges you’re facing in the systems you’re building? Let me know in the comments below.