Agent Memory: The Four Kinds and How They Fit Together

An LLM (large language model) agent is stateless by default. Every API (application programming interface) call starts from scratch. To build an agent that remembers your preferences, learns from past mistakes, and works across sessions, you need four memory systems: in-context, external, episodic, and semantic. Each handles a different timescale and retrieval pattern, and they compose to give the illusion of a persistent mind.

The surprising part is that the most powerful memory isn't the one that stores facts. It's the one that stores stories of what went wrong last time, so the agent never makes the same mistake twice.

In Part 2, we saw how multiple agents coordinate to avoid deadlock. But even a single agent is useless if it can't remember anything between calls. That's the problem we solve here.

The amnesia problem

Picture a brilliant new employee. They understand every domain concept, reason flawlessly, and write perfect prose. But every morning they walk in with total amnesia. They forget your name, your preferences, yesterday's work. You'd spend half your time re-explaining everything. That employee is a raw LLM. Useless until you build memory around it.

The fix is not one memory system. It's four, each with a different job. Together they turn a stateless function call into an agent that feels like it's been working alongside you for months.

Agent memory transforms a stateless LLM

Without memory

• Resets every API call
• No user preferences
• Repeats past mistakes
• No learning over time

With four memory types

• Remembers conversation context
• Stores preferences externally
• Learns from episode summaries
• Applies domain knowledge

What are the four types of agent memory?

The four types differ in what they store, how long they persist, and how they're retrieved. Here's how they fit together in Mailmind, our inbox-resident AI assistant.

In-context memory: the current window

In-context memory is everything inside the current context window. The conversation so far, tool results, system instructions, and any retrieved snippets you've stuffed in. It's fast because the model sees it directly. It's also limited in size, and it vanishes the moment the session ends.

For Mailmind, in-context memory holds the email thread being processed right now. The subject, the sender's message, the draft reply so far, the calendar lookup result. All of it lives in the prompt. This is the only memory the LLM natively sees. Everything else has to be explicitly loaded into context before the call.

External memory: persistent identity

External memory is any persistent storage outside the model. Vector databases, relational databases, graph stores. It survives across runs. This is where the agent keeps its long-term identity.

Mailmind uses external memory for every past conversation, contact preferences, and standing instructions. "Always cc my cofounder on investor emails." "Never schedule before 10am for this client." The agent retrieves the relevant bits before each task and loads them into in-context memory. Without external memory, the agent would treat every email like the first one you ever received.

Episodic memory: the self-improvement loop

Episodic memory stores records of specific past events. Not facts, but stories. What task was attempted, what strategy was used, what worked, what failed. Each run ends with a structured summary. Before the next similar run, the agent retrieves episodes that match the pattern.

Mailmind's episodic memory might hold: "Last time I scheduled with Sam, he rejected morning slots. Propose afternoons." The retrieval is semantic. The embedding captures the experience pattern, not just the name Sam. So when a different slow-to-reply contact asks for a meeting, the episode still matches. The agent learns across individuals.

This is what enables self-improving behavior. The agent doesn't just remember facts. It remembers what to do differently.

Semantic memory: the baseline knowledge

Semantic memory is the general domain knowledge the model already has. It comes from pre-training or from a curated knowledge base you attach via retrieval-augmented generation. It's the foundation.

Mailmind knows what an invoice looks like, what "OOO" means, how flight confirmations are structured. It doesn't need to learn that from your inbox. It already has that baked in. Semantic memory handles the "what is this thing" question so the other memory systems can focus on "what does this person want."

How do the four memory types work together in a single Mailmind run?

Take a concrete task. A colleague emails: "Can we meet next Tuesday to review the Q3 numbers?" Mailmind's semantic memory recognizes this as a scheduling request and knows what a meeting proposal needs. In-context memory holds the email thread and the tool outputs as they arrive. External memory fetches the colleague's calendar preferences (no early mornings) and any standing instructions about Q3 review meetings. Episodic memory searches for past scheduling attempts that had a similar pattern, maybe one where the colleague ignored a morning proposal and only replied to an afternoon slot.

The LLM reasons across all of it, proposes a 2pm slot, and drafts a reply. After the run, Mailmind writes an episode: "Scheduling with colleague who historically ignores morning slots; proposing afternoon worked." Next time, even with a different person who shows the same pattern, that episode surfaces.

How do you prevent context overflow?

You can't load 10,000 emails into a context window. You'd blow the token limit and pay a fortune in latency. The solution is retrieval-augmented context. Embed all past conversations and store them in a vector database. Before a run, retrieve only the 5 most relevant ones and inject them into the prompt.

The same principle applies to episodic memory. You don't load every past episode. You embed the episode summaries and retrieve the top few by semantic similarity to the current task. A sliding window approach keeps recent conversation turns in context while older turns get replaced by summaries. Summarization compresses long threads into a few sentences that capture the gist, trading detail for space.

Quick reference

Memory Type	Timescale	Storage	Retrieval	Mailmind Example
In-context	Current session	LLM context window	Immediate access	The email thread being processed
External	Persistent	Vector DB, relational DB, graph	Semantic search, SQL, graph queries	All past conversations, contact preferences, standing instructions
Episodic	Persistent, event-based	Vector DB (episode summaries)	Semantic similarity on task pattern	"Last time scheduling with Sam, he rejected mornings"
Semantic	Static (or slowly updated)	Model weights or curated knowledge base	Implicit via LLM, or RAG on knowledge base	Knowing what an invoice or OOO reply looks like

Frequently Asked Questions

Q: How does episodic memory differ from just storing conversation history?

Conversation history is raw logs. Episodic memory is distilled experience. It captures the strategy, outcome, and lesson learned, not the full transcript. That abstraction lets it generalize across superficially different but structurally similar situations.

Q: When should I use a vector DB versus a relational DB for external memory?

Use a vector DB for fuzzy, semantic retrieval: "find conversations like this one." Use a relational DB for exact lookups: "get the standing instruction for this contact." Most real systems use both. The relational DB stores structured facts, and the vector DB stores embeddings for retrieval when you don't know exactly what you're looking for.

Q: How do you prevent context overflow when retrieving many episodes?

You embed episode summaries and retrieve only the top few by similarity to the current task embedding. You also set a hard token budget for retrieved memories. If the top episodes would exceed it, you drop the least similar ones. The system never loads all episodes into context.

Q: Can semantic memory be updated without retraining the LLM?

Yes. You can attach a curated knowledge base via retrieval-augmented generation. Before each call, retrieve relevant domain documents and inject them into the context. This gives the agent up-to-date semantic knowledge without touching the model weights.

Q: What's the latency impact of retrieving episodes before each run?

It depends on your vector database and embedding model. A well-tuned system adds tens of milliseconds. The retrieval runs in parallel with other external memory lookups. The cost is negligible compared to the LLM inference time, and the improvement in decision quality more than justifies it.

Test yourself

Mailmind keeps missing package-delivery updates buried in promotional emails from the same retailer. The emails look like marketing, but they contain tracking numbers. You fix the prompt so Mailmind now checks for tracking numbers before discarding a promo. How do you make sure it never makes the same class of mistake on retailers it has never seen?

Answer: Write an episode after the fix. The episode summary says: "Delivery updates can hide inside promo-formatted emails. Before discarding any promotional email, scan for tracking numbers or shipping keywords." Embed that summary and store it in the episodic database. Future runs, regardless of retailer, will retrieve this episode whenever the task involves processing promotional emails. The retrieval is based on the pattern (promo formatting plus the need to extract actionable info), not the sender's name. So even a brand-new retailer triggers the learned behavior. The agent now generalizes the lesson across all similar situations, not just the one you patched.

If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.

In Part 4, we'll look at how agents use tools and function calling to actually do things in the world, building on the memory foundation we just laid.