The Four Types of Agent Memory, and Three Things the Diagram Leaves Out

The plan was simple. Run LoCoMo, a long-conversation memory benchmark, against Letta, score it, move on. Instead the archival-memory run stalled and I lost most of a day to a broken auth path between the framework and the embedding service. Zero LoCoMo numbers that week. (An earlier benchmark of mine on generated sessions did produce numbers, and Letta lost that one to a plain baseline. Different experiment, different data, and it is written up separately.) What I got here was a sharper view of the gap between how agent memory looks in a diagram and how it behaves when you actually run it. Both halves are worth having. You need the clean version to reason and the messy version to ship. ## The taxonomy The four-type split is from CoALA, Cognitive Architectures for Language Agents (Sumers, Yao, Narasimhan and Griffiths, arXiv:2309.02427). It is worth reading in the original rather than in the diagrams that circulate downstream of it. **Episodic memory is what the agent did.** Past interactions, actions, outcomes. After the agent acts, the system persists that action somewhere retrievable, usually a vector store, so "how did we handle this last time" becomes a similarity search. Written by the system at runtime, after the fact. **Semantic memory is what the agent knows.** External facts, internal documentation, anything the agent should hold about itself or its domain. If you have built a RAG application, you have built semantic memory. It describes the world, not the agent's history in it. **Procedural memory is how the agent operates.** The system prompt, the tool definitions, the guardrails. Most people do not think of this as memory at all, which is exactly why it gets mismanaged. It lives in Git and tool registries and it gets versioned like code, because it is code. It changes through deployment, not conversation. **Working memory is what the model sees right now.** This one reframes the other three. They are storage. Working memory is the compiled result: the application pulls from episodic, semantic and procedural stores, assembles one prompt, and sends it. The first three are long-term. Working memory is short-term and rebuilt on every call. The sentence worth keeping: the model never sees your memory system. It sees one string. Everything you call memory is retrieval and compilation happening outside the model, and your agent's quality depends on that compilation step more than on any individual store. There is a failure mode hiding in that step. When you compile the prompt under a token budget, the task usually goes last, so when the budget runs out the task is what gets truncated. Every real framework makes some version of this prioritisation choice and most make it silently. If you have watched an agent confidently answer a question it was never fully shown, you have seen the output of that function. ## Three things the taxonomy does not cover The four types are correct and I recommend them. They are also incomplete in three specific ways, and all three only show up at runtime. **First, all four compete for one context window.** The diagram draws four boxes as if they were four independent resources. At runtime they are four writers sharing one page. Procedural memory alone can be heavy: a serious tool registry plus guardrails claims a real fraction of the window before the task even arrives. Add retrieved episodes and semantic grounding and the prompt budget stops being a tuning parameter and becomes an architecture constraint. **Second, the taxonomy says nothing about invalidation.** A fact changes. Semantic memory gets the update. Episodic memory still holds a dozen interactions built on the old fact, and similarity search has no concept of which of these is still true. The agent contradicts itself with total confidence. Memory that only grows does not make an agent smarter. Past a point it makes the agent confidently stale. **Third, the four boxes do not map cleanly onto real frameworks.** Letta, the framework I was benchmarking, gives you core memory blocks, archival memory and recall storage. The mapping onto episodic, semantic and procedural is approximate at best. The taxonomy is a thinking tool. Your first implementation task is translation, not transcription. ## The part the taxonomy posts skip Before I could measure memory quality at all, both blockers were infrastructure. A broken auth path to the embedding service means semantic retrieval silently degrades to nothing. Not an error, just worse answers. A stalled archival-memory run means the benchmark is measuring your patience rather than the framework. Neither failure is exotic. Both are the kind of thing my day job on quantum cloud infrastructure trains you to expect: the architecture diagram is never the thing that pages you. A memory system is not a data model. It is distributed infrastructure. Storage, an embedding service, retrieval, and a compilation step, each of which fails independently and mostly silently. The taxonomy tells you what to store. It does not tell you what to evict, what to invalidate, or what happens when all four types arrive at the same context window at the same time. I will publish the LoCoMo numbers when the runs complete, including what stalled and why. The diagram is what you believe. The benchmark is what you know.