Add Memory Without Creating a Mess

The real business problem

Someone tries an agent, it forgets what happened two steps ago, and the obvious fix seems to be “give it memory.” So they turn memory on for everything. The agent starts remembering. It also starts remembering things that are no longer true.

Now you have a new problem that is worse than forgetting. The agent confidently repeats a price that changed last month, greets a client by an old contact’s name, or acts on a preference the customer dropped a year ago. Stale memory is harder to catch than no memory, because the output looks complete and sounds sure of itself.

Memory is not a switch you flip on. It is three different tools that solve three different problems, and most of the mess comes from using the wrong one, or storing things you should have looked up fresh. This chapter gives you a way to use each one on purpose.

The core idea

There are three kinds of memory, and they are not interchangeable.

• Session memory is short-term. It holds the state of the task you are doing right now and disappears when the job ends.
• User or account memory is longer-term. It holds a few stable facts about a person or account that should survive between sessions.
• Knowledge retrieval is not really memory at all. It is looking something up from a store that stays the source of truth, at the moment you need it.

The single most useful distinction in this whole chapter is the last one. Memory is what the agent carries forward and writes down. Retrieval is what it fetches fresh. Anything that changes on its own, without the agent touching it, belongs in retrieval, not memory. Get that line right and most of the mess never happens.

A simple model

Kind	Lives for	Good for	Wrong for
Session memory	One task run	The current multi-step job, what was decided so far	Anything you need next week
User or account memory	Across sessions	Stable preferences, standing instructions, settings	Fast-changing facts like balances or statuses
Knowledge retrieval	Not stored, fetched each time	Large or changing reference material, the system of record	Tiny stable facts you could just remember

Read the table as a decision, not a menu. Start by asking whether the fact changes on its own. If it does, retrieve it. If it does not and you need it later, it is user memory. If you only need it until this job finishes, it is session memory. Most facts fit one box cleanly. The ones that do not are usually a sign you are about to store something you should look up.

A concrete example

Take an accounting firm that runs an agent to draft client correspondence. Each of the three kinds of memory has a clear job.

Session memory holds the thread the agent is working on right now. Which client, which letter, what it has drafted so far. When the letter is done, that state is gone, and that is correct.

User memory holds a small number of stable facts. The firm’s house style, that this partner signs off as “Kind regards” not “Best,” that this client prefers plain English over jargon. These change rarely, so storing them is safe and saves re-explaining every run.

Retrieval handles the numbers. The client’s current balance, their latest invoices, what they have paid. The agent pulls these from the ledger at the moment it drafts, every time.

Here is the failure that teaches the whole lesson. Imagine the agent stores “this client owes 4,200” in memory to save a lookup. That number is an example, not a real figure, and that is the point. Next month the client has paid, the balance is different, and the agent sends a letter with the wrong amount to a real customer. The fix is not better memory. The fix is that a balance is never memory. It is always a fresh lookup, because the ledger, not the agent, owns that fact.

The implementation pattern

Keep the three stores physically separate. When they blur together, you lose the ability to reason about what is safe to trust.

1. Session state lives in the run. It is part of the control loop from the last chapter, not a database. It starts empty, fills during the task, and is discarded at the end. Do not persist it “just in case.”

2. User memory is a small, keyed, timestamped store. Not a growing pile of notes. A handful of fields per account, each with what it is, when it was written, and where it came from. Something like this:

{
  "account_id": "acme-legal",
  "key": "correspondence_style",
  "value": "plain English, no jargon",
  "written_at": "2026-07-23",
  "source": "client request 2026-07-23",
  "expires_at": null
}

Only write an entry when it is a stable fact or an explicit correction from the user. “Draft it shorter next time” is worth storing. “The invoice total is X” is not, because X moves.

3. Expire things that have a shelf life. Give volatile entries an expires_at and drop them on read once they pass it. A standing style preference has no expiry. A note about a temporary situation should die on its own. Expiry is what keeps the store from slowly filling with facts that were true once.

4. Handle contradictions on purpose. When a new fact conflicts with an old one, take the newest by timestamp and, where it matters, surface the conflict rather than silently picking. Silent overwrites are how memory quietly drifts away from reality.

5. Retrieve against the source of truth. For anything the business already stores, the ledger, the CRM, the document system, pull it fresh through a tool. Retrieval does not go stale the way stored memory does, because you are reading the current value every time.

Memory hygiene and privacy

Three rules keep a memory store clean.

What to write: stable preferences, standing instructions, and explicit corrections. Things that will still be true next month and that save real effort by not being re-derived.

What to expire: anything time-bound. Statuses, temporary arrangements, prices, anything with a shelf life. Tag it with an expiry when you write it.

What to never store: secrets, card numbers, passwords, and full copies of sensitive customer records. Do not copy the system of record into agent memory by default. Store a pointer, an account id or a record reference, and fetch the sensitive detail on demand. A memory store is one more place data can leak from, so the safest sensitive data is the data you never put there.

This is the difference between memory and retrieval showing up as a privacy control. Retrieval reads from the system that already owns the data, under its existing access rules. Memory makes a second copy that you now have to secure, keep current, and eventually delete. Prefer the lookup.

Practical checklist

What changes after this chapter

You stop treating memory as a single feature to switch on and start treating it as three tools with different jobs. The question is no longer “should the agent remember this,” it is “is this a stable fact I store, a task detail I hold for one run, or a value I look up fresh.” That one habit removes most stale-fact bugs before they reach a customer, and it keeps sensitive data where it belongs.

Next: Chapter 7, where memory and retrieval feed into giving the agent the right context at the right time.