What AI Memory Actually Is

Strip the marketing and AI agent memory is just a structured store of context an agent can read and write to across sessions. The “agent” part is the important word. Most people who use ChatGPT or Claude through a chat interface get a working memory of the conversation. The model holds the recent messages in the context window. What it does not have is a memory that survives the session. Close the tab, open a new chat, and the model has no idea who you are or what you worked on yesterday.

AI Memory is the layer that fixes that. Technically it is a combination of three things working together. A write path that takes the current conversation, extracts what is worth remembering, and stores it. A retrieval path that, given a new prompt, fetches the relevant stored items. And a prompt assembly step that injects the retrieved items into the model call before the user’s message is sent.

The actual storage under the hood is usually a vector database for semantic search, a relational or document store for structured facts, and a small LLM call for extraction. The retrieval is typically a hybrid search that combines vector similarity with keyword or metadata filters. The output is a block of text that gets prepended to the system prompt or placed near the top of the message stack.

The reason this matters is that without a memory layer, an agent is amnesiac. It cannot learn a user’s preferences, cannot track tasks across days, and cannot build up a working model of a codebase, a customer, or a workflow. With a memory layer, the same agent becomes something closer to a colleague who has worked with you for months.

Setup and Authentication

Most AI memory tools in this category ship with a Python or Node SDK, plus a REST API. The setup shape is similar across vendors. Install the package, set an environment variable for the API key, and call the client.

The first real choice is whether to run the memory layer as a managed service or self-hosted. Managed is faster to start and what most people should use until they have a specific reason to self-host. Self-hosted versions of open frameworks typically require a vector database such as Qdrant, pgvector, or Chroma, plus an LLM for the extraction step.

For a managed setup the install and auth looks like this. Create an account, generate an API key from the dashboard, then install the client in your project. In Python that is pip install followed by setting the key in your environment. In Node it is npm install and either a dotenv file or a runtime secret manager.

The key thing most people miss is that the memory API key is separate from the LLM API key. The memory service still has to call an LLM to do extraction and sometimes to do summarization, so you will either provide your own LLM key at setup time or pay the memory service a markup for the inference it does on your behalf. Budget accordingly. In the range you would expect for a tier-one API, a few thousand stored memories can cost meaningful dollars per month once you factor in the extraction calls.

For local development you can run most of these with a single Docker compose file that brings up the memory server, the vector store, and a small embedding model. Production setup is the same plus a managed vector database, separate secrets in a vault, and rate limiting on the memory endpoints so a runaway agent cannot fill your storage with garbage.

First Working Example

A working example is worth more than a feature list. Here is a minimal end-to-end flow in Python that adds a memory, then retrieves it in a new session that has no other context.

Start by importing the client and initializing it with the API key. Then call the add method with a list of messages. The service takes the conversation, extracts facts, and stores them. You will get back a list of memory IDs that you can ignore for now. The point is the data is in.

Next, create a new client instance, or just clear your local context, to simulate a fresh session. Call the search method with a natural language query that has no keyword overlap with the original message. The retrieval should still return the stored fact because the search is semantic, not lexical.

Then pass the result into an LLM call as a system message. Something like “The following facts are known about the user. Use them when relevant.” followed by the retrieved items. From the model’s perspective, it now knows something it never learned from this conversation. That is the whole trick.

A curl version of the same flow uses POST requests to the memories endpoint for writes and POST requests to the search endpoint for reads. The bodies are JSON, the auth is a bearer token in the header, and the response includes a score for each retrieved item so you can filter by relevance threshold.

The thing to notice is how little code is involved. The hard part is not calling the API. The hard part is deciding what to write, when to write it, and when to read it. Those decisions live in your agent code, not in the memory service.

Key Settings That Matter

The defaults are tuned for generic chatbot use cases. Production agents usually need a few of these dialed in.

The first is the extraction prompt. The memory service uses a system prompt to decide what is worth remembering. The default usually extracts user preferences and stable facts. If your agent is working on a codebase, you want it to extract file paths, function names, and unresolved bugs. If it is a sales agent, you want deal stage, decision makers, and objections. Override the extraction prompt with your own instructions.

The second is the retrieval limit. The default is often five or ten items, which is fine for a chatbot and wrong for a coding agent. A coding agent that only sees five memories at a time will forget the project structure the moment it dives into a file. Raise the limit to a number that makes sense for your context window budget.

The third is the relevance threshold. Retrieved memories come back with a similarity score. Memories below your threshold are noise. Set this too low and you pollute the context with irrelevant items. Set it too high and you miss genuinely useful recall. A common starting point is in the range of 0.7 to 0.8 for cosine similarity, but the right number depends on your embedding model.

The fourth is the update versus insert behavior. When a user says “I moved to Berlin” after previously being recorded as being in London, do you overwrite, append, or create a new memory and let the retrieval sort it out. Most services let you choose. For facts that have a single current value, overwriting is correct. For facts that have history, appending is correct.

The fifth is the token budget for the memory block. Injections eat into your context window. If you allocate 2000 tokens for memory and your window is 8000, you have lost a quarter of your capacity before the user has typed a word. Track this and tune.

Where It Shines

AI memory is genuinely useful in a small number of well-defined patterns.

Long-running assistants are the first. A research agent that you spin up daily, ask questions about the same project, and expect to remember the prior answers is the textbook case. Memory makes the agent feel like a teammate instead of a stranger.

Customer-facing agents with a stable user base are the second. If you have a few thousand users chatting with a support agent, memory lets the agent remember prior tickets, product versions, and stated preferences. Without memory, every conversation starts from zero and customers notice.

Codebase-aware agents are the third. An agent that works on a long-lived repo, where the structure, conventions, and known bugs are stable enough to be worth remembering, benefits enormously from memory. The agent stops re-asking the same questions about where things live.

Multi-agent systems are the fourth. When two or more agents collaborate on a task, shared memory acts as the coordination layer. Agent A writes a plan, agent B reads it