Give the Agent the Right Context

The real business problem

You give an agent a task and it hands back a confident, wrong answer. You look closer and the reason is almost always the same. It did not have the right information in front of it, or it had far too much and could not tell which part mattered.

This is a context problem, not a model problem. A strong model with the wrong context will fail. A modest model with clean context usually does the job. Most of the reliability you get out of an agent comes from one decision. What does it see, when does it see it, and what do you keep out.

After this chapter you will be able to split an agent’s information into three separate kinds, feed it knowledge on demand instead of dumping everything in, and name the things you deliberately leave out. The worked example is a research and brief agent, the kind of thing that turns a company name into a one page brief before a meeting.

The core idea

Context is not one thing. It is three things that people cram into a single prompt and then wonder why the agent gets muddled.

• Stable instructions. Who the agent is, the rules it always follows, the format it must produce. This barely changes from one run to the next.
• Task state. The specific job right now. The client name, the question, the sources found so far, the running notes. This changes every run and often every step.
• Retrieved knowledge. The facts it needs to answer well. Policy documents, past reports, product data, pricing. This is large, mostly irrelevant to any one task, and should be pulled in a few pieces at a time.

Keep these three separate in your head and in your build. When they blur together you get an agent that forgets its own rules halfway through, or drops last week’s client details into this week’s report.

A simple model

STABLE INSTRUCTIONS   role, rules, output format      loaded every run, never edited
+
TASK STATE            this client, this question,      changes every step
notes so far
+
RETRIEVED KNOWLEDGE   the 2 or 3 docs needed now       pulled on demand, then dropped
|
v
[ context window ]  ->  model  ->  next step

Think of the context window as a desk, and the desk is small. You do not stack the whole filing cabinet on it. You put down the standing rules, the job you are working on, and the two or three files you actually need for the step in front of you. Everything else stays in the cabinet until it is needed.

A concrete example

Take a research and brief agent. The job is simple to state. Given a company and a question, produce a one page brief a partner can read before a meeting.

• Stable instructions: you are a research analyst, you cite every claim, you never invent numbers, the brief is 300 words across three sections. Fixed for every run.
• Task state: the company is Acme Freight, the question is whether they fit our audit service, and here are the four sources found so far with short notes on each. This grows as the run goes.
• Retrieved knowledge: your own past briefs on similar firms, the fit scoring rubric, the service description. Pulled in only at the moment the agent needs them.

Now watch what happens if you paste all of that into one giant prompt on every step. The run slows down. It costs more, because you pay per token on every single step and you are re-sending the whole rubric each time. Worst of all, the agent starts bleeding the rubric wording into the brief, or quoting a past client by name because that brief was sitting in the window. Separate the three and each step stays clean and cheap.

The implementation pattern

Here is the pattern that holds up in real use. It is four moves, and none of them are exotic.

1. system_prompt   = role + rules + output format      # written once, loaded every run
2. task_state      = { company, question,              # small structured object,
                       sources_found[], draft{}, step } #   updated each step, not free text
3. needed          = agent.decide_what_to_look_up(task_state)
   chunks          = document_store.search(needed, top_k = 3)   # retrieve, do not dump
4. window          = system_prompt + task_state + chunks        # assemble fresh each step
   result          = model(window)

A few things to hold onto.

Write the system prompt once. Role, rules, output format. Load it on every run and never edit it mid task. This is the layer that keeps the agent behaving like itself.

Hold task state as a small structured object, not a growing wall of text. A short record of the company, the question, the sources with one line notes, and the draft so far. Update it each step. Structured state is easy to trim, easy to inspect, and easy to hand to a human when something goes wrong.

Retrieve, do not dump. When the agent needs a fact, search your document store and pull the top few matching chunks, not the whole library. This is retrieval, often called RAG. The point is not the acronym. The point is that the agent reads three relevant paragraphs instead of three hundred irrelevant ones.

Trim as you go. After a step is done, drop the retrieved chunks you no longer need. Keep the task state, keep the rules, let the rest go.

What breaks

Decision rules

Related lab

The Research and Brief Agent lab is where this becomes real. You build the three layer split by hand, wire a small document store, and watch the difference between dumping everything into one prompt and retrieving three clean chunks. You will feel the cost drop and see the answers get sharper. The same pattern carries straight into a document to report agent, which is the next lab up.

Practical checklist

What changes after this chapter

You stop treating the prompt as a bucket you fill to the top. You start designing the window on purpose, one layer at a time, with a clear answer to what goes in, what stays out, and what gets pulled fresh. That single shift removes a large share of the wrong answers people blame on the model.

Next: Chapter 4, giving the agent safe tools so it can act on the world, not just describe it.