Evaluate Before You Trust It

The problem this chapter solves

You built the agent. You tried it, it produced a clean result, and you thought “that works.” Then you handed it real work and it made a mistake you never saw coming. This is the most common way an agent project fails after launch. Not because the model is bad, but because “it looked right when I tried it” was never evidence in the first place.

One good run tells you the agent can succeed. It does not tell you how often it succeeds, on which inputs it fails, or whether the change you made last week quietly broke something that used to work. For a demo, one run is enough. For a business system, you need to know the failure rate before you trust the agent with anything that costs money or touches a customer.

This chapter gives you the discipline that closes that gap. By the end you will be able to build a set of real tasks with known correct answers, grade the agent’s output automatically, catch regressions before they ship, and decide whether the agent is actually good enough to trust. This is where a demo becomes a system.

The core idea

The thesis of this playbook is that an agent is only useful when it can see the work, do the work and prove the work was done. Evaluation is how you prove it. If you cannot verify the outcome, the task is not finished. That is true of a single run, and it is true of the agent as a whole.

Evaluation here means what it means everywhere else in engineering. You collect a set of representative tasks, you know the correct answer for each one, you run the system against all of them, and you measure how many it got right. The result is a number you can act on. “The agent categorised 46 of 50 expense lines correctly, and the four it missed were all foreign currency” is a fact you can work with. “It looked right” is not.

The shift in mindset is this. Stop asking “did it work when I tried it.” Start asking “what is its score on the tasks I care about, and did that score go up or down since the last change.”

A simple model

An evaluation harness is a loop with four parts, and one idea sitting on top of them.

Task set  ──▶  Run agent  ──▶  Grade output  ──▶  Score + failures
▲                                                      │
│                                                      ▼
└──────────  Regression check on every change  ◀───────┘

The four parts are the task set, the run, the grader and the report. The task set is real inputs paired with the known-correct outcome for each one. The run sends every task through the agent. The grader scores each output against its expected outcome. The report is a pass rate plus the exact list of what failed and why.

The idea on top is regression. Every time you change the prompt, the model, a tool or the workflow, you run the whole set again and compare the new score to the last one. A change that fixes one thing and breaks two others shows up straight away, instead of surfacing three weeks later in front of a client.

A concrete example

Take an agent that categorises expenses for a monthly bookkeeping close. In a demo it reads one invoice, picks the right account, and looks flawless.

To evaluate it, you build a task set of 50 real expense lines pulled from past months, each already assigned to the correct account by a bookkeeper you trust. That known-correct assignment is the expected outcome. You run the agent against all 50 and compare its account choice to the human’s, line by line.

Now you have a real number. Say it gets 46 right. You look at the four it missed. Three are foreign currency transactions and one is a refund it booked as an expense. That is not a vague worry, it is a specific and fixable set of failures. You improve the prompt, run all 50 again, and check whether you fixed those four without breaking the other 46. When the score is high enough and stays there, you trust the agent with this month’s real close. Not before.

The implementation pattern

Three moves build a working harness.

1. Build the task set. Pull 30 to 100 real examples, not invented ones. Include the messy cases: the odd invoice, the ambiguous request, the input that broke it last time. For each task, record the input and the expected outcome. Store them as plain files so they live in version control next to the code.

tasks/
  expense-001.json   { "input": {…}, "expected": { "account": "6-1200" } }
  expense-002.json   { "input": {…}, "expected": { "account": "6-4000" } }
  …

2. Choose a grader per task. Not every task is graded the same way. There are three main kinds, and each one has an honest limit you should know before you rely on it.

• Exact match. The output must equal the expected value. This is right for account codes, yes or no answers, and extracted figures. It is objective and cheap. The limit is that it cannot judge anything open ended, and it will wrongly fail a correct answer that is simply worded differently.
• Rule-based check. The output must satisfy a rule you write in code. “Contains a valid invoice number.” “Total equals the sum of the lines.” “No dollar figure appears that is not in the source.” This is strong for structure and hard constraints. The limit is that it only checks what you thought to write a rule for.
• LLM as judge. A second model scores the output against a rubric you give it. This is the only practical way to grade open-ended writing, such as whether a client summary is accurate and clear. The limit is real and worth stating plainly. The judge is itself a model, so it can be wrong, it can be biased toward its own style, and it must never grade using the same prompt that produced the work. Pin it with a tight rubric, use it only for genuine judgement calls, and spot-check its scores against a human now and then.

Most real harnesses mix all three. Exact match and rules for anything you can pin down, an LLM judge only for the parts that truly need judgement.

3. Run, score and gate. Wire it so one command runs every task, grades each one, and prints a pass rate plus the list of failures. Set a bar, for example “must score at least 90 percent and never regress on a task it previously passed.” Run it before you trust the agent with live work, and run it again on every change.

$ npm run eval
  expenses      46/50  (92%)
  summaries     18/20  (90%, LLM judge)
  REGRESSION    expense-014 passed last run, fails now

That last line is the whole point. The regression check is what stops a quiet break. A change that lifts the summary score but breaks an expense case that used to pass is not an improvement, and now you can see it before it ships rather than after a client does.

This is what evaluation-driven development means in practice. You write the check before you trust the work, and the check, not your gut, decides whether the agent is ready.

Decision rules

Related lab

The lab turns this chapter into a harness you can run on your own work. You will collect a real task set, write a grader for each type, wire the run-and-score command into one line, and add a regression check so a future change cannot quietly break what works today. You finish with a number you can point to instead of a feeling.

Practical checklist

• You have 30 or more real tasks, including the hard and messy ones.
• Every task has a known expected outcome recorded next to it.
• Each task has a grader, and you chose the type on purpose.
• One command runs the whole set and prints a pass rate plus the failures.
• A regression check flags any task that used to pass and now fails.
• You run the eval before trusting the agent, and again on every change.

What changes after this chapter

You stop trusting the agent because it looked right once, and start trusting it because it scores well on the tasks that matter and holds that score when you change things. That is the line between a demo and a system you can put in front of a client. If you cannot verify the outcome, the task is not finished, and now you have the machinery to verify it.

Next: Chapter 9, where the same evaluation discipline moves from a pre-launch gate to a live signal you watch while the agent runs in production.