Build an Agent Evaluation Harness

What you will build

A small evaluation harness for an agent. You write a test set of tasks with known correct answers, run the agent over all of them, grade each result three different ways, and print a pass or fail summary. This is the difference between “it looked good the three times I tried it” and “it passes 18 of 20 every time I change the prompt.”

Why this lab matters

You cannot trust an agent you have never measured. The moment you change a prompt, swap a model or add a tool, something quietly breaks and you have no way to see it. An evaluation harness is the regression test for probabilistic work. Evaluate before you trust it, then keep evaluating every time you change it.

Expected output

• A versioned dataset of tasks, each with a known expected outcome.
• A run over every task, with the agent’s raw output saved.
• Three graders: exact match, a rule based check, and an LLM as judge with its limits stated.
• A pass or fail regression summary with a pass rate and a threshold.

Prerequisites

• You have read Chapter 8, Evaluate Before You Trust.
• Python 3.10 or later and pytest installed (pip install pytest).
• Optional for the LLM judge only: one model provider key in ONE_MODEL_PROVIDER_KEY.

Steps

1. Make a folder labs/06-evaluation-harness and add the four files below.

2. Write the dataset. Keep it small and honest. Each task has an id, a prompt, an expected answer, and which grader to use. Save as tasks.json. This file is your regression baseline, so commit it and diff it over time.

[
  { "id": "cap-nz",  "prompt": "Capital of New Zealand? One word.", "grader": "exact",  "expected": "Wellington" },
  { "id": "total",   "prompt": "Invoice line totals are 40 and 60. Total?", "grader": "exact", "expected": "100" },
  { "id": "refund",  "prompt": "Customer wants a refund. Reply.", "grader": "rule", "must_include": ["refund"], "must_exclude": ["guarantee"] },
  { "id": "summary", "prompt": "Summarise: the server was down for two hours.", "grader": "judge", "expected": "A short factual summary that mentions the outage and its two hour length." }
]

3. Write the graders. Exact match is strict. The rule based check enforces things that must and must not appear. The LLM judge handles open ended answers, and you state its limits plainly. Save as graders.py.

import os

def exact(output, task):
    return output.strip().lower() == task["expected"].strip().lower()

def rule(output, task):
    text = output.lower()
    if any(t.lower() not in text for t in task.get("must_include", [])):
        return False
    if any(t.lower() in text for t in task.get("must_exclude", [])):
        return False
    return True

def judge(output, task, call_model=None):
    # Honest limits: this grader is non deterministic, it can be wrong,
    # it costs money, and it needs ONE_MODEL_PROVIDER_KEY. Use it only for
    # open ended answers, never for anything exact or rule can decide.
    if call_model is None:
        return None  # skipped, not passed
    prompt = (
        "You grade one answer. Reply PASS or FAIL only.\n"
        f"Expected qualities: {task['expected']}\n"
        f"Answer: {output}"
    )
    verdict = call_model(prompt).strip().upper()
    return verdict.startswith("PASS")

4. Write the runner. It loads the tasks, runs the agent over each one, grades the output, and returns a report. The agent is passed in, so the harness never cares which model or provider sits behind it. Save as run_eval.py.

import json
import graders

def evaluate(tasks, agent, call_model=None):
    results, passed, scored = [], 0, 0
    for task in tasks:
        output = agent(task["prompt"])
        grader = getattr(graders, task["grader"])
        verdict = grader(output, task, call_model) if task["grader"] == "judge" \
            else grader(output, task)
        if verdict is None:
            status = "skipped"
        else:
            scored += 1
            passed += 1 if verdict else 0
            status = "pass" if verdict else "fail"
        results.append({"id": task["id"], "status": status, "output": output})
    rate = passed / scored if scored else 0.0
    return {"pass_rate": round(rate, 3), "passed": passed,
            "scored": scored, "results": results}

if __name__ == "__main__":
    tasks = json.load(open("tasks.json"))
    # Replace stub with your real agent, and pass a real call_model for the judge.
    from test_eval import stub_agent
    report = evaluate(tasks, stub_agent)
    json.dump(report, open("results.json", "w"), indent=2)
    print(f"pass_rate {report['pass_rate']}  ({report['passed']}/{report['scored']})")

5. Write the test. It runs the harness against a stub agent with known behaviour, so the harness itself is deterministic and provable without a network call. The stub gets one answer wrong on purpose, so you can see the gate actually catch a failure. Save as test_eval.py.

import json
from run_eval import evaluate

def stub_agent(prompt):
    if "New Zealand" in prompt: return "Wellington"
    if "Total"       in prompt: return "100"
    if "refund"      in prompt: return "Happy to arrange your refund today."
    return "Something unrelated."          # wrong on purpose

def test_regression_gate():
    tasks = json.load(open("tasks.json"))
    report = evaluate(tasks, stub_agent)   # judge skipped, no key needed
    assert report["scored"] == 3           # three deterministic graders ran
    assert report["pass_rate"] >= 0.75     # the gate: fail the build below this

6. Run it.

cd labs/06-evaluation-harness
pytest -q

7. Now swap stub_agent for your real agent (any provider), lower the wrong answer count to zero, and re run. To grade the open ended task, pass a real call_model into evaluate. Change your prompt, run again, and watch the pass rate move.

Verification checklist

• pytest -q passes and reports one passing test.
• The stub’s wrong answer shows as fail in results.json, not hidden.
• The judge task shows skipped when no key is set, never counted as a pass.
• Breaking one expected value in tasks.json makes the test fail. That proves the gate works.

Common failure modes

• Grading everything with an LLM judge. Use exact and rule wherever the answer is decidable. The judge is the last resort, not the default.
• Treating a skipped judge as a pass. Skipped is skipped. Only scored tasks count toward the rate.
• A test set of three easy tasks. Add the cases that actually broke in real use, especially the ugly ones.
• Never committing the dataset, so you cannot tell whether today’s run is better or worse than last week’s.

Extension ideas

• Save each run to a dated file and chart the pass rate over time.
• Add a cost and latency field per task so you catch a regression that is slower or pricier, not just wrong.
• Have the judge return a one line reason and log it, so a human can audit its calls.

Related chapter

Chapter 8, Evaluate Before You Trust.