What AI Evals Actually Is

AI Evals is a framework for systematically testing large language model outputs against a defined standard. Strip away the marketing and it is three things bundled together: a structured test case format, a grading function that decides pass or fail, and a runner that executes the model against each test and reports aggregate results.

It is not a magic quality detector. It is a unit testing harness for prompts. If you have ever shipped a prompt change, watched it break three downstream behaviors you did not anticipate, and then spent an afternoon manually checking outputs, AI Evals is the missing piece that would have caught those failures in under a minute.

The mental model that makes the rest of this guide click is this: your prompt is the function under test, the test cases are the inputs and expected behaviors, and the grader is the assertion. A typical evaluation suite for a production LLM application might contain 50 to 500 test cases depending on use case complexity, each one a specific scenario your system needs to handle correctly.

Most teams start with a single eval file, grow it into a directory of scenario-specific suites, and end up with a regression net that runs in a couple of minutes before every deploy. The frameworks people reach for most often include the OpenAI Evals open source framework, Braintrust, Promptfoo, and DeepEval. The patterns below apply to all of them, with the OpenAI framework as the concrete reference since it is widely deployed and the API surface is stable.

Setup and Authentication

Before you write a single test case you need three things: a working Python environment, API access to at least one model provider, and a directory structure that will not collapse the moment your suite grows past ten files.

The current version of the OpenAI Evals framework installs in under a minute on a clean Python 3.10+ environment. The standard flow looks like this. Create a virtual environment, activate it, then install the framework with pip. You will also need the OpenAI Python client if you intend to grade against GPT models, or the equivalent client for whichever provider you use.

Authentication is handled through environment variables. Set OPENAI_API_KEY for OpenAI models, ANTHROPIC_API_KEY for Claude, and so on. The framework reads these at run time, which means you can keep your eval configs provider-agnostic and swap models by changing a single line.

A clean project layout looks like this. A top level evals directory, a registry subfolder containing the individual eval definitions, a data subfolder for fixture files, and a logs subfolder where the runner writes output. The current version of the framework supports YAML, JSON, and Python-based eval definitions. YAML is the right starting point because it is easy to read, easy to diff, and easy to extend with new cases without writing code.

First Working Example

Let us build an eval for a customer support classifier that needs to label incoming tickets into one of four categories: billing, technical, account, or other. The eval will run 20 tickets through the classifier, check that the output category matches the expected one, and report the accuracy.

The first file is a list of test cases. Each case has an input, the message text, and a target, the expected category. Start with ten to twenty hand-written cases that cover the obvious buckets, then add ten more tricky cases that you have seen fail in production.

The second file is the eval definition. It names the eval, points at the test cases, declares the model to test against, and specifies the grader. For categorical output, an exact-match grader is the right starting point. The model-graded grader is better for open-ended outputs but adds cost and latency.

The third file is the completion function, the actual prompt your classifier uses in production. This is important: you should not be testing a different prompt than the one running live. The whole point of evals is regression detection on the real system.

Once the three files exist, the runner invocation is a single command. The output shows you a pass rate, a per-case breakdown, and the model’s actual response next to the expected one. For a healthy classifier you should see pass rates in the 80 to 95 percent range on a hand-curated set, with the failures clustered in genuinely ambiguous cases.

Key Settings That Matter

The defaults will get you a working eval, but the production-quality settings live below the surface. Here are the dials that change the meaning of your results.

Model selection is the first dial. Running the same eval against multiple models is one of the highest-leverage uses of the framework. A typical pattern is to test the production model as the system under test, and a stronger model as the grader. The cost difference is real, often a factor of three to ten, but the signal is also stronger.

Grader choice is the second dial. String match is fast and deterministic, embedding similarity handles paraphrase, and model-graded is the only option for open-ended quality. Most teams run a mix: exact match for structured outputs, model-graded for tone, helpfulness, and factual accuracy. A common pattern in the range you would expect for production is 40 percent exact match, 60 percent model-graded.

Sample size and temperature are the third dial. Running each test case multiple times with non-zero temperature gives you a distribution rather than a single data point. For high-stakes eval suites, three to five samples per case is a reasonable starting point. Anything below that gives you noise masquerading as signal.

Pass thresholds are the fourth dial. A 100 percent pass rate requirement is almost always wrong because it punishes the eval for catching genuine ambiguity. A more useful approach is to set a floor, say 85 percent, that triggers a deploy block, and treat anything below that as a required investigation before merge.

Finally, the isolation setting matters for any eval that hits a real API. Network calls during eval runs are flaky in ways local mocks are not. The current version of the framework supports a sandbox mode that lets you replay recorded completions, which is the right approach for any large suite that runs on every commit.

Where It Shines

AI Evals earns its keep in four specific scenarios, and the ROI is sharpest when you lean into the first one.

The first scenario is regression testing after prompt changes. Any non-trivial prompt edit carries the risk of breaking a behavior that was previously working. Running an eval suite before merge catches the breakage in seconds rather than after a customer reports it.

The second scenario is model migration. When you swap GPT-4o for a newer release, or move from one provider to another, the eval suite is the only way to know whether the new model is actually better on your workload. Aggregate benchmarks lie, your own data does not.

The third scenario is pre-deployment gates. The strongest teams run their eval suite as a required check in CI. A failure blocks the deploy, full stop. This is how you turn “we should test prompts” into “prompts cannot ship without passing tests”, which is the only version that actually happens.

The fourth scenario is comparing prompt strategies. When you have a real question about whether to add few-shot examples, switch to chain-of-thought, or restructure the system prompt, the eval suite gives you an answer. A/B tests in production are slower, noisier, and riskier than running the candidates through a curated set.

Where It Fails

Honesty about limitations is part of using the tool well. AI Evals does not solve every quality problem, and pretending otherwise leads to bad decisions.

The first limitation is coverage. A 200-case eval suite covers what you thought to test. It does not cover the long tail of inputs your system will see in production. The eval suite is a guardrail, not a guarantee. New failure modes will appear in production that no eval anticipated, which is why human review of samples remains necessary.

The second limitation is grader cost and reliability. Model-graded evals can be expensive at scale, and the grader is itself an LLM with its own failure modes. A grader that hallucinates a pass for a clearly wrong answer is worse than no grader at all. You need to spot-check grader output the same way you spot-check the system under test.

The third limitation is overfitting to the eval. Once a prompt is tuned to ace the eval suite, you have built a system that is good at the eval and nothing else. The fix is to hold out a portion of cases from the prompt iteration loop, and rotate in fresh cases from production logs on a regular cadence.

The fourth limitation is non-determinism. Same prompt, same model, different answer on different runs. The current version of the framework handles this with multi-sample averaging, but you should still expect a few percentage points of jitter between runs of an identical suite. Treat eval results as a signal, not a