What Braintrust actually is

Braintrust is an evaluation and observability platform purpose-built for teams shipping LLM applications. Strip away the marketing and it is essentially four things stitched together: a dataset store, a prompt playground, a scoring engine, and a tracing proxy for production traffic.

The core loop is straightforward. You define a dataset of inputs and expected outputs. You define a task, which is just a function that takes an input and produces an output, usually by calling an LLM. You define scorers, which are functions that grade each output against the expected answer or some quality criteria. You run an experiment, which executes the task across every row of the dataset, applies the scorers, and logs everything to a project.

Underneath, Braintrust runs on its own managed infrastructure. You do not deploy anything. You install an SDK, point it at an API key, and your evals run on their servers with results viewable in a web UI. There is also a self-hosted option for the core eval loop, but the hosted version is what most teams use.

The SDKs are in Python and TypeScript. The Python one is the more mature of the two and tends to get new features first. Both expose the same primitives: projects, datasets, experiments, scorers, and logs.

Setup and authentication

The fastest path is through the Python SDK.

Install the package:

pip install braintrust

Then grab an API key from the Braintrust dashboard at braintrust.dev. Once you have it, set it as an environment variable:

export BRAINTRUST_API_KEY=your_key_here

You can also log in interactively from the CLI:

braintrust login

This stores credentials in a local config file, which is convenient when you are running evals from your laptop without polluting your shell environment.

For TypeScript projects, the equivalent setup is:

npm install @braintrust/core

Then either set BRAINTRUST_API_KEY in your environment or pass it explicitly when initializing the SDK.

A quick sanity check that everything is wired up correctly:

import braintrust
print(braintrust.projects())

If this returns a list of project objects without throwing, you are authenticated and ready to go.

First working example

Let us walk through a real eval from scratch. The goal is to grade a small summarization model on a dataset of news headlines and reference summaries.

First, create a project. Projects are the top-level container for datasets, experiments, and logs.

import braintrust
from braintrust import Eval, init_dataset, init_logger

project = braintrust.projects.create(name="summarizer-evals")

Now define the dataset. A dataset is just a list of records with input and expected fields, plus whatever metadata you want.

dataset = braintrust.datasets.create(
    project_id=project.id,
    name="news-summaries",
    items=[
        {"input": {"headline": "Fed raises rates by 25 basis points"}, "expected": "The Federal Reserve increased interest rates by a quarter point."},
        {"input": {"headline": "Apple announces new Vision Pro headset"}, "expected": "Apple revealed an updated mixed reality headset."},
        {"input": {"headline": "Mars rover discovers organic molecules"}, "expected": "NASA's Mars rover found signs of organic compounds."},
    ],
)

Next, define the task. The task receives each row’s input and must return an output. In this case we are calling an LLM to summarize.

def summarizer_task(input):
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize the headline in one sentence."},
            {"role": "user", "content": input["headline"]},
        ],
    )
    return response.choices[0].message.content

Then define a scorer. Scorers take the output and expected value and return a number, usually between 0 and 1.

def exact_match_scorer(output, expected):
    return 1.0 if output.strip().lower() == expected.strip().lower() else 0.0

Now run the eval:

Eval(
    "summarizer-evals",
    data=braintrust.datasets.load(dataset.id),
    task=summarizer_task,
    scores=[exact_match_scorer],
)

This kicks off an experiment, runs the task across every row, applies the scorer, and writes everything to the project. Open the Braintrust UI and you will see the experiment with per-row scores, latency, token usage, and the full prompt and response for each call.

Key settings that matter

Most people run their first eval and stop there. The settings that actually move the needle are the ones below.

Experiment metadata. Every experiment accepts a metadata dict. Use it to tag runs with the model version, prompt version, or git commit. This is how you turn a wall of eval results into something you can actually diff against later.

Scorer composition. You can attach multiple scorers to a single eval. A typical setup is one deterministic scorer (exact match, regex, JSON validation) plus one LLM-as-judge scorer that grades subjective quality. The LLM judge is more expensive but catches things deterministic scorers miss.

LLM-as-judge scorers. Braintrust has a helper called AutoScorer that wraps an LLM judge prompt. You give it a rubric and it returns a score. This is the right tool when you need to grade things like tone, helpfulness, or factual consistency where there is no single correct answer.

Concurrency. The Eval constructor accepts a max_concurrency parameter. Default behavior is sequential, which is fine for tiny datasets but painfully slow for anything over a few hundred rows. Bump this up when you are running against rate-limited APIs.

Dataset versioning. When you update a dataset, Braintrust keeps the old version around. Every experiment records which dataset version it ran against, which means you can always reproduce a result even after the dataset has moved on.

Tracing proxy. This is the production observability side. You wrap your LLM client with Braintrust’s proxy and every call gets logged automatically, including token counts, latency, and cost. The proxy works with OpenAI, Anthropic, and other providers through a drop-in replacement client.

Online scoring. Production logs can be scored in real time using the same scorer definitions you wrote for offline evals. This closes the loop between what you tested in development and what users actually see.

Where it shines

Braintrust is genuinely good at a few specific things.

Prompt regression testing is the headline use case. If you change a prompt and want to know whether it got better or worse, Braintrust gives you a side-by-side view of two experiments with the same dataset. The diff view shows which rows improved, which regressed, and by how much. This is the kind of thing that is hard to build yourself and easy to underestimate until you have tried it.

Team collaboration is another strength. Datasets and scorers are first-class objects that multiple people can edit, version, and comment on. If you have a product manager who wants to add test cases or a domain expert who wants to refine a rubric, they can do it through the UI without touching code.

The LLM-as-judge tooling is more polished than most alternatives. The AutoScorer helper handles prompt templating, output parsing, and caching of judge calls. For teams that need to grade subjective quality at scale, this saves real engineering time.

Production tracing through the proxy is the underrated feature. Most eval platforms stop at offline testing. Braintrust extends the same scorer definitions into production, which means you can write a scorer once and use it both to gate releases and to monitor live traffic.

Where it fails

No platform is right for everything, and Braintrust has real limitations.

The self-hosted story is thin. You can run the core eval loop on your own infrastructure, but you lose access to the UI, the dataset store, and the proxy. For teams with strict data residency requirements, this is a dealbreaker.

The ecosystem is smaller than LangSmith or Langfuse. There are fewer third-party integrations, fewer community-built scorers, and fewer pre-built dataset templates. If you are looking for a turnkey solution that plugs into twenty other tools, you will be disappointed.

LLM-as-judge costs add up. Every scored call is another LLM call, and judge models are typically larger and more expensive than the model being graded. A dataset of a few thousand rows can cost real money to score, especially if you are running experiments frequently.

The UI can lag on large experiments. Once you cross a few thousand rows, the experiment view gets slow to load and the diff between two large experiments is not as snappy as you would hope.

There is no first-class support for multi-modal evals. If you are grading image generation or audio transcription, you will be doing more custom work than the platform supports out of the box.

Practical workflow pattern

The pattern that works in practice looks like this.

During local development, you write your task and scorers as plain Python functions. You run small evals against a local dataset to catch obvious regressions before you commit. This is fast feedback, no network round trip, no UI.

When you are ready to compare two prompt versions, you push the dataset to a Braintrust project and run a full experiment. The results land in the UI where you and your team can review them, leave comments, and decide which version to ship.

Once a version is in production, you wrap your LLM client with the Braintrust proxy. Every call gets logged with its inputs, outputs, latency, and cost. Online scorers run against a sampled subset of traffic and flag rows where quality drops below a threshold.

When a regression is detected in production, you add the failing rows to your dataset, re-run your offline evals against the updated dataset, and iterate on the prompt until the new version passes. The loop closes.

For CI integration, Braintrust has a GitHub Actions example that runs a smoke eval on every pull request. The eval uses a small representative subset of the dataset and fails the build if scores drop below a threshold. This catches prompt regressions before they reach main.

The key insight is that evals are not a one-time thing. They are a continuous feedback loop that connects development, testing, and production. Braintrust is one of the better tools for keeping that loop tight, especially for teams that ship LLM features frequently and need to know whether each change made things better or worse.

For a deeper walkthrough of tools like this and how they fit together, the free Working With Claude field guide covers the ecosystem end to end. Get the guide.