Build a Data-to-Decision Agent

What you will build

An agent that takes one defined dataset, a CSV, and turns it into a decision you can act on. It computes the numbers with code, draws a chart, and writes a short summary that recommends a next step. Then it does the part most people skip. Before it reports the decision, it checks that every number in the summary matches the numbers the code actually computed. If they do not match, the run is not done.

Why this lab matters

A model can write a confident paragraph about your data without the data supporting a word of it. That is the failure that makes people distrust agents. The fix is not a better prompt. The fix is a control loop where the numbers come from code, not from the model, and a check that catches the model if it drifts. This lab is the smallest honest version of “analyse this and tell me what to do.”

Expected output

• A short analysis of one metric from your CSV.
• A chart saved as a PNG file.
• A decision summary of a few sentences with a clear recommendation.
• A verification step that fails the run if any stated number does not match the computed number.

Prerequisites

• You have read Chapter 5, The Workflow and the Control Loop.
• You can run a small Python script and set an environment variable.
• pip install pandas matplotlib and one model provider SDK.
• A CSV with at least a category column and a numeric column. Monthly sales by region works well.

Steps

1. Define the decision, not just the analysis. Write one sentence: “Which region should we cut budget from next quarter, based on this data.” A decision gives the agent a finish line.
2. Compute the metric in code and treat that output as the only source of truth. The model never invents a number. It only reads the numbers your code produced.
3. Draw the chart from the same computed numbers, so the picture and the prose come from one place.
4. Have the model write the summary using only those numbers.
5. Add the verification step. Pull every number out of the summary and confirm each one appears in the computed results. A mismatch fails the run.

Here is the shape. Load the CSV, compute a metric, and lock those figures.

import pandas as pd

def compute_metric(csv_path: str) -> dict:
    df = pd.read_csv(csv_path)
    by_group = df.groupby("region")["sales"].sum()
    facts = {region: round(float(total), 2)
             for region, total in by_group.items()}
    facts["lowest_region"] = str(by_group.idxmin())
    facts["lowest_value"] = round(float(by_group.min()), 2)
    return facts  # the only source of truth for numbers

The chart draws from the same computed figures, not from the model.

import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt

def draw_chart(facts: dict, out_path: str = "chart.png") -> str:
    groups = [k for k in facts if k not in ("lowest_region", "lowest_value")]
    values = [facts[g] for g in groups]
    plt.figure()
    plt.bar(groups, values)
    plt.title("Total sales by region")
    plt.tight_layout()
    plt.savefig(out_path)
    return out_path

Now the verification. This is the point of the lab. Find every number the summary states, and confirm each one came from compute_metric. If the model wrote a figure the data does not contain, the check fails and the agent goes back to fix it.

import re

def verify_summary(summary: str, facts: dict) -> list[str]:
    allowed = {f"{v:.2f}" for v in facts.values()
               if isinstance(v, (int, float))}
    allowed |= {str(int(v)) for v in facts.values()
                if isinstance(v, (int, float)) and float(v).is_integer()}
    stated = re.findall(r"\d[\d,]*\.?\d*", summary.replace(",", ""))
    unsupported = [n for n in stated if n not in allowed]
    return unsupported  # empty list means every number checks out

def report_or_rework(summary: str, facts: dict) -> str:
    bad = verify_summary(summary, facts)
    if bad:
        raise ValueError(f"Unsupported numbers, not done: {bad}")
    return summary  # verified, safe to report the decision

The control loop ties it together. Compute, draw, write, verify. Only a clean verification lets the agent report the decision. A failed check sends the summary back to be rewritten from the real numbers, capped at a couple of retries so it cannot loop forever.

Verification checklist

• The chart PNG exists and was drawn from the computed numbers.
• verify_summary returns an empty list for the shipped summary.
• Feed the agent a summary with a made-up figure and confirm the run raises “not done.”
• The recommendation names a real group from the data, not an invented one.

Common failure modes

• Letting the model state the numbers instead of reading them from code. This is the whole trap. Numbers come from compute_metric, always.
• A verification that only checks the total and ignores the per-group figures the decision rests on.
• Rounding in the prose that does not match rounding in the code, so real numbers read as mismatches. Round once, in one place.
• Treating “the summary reads well” as “the summary is correct.”

Extension ideas

• Add a second check that the recommended action follows from the metric, for example that the cut targets the lowest region, not just any region.
• Have the agent flag low confidence when two groups are within a small margin, instead of forcing a decision.
• Swap the model provider and confirm the loop still catches an unsupported number.

Related chapter

Chapter 5, The Workflow and the Control Loop.