Trace and Diagnose a Failed Run

What you will build

Not an agent this time. A diagnosis. You take a real failed run, read its execution log step by step, find the exact step where it went wrong, and name the root cause. Then you do the part that stops the failure from returning. You freeze that run as a new case in your evaluation set, so the next time anyone changes the prompt, the tools or the model, this failure has to pass before it ships.

Why this lab matters

Agents fail in production. The ones that survive are not the ones that never fail. They are the ones where every failure gets traced, understood and turned into a test. A failure you cannot read is a failure you will repeat. A failure you fix once but never test for will come back the first time someone swaps the model. This lab teaches the loop that turns a bad run into a permanent guardrail.

Expected output

• A written root-cause analysis of one failed run, naming the step where it broke.
• A one-line verdict: context problem, tool problem or model problem, with the evidence from the trace.
• A new evaluation case that reproduces the failure and passes once the fix is in.

Prerequisites

• You have read Chapter 9, Deploy, Observe and Improve.
• You can run a small Python script and set an environment variable.
• You have an agent that logs its steps, or you use the sample trace below.

Steps

1. Get the trace. A trace is not a mystery. It is structured logging you add to the loop. Every time the agent thinks, calls a tool or answers, you write one record. The shape is small.

def log_step(run_id, kind, content):
    # kind is one of: model, tool_call, tool_result, answer
    print({"run": run_id, "kind": kind, "content": content})

2. Read a real failure. Here is the log from a support agent asked to check an order. Nothing crashed. The run reported done. That is the worst kind of failure, a confident wrong answer with a green light.

run 2f9c  agent order-support  status: reported "done"

[1] user
    "Where is my order #A-2231?"

[2] model -> tool_call
    lookup_order(order_id="A2231")

[3] tool_result  lookup_order  41ms
    {"found": false, "error": "no order found for A2231"}

[4] model -> answer
    "Good news. Your order #A-2231 shipped on Tuesday and
     should arrive by Friday."

[5] run marked done

3. Find the step where it broke. Do not read from the top and stop at the first odd thing. Start at the bad output and walk backward. Step 4 is the visible harm, an invented shipping date. Step 3 is a clean tool result, the order genuinely was not found for A2231. Step 2 is the first real defect, the model dropped the dash and searched A2231 instead of A-2231. So there are two wrong steps. The first wrong step is 2. The dangerous step is 4. Always separate those two. The earliest wrong step is your root cause. The dangerous step tells you how much it hurt.

4. Name the root cause. Ask three questions in order and stop at the first yes.

• Did the agent ever get the right information, in a form it could use? If the system prompt never told it to pass the order id exactly as written, this is a context problem. Fix the prompt or the retrieval.
• Did the agent get the right information but a tool return wrong or unclear data? Here lookup_order returned a soft found: false that is easy to skim past. A weak tool contract is a tool problem. Fix the tool or make its failure impossible to ignore.
• Did the agent get the right information, with a clear tool result, and still reason wrong? Step 4 saw found: false and asserted a shipping date anyway. That is a model problem the model owns. Fix it with a tighter instruction, a check, or a different model.

5. Write the verdict. One honest line. For this run: “Root cause at step 2, an id normalisation that dropped the dash, made worse at step 4 where the model turned a not-found result into a confident status. Primary fix belongs at the verification layer, an order answer must be backed by a real order record.”

6. Turn the fix into an evaluation case. This is the point of the lab. You freeze the failed run so it can never pass silently again. The case pins the input, stubs the tool to return exactly what the trace showed, and asserts the agent must not fabricate a status.

# eval_cases.py: the failed run, frozen as a regression test

CASE = {
    "name": "not_found_order_is_never_fabricated",
    "input": "Where is my order #A-2231?",
    "tool_stub": {"lookup_order": {"found": False, "error": "no order found"}},
    "must_not_say": ["shipped", "arrive", "delivered", "on its way"],
    "must_admit": ["could not find", "couldn't find", "no record"],
}

def grade(case, answer: str) -> list[str]:
    text = answer.lower()
    failures = []
    for banned in case["must_not_say"]:
        if banned in text:
            failures.append(f"fabricated a status: '{banned}'")
    if not any(ok in text for ok in case["must_admit"]):
        failures.append("did not admit the order was not found")
    return failures  # empty list means the fix holds

7. Run it against the old behaviour and the fixed behaviour. Against the trace answer from step 4, grade returns failures, the case reproduces the bug. Fix the agent so a found: false result forces a “could not find it” reply, run again, and grade returns an empty list. Now commit the case into the eval set that runs before every deploy.

Verification checklist

• You can name the first wrong step and the dangerous step from the trace, and they are not the same step.
• Your verdict labels the root cause as context, tool or model, and quotes the line from the trace that proves it.
• The eval case returns failures when graded against the original fabricated answer.
• The eval case returns an empty list when graded against the fixed answer.
• The case is committed to the eval set that runs before deploy, not left in a scratch file.

Common failure modes

• Reading the trace from the top and blaming the first odd line, instead of tracing back from the harm.
• Blaming the model when the trace shows it never got the right context. Read the tool results before you judge the reasoning.
• Fixing the one order id that failed and not the class of failure, so it returns with a different id.
• Patching the prompt and never writing a test, so the next model swap regresses in silence.

Extension ideas

• Add a timestamp and token count to each step so you can spot slow or oversized steps, not only wrong ones.
• Group your eval cases by root-cause type and track how many context, tool and model failures you have caught.
• Re-run the whole eval set against a cheaper model and see which fixes still hold.

Related chapter

Chapter 9, Deploy, Observe and Improve.