Capstone: Multi-Step Operations Agent

What you will build

One agent that runs a real business workflow from end to end, using every piece from the earlier labs. It takes an incoming request, gathers the data it needs, does the work across several tools, stops for human approval on the one high-risk step, verifies the outcome actually happened, and logs a trace of the whole run. Then it proves itself against a small evaluation set of past cases with known correct answers.

The example used here is a refund request. A request comes in, the agent looks up the order and the policy, decides eligibility from the real numbers, proposes the refund, waits for a person to approve it, confirms the money actually moved, and records every step. Swap in your own workflow if you have one. The architecture does not change.

Why this lab matters

Every earlier lab built one muscle in isolation. Context, tools, a control loop, a completion check, a human gate. On their own each is a demo. A business-ready agent is all of them wired into one loop that can survive a real request and leave a record behind. This is the formula made whole: Model plus Context plus Tools plus Workflow plus Verification, with tracing and an evaluation set on top so you can trust it tomorrow, not just today.

An agent is only useful when it can see the work, do the work, and prove the work was done. This lab is where you prove all three at once.

Expected output

• An operations agent that runs a multi-step workflow from a single incoming request.
• Tools with narrow contracts, every call logged to one trace.
• A control loop that stops at the high-risk step and refuses to act without approval.
• A verification step that confirms the outcome before the run reports done.
• An evaluation set of past cases the agent is scored against.

Prerequisites

• You have finished the earlier labs, or can reuse their pieces: the tool contract from Chapter 4, the control loop from Chapter 5, and the approval machine from Chapter 7.
• You can run a Python script and set an environment variable.
• You have one real multi-step workflow in mind. Chapter 2 is where you picked it.

Steps

1. Write the workflow down as a fixed list of steps before any code. Intake, gather, decide, propose, approve, act, verify, log. A named finish line is what makes it checkable.
2. Give the agent the right context and nothing more, the lesson from Chapter 3. The request, the order record, the policy. Not the whole database.
3. Register each tool behind a narrow contract, the lesson from Chapter 4. Every tool call goes through one wrapper that logs it to the trace.

import time, uuid

def new_trace(workflow):
    return {"trace_id": str(uuid.uuid4()), "workflow": workflow, "events": []}

def log(trace, step, actor, detail=None):
    trace["events"].append({
        "at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "step": step, "actor": actor, "detail": detail,
    })

def call_tool(trace, tools, name, **kwargs):
    if name not in tools:
        raise KeyError(f"unknown tool {name}")
    out = tools[name](**kwargs)
    log(trace, f"tool:{name}", "agent", {"in": kwargs, "out": out})
    return out

4. Run the control loop, the lesson from Chapter 5. The loop drives the steps in order. The decision comes from code and the real numbers, never from the model guessing. At the high-risk step the loop stops.

def run_workflow(request, tools, trace):
    log(trace, "intake", "agent", request)
    order = call_tool(trace, tools, "lookup_order", order_id=request["order_id"])
    policy = call_tool(trace, tools, "lookup_policy", reason=request["reason"])
    decision = decide_eligibility(order, policy)   # numbers from code, not the model
    log(trace, "decision", "agent", decision)
    if not decision["eligible"]:
        return {"state": "declined", "trace": trace}
    # High-risk step. Gated by the Chapter 7 approval machine.
    proposal = propose("issue_refund",
                       {"order_id": order["id"], "amount": decision["amount"]})
    log(trace, "proposed", "agent", proposal["payload"])
    return {"state": "held", "proposal": proposal, "trace": trace}

5. Gate the risky step with the approval machine from Chapter 7. The loop hands back a held proposal and stops. A person reads it and calls approve or reject. Only an approved proposal reaches the action. The gate is the same one you already built, so it is not repeated here.

6. Verify the outcome, the thread running through the whole playbook. After the approved action runs, do not trust that it worked. Look it up and confirm it. A run that cannot confirm its own result is not done.

def verify_outcome(trace, tools, proposal):
    posted = call_tool(trace, tools, "lookup_refund",
                       order_id=proposal["payload"]["order_id"])
    ok = bool(posted) and posted["amount"] == proposal["payload"]["amount"]
    log(trace, "verify", "system",
        {"expected": proposal["payload"]["amount"], "found": posted, "ok": ok})
    if not ok:
        raise ValueError("refund not confirmed, run is not done")
    return ok

7. Score the agent against an evaluation set. Collect a handful of past requests where you already know the right answer. Run the agent over all of them and count how many it gets right. This is what tells you a prompt change made things better and not worse.

def evaluate(cases, tools):
    passed = 0
    for case in cases:
        trace = new_trace("refund")
        out = run_workflow(case["request"], tools, trace)
        got = out["state"]           # "declined" or "held"
        if got == case["expected"]:
            passed += 1
    return {"passed": passed, "total": len(cases),
            "rate": round(passed / len(cases), 2)}

Verification checklist

• One incoming request drives the full run, intake through verify, with no manual steps in the middle except the approval.
• Every tool call and every decision appears in the trace, in order, with an actor.
• A rejected proposal never reaches the action, and the trace shows the block.
• verify_outcome fails the run when the refund did not actually post.
• The evaluation set runs clean, and a deliberately broken decision rule drops the pass rate.

Common failure modes

• Letting the model make the eligibility call instead of code. The numbers decide, not the paragraph.
• Skipping verification because the action returned success. A returned value is not a confirmed outcome.
• A trace that logs only the happy path. The blocked and failed events are the ones that prove the guardrails hold.
• An evaluation set of cases you have never seen fail. If it always passes, it is not testing anything.
• Rebuilding the approval gate loosely for the capstone. Reuse the one from Chapter 7 exactly.

Extension ideas

• Add a second grader to the evaluation set that checks the refund amount, not just the eligible or declined outcome.
• Move the human step to a Slack approval, keeping the gate and the trace unchanged.
• Store each trace to disk keyed by trace id, so you can replay any past run and see exactly what happened.

Related chapter

Chapter 10, The AI Operating Layer.