Build a Document-to-Report Agent

What you will build

An agent that takes a folder of messy source material, meeting notes, a call transcript, a couple of loose files, and turns it into one clean structured report. The report has real sections. The point of the lab is the last step. Before the agent says it is done, it checks that every claim in the report points back to a document it actually read. A confident report built on invented facts is worse than no report at all.

Why this lab matters

This is the failure that quietly ruins agent projects. The model reads a few files, writes a tidy summary, and slips in a number or a name that was never in the sources. It reads well, so nobody checks. In Chapter 4 you gave the agent safe, clear tools. Here you see why the tool contract matters. The document reader hands back stable source ids, and the report is only allowed to say things it can trace to one of them.

Expected output

• A structured report of five to eight sections built from your input documents.
• Every factual claim tagged with the source id it came from.
• A grounding check that fails the run if any claim has no source, or points to a source that was never read.

Prerequisites

• You have read Chapter 4, Give the Agent Safe and Clear Tools.
• You can run a small Python script and set an environment variable.
• You have three to five real documents to feed it. Rough notes are fine. Messy is the point.

Steps

1. Put your source files in one folder. Text, markdown, or a plain transcript. Give each a clear filename.
2. Write a document-reading tool with a narrow contract. It returns the text and a stable source id, and nothing the agent does not need.
3. Have the agent read all sources first and hold each one under its id. No writing yet.
4. Ask for the report as structured data, not free prose. Each section carries its claims, and each claim carries the source id it came from.
5. Run the grounding check. Walk every claim, confirm its source id was really read and the claim text actually appears in that source.
6. If the check fails, the agent goes back, fixes or drops the unsupported claim, and runs the check again before it reports done.

Here is the shape of the reader tool. Short and honest, not a framework.

import os
from dataclasses import dataclass

@dataclass
class SourceDoc:
    source_id: str   # stable handle the report will cite, e.g. "doc-02"
    name: str        # original filename, for the human
    text: str        # full extracted text

def read_documents(folder: str) -> dict[str, SourceDoc]:
    """Read every file in a folder into a source map keyed by source_id.

    The contract is deliberately narrow. It only reads inside `folder`,
    it never writes, and it hands back ids the report is required to cite.
    """
    sources = {}
    files = sorted(f for f in os.listdir(folder) if f.endswith((".txt", ".md")))
    for i, name in enumerate(files, start=1):
        source_id = f"doc-{i:02d}"
        path = os.path.join(folder, name)
        with open(path, "r", encoding="utf-8") as fh:
            text = fh.read()
        sources[source_id] = SourceDoc(source_id, name, text)
    return sources

Now the completion check. The report comes back as sections, each holding claims, each claim naming a source id. The check confirms two things. The id was really read, and the claim is grounded in that source rather than invented.

def check_grounded(report: dict, sources: dict[str, SourceDoc]) -> list[str]:
    """Return a list of problems. Empty list means the report is grounded."""
    problems = []
    for section in report["sections"]:
        for claim in section["claims"]:
            sid = claim.get("source_id")
            if not sid:
                problems.append(f"No source on claim: {claim['text'][:80]}")
                continue
            if sid not in sources:
                problems.append(f"Claim cites unknown source {sid}")
                continue
            # Cheap grounding signal. Do the key terms of the claim
            # actually appear in the source it points to?
            haystack = sources[sid].text.lower()
            terms = [w for w in claim["text"].lower().split() if len(w) > 4]
            hits = sum(1 for w in terms if w in haystack)
            if terms and hits / len(terms) < 0.3:
                problems.append(f"Claim not supported by {sid}: {claim['text'][:80]}")
    return problems

problems = check_grounded(report, sources)
if problems:
    raise SystemExit("NOT DONE. Grounding failed:\n" + "\n".join(problems))
print("Report grounded. Every claim maps to a source it was read from.")

The term-overlap test is a blunt instrument. It will not catch a claim that reworded the source into something false. It does catch the common failure, a claim pointing at a source that never mentioned it. Treat it as a floor, not a ceiling. The Extension ideas below tighten it.

Verification checklist

• The report exists and has the section count you asked for.
• Every claim carries a source id.
• Every cited source id is one the reader actually returned.
• The run prints “NOT DONE” and stops when you plant a fake claim.

Test that last one on purpose. Add a claim with a made-up number and a real source id. A grounding check you have never seen fail is not a check you can trust.

Common failure modes

• The agent writes the report first and back-fills source ids to whatever looks close. Read first, write second, and make the id come from the read step.
• Passing the full text of every document into one prompt and losing track of which fact came from where. Keep the source map and cite against it.
• Treating “the report was produced” as “the report is correct.” Produced and grounded are two different states.

Extension ideas

• Replace the term-overlap test with a second model pass that reads the claim and its cited source and answers supported or not supported.
• Add a coverage check. Flag any source document that no claim drew from, so nothing important is silently dropped.
• Log every failed run with the claims that failed, so you can see which sources the agent keeps misreading.

Related chapter

Chapter 4, Give the Agent Safe and Clear Tools.