What DSPy actually is
DSPy comes out of Stanford NLP, specifically the group that built ColBERT and the original Demonstrate-Search-Predict paper. Despite the name, it’s not an acronym anymore. It’s a programming model for LLM pipelines and the most important thing to understand is what it replaces.
If you’ve built anything with OpenAI’s API you’ve written something like an f-string that mixes instructions, context variables, few-shot examples, and JSON formatting. That prompt lives in your code as a string. When you want to swap models, change the temperature, add an example, or restructure the output, you edit the string. There’s no type checking, no structure, and no systematic way to improve it.
DSPy flips this. You declare typed signatures that describe inputs and outputs, you compose them into modules like ChainOfThought and ReAct, and the library handles how the actual prompt is rendered. Critically, DSPy also ships optimizers (historically called teleprompters) that take a small training set and a metric, then automatically search for better instructions and few-shot demonstrations. The thing that makes this work is the separation between program logic and prompt text. Without that separation the optimizers have nothing to optimize.
The mental model is closer to PyTorch than to LangChain. You write a forward pass, you pick a metric, you call a compile step. The framework generates the actual prompts the way PyTorch generates kernels.
Setup and authentication
The install is a single pip command. The current DSPy release works with Python 3.9 and up.
DSPy needs a language model configured before anything else works. The model is set globally on a configure call and every module picks it up.
For OpenAI the setup reads the OPENAI_API_KEY environment variable, so you just need to export it in your shell or load it from a .env file with python-dotenv. Configure the model with dspy.LM(“openai/gpt-4o-mini”) and then dspy.configure(lm=lm).
For Anthropic you set ANTHROPIC_API_KEY the same way, and you reference the model with the “anthropic/” prefix. DSPy knows about Claude 3.5 Sonnet, Claude 3 Haiku, and newer models as they appear. The call looks like dspy.LM(“anthropic/claude-3-5-sonnet-20241022”, api_key=”…”) if you want to pass the key directly.
For local work the cleanest path is Ollama. Install Ollama, pull a model like llama3.1, and configure DSPy to use “ollama/llama3.1”. This is the path I’d recommend while you’re learning because there’s no per-call cost and you can see the raw prompts DSPy generates in the cache.
One thing people miss: every DSPy call caches by default. The cache lives at ~/.dspy_cache. Disable it with dspy.configure(cache=False) when you actually want to test prompt changes, otherwise you’ll see stale outputs and think your edits did nothing.
First working example
Here’s a complete runnable example. Question answering over a context, the DSPy version of the canonical prompt-engineering demo.
First, define a signature. A signature is a class with input and output fields and a docstring that doubles as the high-level instruction. The code looks like this:
class GenerateAnswer(dspy.Signature): """Answer questions with short factoid answers.""" context = dspy.InputField() question = dspy.InputField() answer = dspy.OutputField()
Then wrap it in a module. ChainOfThought adds a reasoning field before the answer, which usually improves quality on multi-hop questions without you writing a single instruction. The wrapping looks like this:
generate = dspy.ChainOfThought(GenerateAnswer) pred = generate(context=“Paris is the capital of France.”, question=“What is the capital of France?”) print(pred.answer)
Finally, call the module and inspect the prediction. The answer is on pred.answer and if you used ChainOfThought the reasoning is on pred.reasoning.
For RAG you swap the hardcoded context for a Retrieve module. DSPy has built-in retrievers for ColBERTv2, you.com, and a generic RM client that works with anything exposing a /search endpoint. You point it at a corpus once, then call retrieve(“query”) inside your forward pass.
The total code to get a working RAG pipeline is roughly thirty lines, including imports. Compare that to a hand-written version with chunking, embedding, retrieval, prompt construction, and output parsing.
Key settings that matter
Most of the interesting behavior in DSPy is below the surface of the basic Predict call. These are the dials to know.
The adapter controls how signatures get rendered into actual prompts. dspy.ChatAdapter is the default and produces clean message-list prompts. dspy.JSONAdapter forces strict JSON output, useful when downstream code parses the response. Switching adapters is a one-line change and can rescue a pipeline that’s struggling with format compliance.
Assertions let you add runtime constraints. dspy.Assert(answer in valid_set) hard-fails the call if violated and retries with feedback. dspy.Suggest(…) is the soft version that just nudges the model. These are powerful but they add latency on retries, so use them for invariants like “the answer must be one of these categories” rather than stylistic preferences.
The optimizer is where DSPy earns its keep. The three to know are BootstrapFewShot (cheap, good baseline), MIPRO (more expensive but typically the strongest out-of-the-box optimizer), and BootstrapFinetune (compiles a finetuned model rather than prompts). MIPRO works by giving the LM a meta-prompt describing your program and asking it to propose instructions and few-shot demos, then evaluating each proposal against your trainset using the metric you wrote. The compile call is a single line, but under the hood it’s running a Bayesian search over thousands of candidate prompts.
The metric is the part you have to write yourself. It’s a function that takes an example and a prediction and returns a--- title: “DSPy Tutorial: LLM Pipelines the Stanford Way” description: “A hands-on walkthrough of DSPy, the Stanford framework for programmatic LLM pipelines. Setup, signatures, modules, optimizers, and a real workflow.” publishDate: “2026-06-28” author: “Sam McKay” difficulty: “intermediate” service: “general” tags:
- ai-tools
- tutorial draft: false
What DSPy actually is
DSPy comes out of Stanford NLP, specifically the group that built ColBERT and the original Demonstrate-Search-Predict paper. Despite the name, it’s not an acronym anymore. It’s a programming model for LLM pipelines and the most important thing to understand is what it replaces.
If you’ve built anything with OpenAI’s API you’ve written something like an f-string that mixes instructions, context variables, few-shot examples, and JSON formatting. That prompt lives in your code as a string. When you want to swap models, change the temperature, add an example, or restructure the output, you edit the string. There’s no type checking, no structure, and no systematic way to improve it.
DSPy flips this. You declare typed signatures that describe inputs and outputs, you compose them into modules like ChainOfThought and ReAct, and the library handles how the actual prompt is rendered. Critically, DSPy also ships optimizers (historically called teleprompters) that take a small training set and a metric, then automatically search for better instructions and few-shot demonstrations. The thing that makes this work is the separation between program logic and prompt text. Without that separation the optimizers have nothing to optimize.
The mental model is closer to PyTorch than to LangChain. You write a forward pass, you pick a metric, you call a compile step. The framework generates the actual prompts the way PyTorch generates kernels.
Setup and authentication
The install is a single pip command. The current DSPy release works with Python 3.9 and up.
DSPy needs a language model configured before anything else works. The model is set globally on a configure call and every module picks it up.
For OpenAI the setup reads the OPENAI_API_KEY environment variable, so you just need to export it in your shell or load it from a .env file with python-dotenv. Configure the model with dspy.LM(“openai/gpt-4o-mini”) and then dspy.configure(lm=lm).
For Anthropic you set ANTHROPIC_API_KEY the same way, and you reference the model with the “anthropic/” prefix. DSPy knows about Claude 3.5 Sonnet, Claude 3 Haiku, and newer models as they appear. The call looks like dspy.LM(“anthropic/claude-3-5-sonnet-20241022”, api_key=”…”) if you want to pass the key directly.
For local work the cleanest path is Ollama. Install Ollama, pull a model like llama3.1, and configure DSPy to use “ollama/llama3.1”. This is the path I’d recommend while you’re learning because there’s no per-call cost and you can see the raw prompts DSPy generates in the cache.
One thing people miss: every DSPy call caches by default. The cache lives at ~/.dspy_cache. Disable it with dspy.configure(cache=False) when you actually want to test prompt changes, otherwise you’ll see stale outputs and think your edits did nothing.
First working example
Here’s a complete runnable example. Question answering over a context, the DSPy version of the canonical prompt-engineering demo.
First, define a signature. A signature is a class with input and output fields and a docstring that doubles as the high-level instruction. The code looks like this:
class GenerateAnswer(dspy.Signature): """Answer questions with short factoid answers.""" context = dspy.InputField() question = dspy.InputField() answer = dspy.OutputField()
Then wrap it in a module. ChainOfThought adds a reasoning field before the answer, which usually improves quality on multi-hop questions without you writing a single instruction. The wrapping looks like this:
generate = dspy.ChainOfThought(GenerateAnswer) pred = generate(context=“Paris is the capital of France.”, question=“What is the capital of France?”) print(pred.answer)
Finally, call the module and inspect the prediction. The answer is on pred.answer and if you used ChainOfThought the reasoning is on pred.reasoning.
For RAG you swap the hardcoded context for a Retrieve module. DSPy has built-in retrievers for ColBERTv2, you.com, and a generic RM client that works with anything exposing a /search endpoint. You point it at a corpus once, then call retrieve(“query”) inside your forward pass.
The total code to get a working RAG pipeline is roughly thirty lines, including imports. Compare that to a hand-written version with chunking, embedding, retrieval, prompt construction, and output parsing.
Key settings that matter
Most of the interesting behavior in DSPy is below the surface of the basic Predict call. These are the dials to know.
The adapter controls how signatures get rendered into actual prompts. dspy.ChatAdapter is the default and produces clean message-list prompts. dspy.JSONAdapter forces strict JSON output, useful when downstream code parses the response. Switching adapters is a one-line change and can rescue a pipeline that’s struggling with format compliance.
Assertions let you add runtime constraints. dspy.Assert(answer in valid_set) hard-fails the call if violated and retries with feedback. dspy.Suggest(…) is the soft version that just nudges the model. These are powerful but they add latency on retries, so use them for invariants like “the answer must be one of these categories” rather than stylistic preferences.
The optimizer is where DSPy earns its keep. The three to know are BootstrapFewShot (cheap, good baseline), MIPRO (more expensive but typically the strongest out-of-the-box optimizer), and BootstrapFinetune (compiles a finetuned model rather than prompts). MIPRO works by giving the LM a meta-prompt describing your program and asking it to propose instructions and few-shot demos, then evaluating each proposal against your trainset using the metric you wrote. The compile call is a single line, but under the hood it’s running a Bayesian search over thousands of candidate prompts.
The metric is the part you have to write yourself. It’s a function that takes an example and a prediction and returns a