From Prompt Engineering to Programmatic Optimization: A Practical DSPy Primer

There is a specific kind of technical debt that does not feel like debt while you are accumulating it. The prompt file. It starts as a single string in a config somewhere. Someone adds a special case in a comment. Another engineer tweaks it for a regression they caught in production. A third person rewrites the whole thing for a new model version. A year later you have a 400-line instruction block that nobody fully understands and everybody is afraid to touch.

I have been in that situation. The frustrating part is not the complexity itself. It is that the prompt is doing real work. It is the most load-bearing piece of the application and simultaneously the least disciplined artifact in the codebase. No tests. No version history that explains why specific phrasing was added. No systematic way to evaluate whether a proposed change is an improvement or a regression. It gets treated like configuration but it behaves like code.

DSPy has been around since Stanford released it in late 2022. I looked at it twice before and put it down both times. The 3.2.1 release changed my assessment. It ships a feature called BetterTogether that chains prompt optimization and weight optimization into a single pipeline. That is the version that closes a gap I had been waiting on. This post is the practical evaluation I wish had existed before I looked at it seriously.

Table of Contents#

• The prompt engineering problem at production scale
• What DSPy actually is
• The BetterTogether release: what changed in 3.2.1
• When to use DSPy, and when not to
• DSPy vs the alternatives
• Getting started: a practical first project
• What this means for your team's LLM practices
• FAQ

The Prompt Engineering Problem at Production Scale#

Prompt engineering is iterative manual search over a very high-dimensional space. The dimension is not just the wording. It is the structure, the order of instructions, the presence or absence of examples, the level of specificity in the task description, and a dozen other variables that interact in non-obvious ways.

At small scale, that is fine. A good engineer can load the top-20 failure cases into context, iterate for a few hours, and converge on something that handles the observable inputs well. Intuition gets you far when the problem space is small enough to hold in your head.

The problems compound as the system grows.

The prompt that handles your top-20 cases will fail on edge cases you have not seen yet. This is not a criticism of prompt engineering as a practice. It is a statistical fact about manual search. You find a local optimum for the distribution of inputs you have seen. The inputs you have not seen are always a larger set.

Model portability is another real cost. The prompt tuned carefully for one model needs rework for every other model you want to run it on. This is fine on day one with one provider. It becomes expensive when you want to test a cheaper model, or when a provider raises prices and you want to evaluate alternatives. Every migration is a manual re-optimization effort.

The subtler problem is collaborative drift. Engineer A writes a prompt optimized for a specific task structure. Engineer B adds a new requirement and patches the prompt to handle it. The patch fixes the new case and subtly breaks two old ones. Neither engineer knows this because there is no regression suite. The prompt degrades over time in ways that are invisible until a production incident makes them visible.

The root tension: prompts are the most critical piece of your LLM application and the least disciplined artifact in your codebase. They are not tested. They are not versioned in any meaningful way. They have no type system. They are strings. Strings do not have test suites.

What DSPy Actually Is#

DSPy is not a prompting library. This framing matters because if you approach it as a prompting library you will be confused about why it requires an evaluation dataset and why it takes longer to start than just writing a prompt.

DSPy is a framework for building LLM pipelines where the prompt is a variable that gets optimized, not a constant that gets hand-written. You describe the task. You provide labeled examples and a metric. DSPy finds the prompt.

There are three core abstractions.

Signatures define the input/output contract for an LLM call. They specify what goes in, what should come out, and type constraints on the fields. A Signature is a declaration of intent, not a prompt. You are telling DSPy what the task is. The optimizer figures out how to do it.

Modules are composable units that wrap one or more LLM calls with a Signature. The simplest module, dspy.Predict, wraps a single call. dspy.ChainOfThought wraps a call that reasons before producing output. You can compose modules into multi-step pipelines. The optimizer can optimize the whole chain as a unit.

Optimizers (historically called Teleprompters) take a module, a training dataset, and a metric function. They search for the configuration of that module that maximizes the metric. The search space includes prompt phrasing, few-shot examples drawn from your training set, chain-of-thought scaffolding, and, with BetterTogether, model weights.

The mental model I find most useful: DSPy is to LLM pipelines what PyTorch autograd is to neural networks. Autograd makes gradient computation automatic so you do not have to derive it by hand. DSPy makes prompt optimization automatic so you do not have to search for it by hand. You define the objective. The framework handles the search.

What you write: task descriptions, input/output schemas, evaluation metrics.

What DSPy generates: the actual instruction text, the few-shot examples, the chain-of-thought scaffolding.

The BetterTogether Release: What Changed in 3.2.1#

Before 3.2.1, DSPy had two separate optimization paths that did not connect. You could run a prompt optimizer like MIPRO to find better instruction text and few-shot examples. Or you could use DSPy's fine-tuning integrations to update model weights. These were separate workflows. You could not chain them into a single pipeline.

BetterTogether chains them. The concept: run prompt optimization first to find the best prompt variant, then use the optimized prompt to run inference on your training set and collect high-quality input/output pairs, then fine-tune a smaller model on those pairs. The output is a fine-tuned smaller model paired with an optimized prompt, evaluated against your metric on a held-out test set.

Why this matters is about inference cost. A fine-tuned smaller model with an optimized prompt often matches or outperforms a prompted frontier model on domain-specific structured tasks, at 10 to 20 percent of the inference cost. A fine-tuned 7B or 13B model running on your own infrastructure costs a fraction of a per-token API call to a frontier model. For tasks running at volume, the unit economics of that gap compound quickly.

The practical workflow with BetterTogether looks like this:

1. Define your task as a DSPy Signature with typed input fields, output fields, and a plain-language description of the task
2. Collect 50 to 200 labeled examples with ground truth outputs, split into training and validation sets
3. Write an evaluation metric function that scores predictions against ground truth
4. Run MIPROv2 to search for the best prompt variant across your training set
5. Chain into a LoRA fine-tuning pass using BetterTogether, which uses the MIPROv2-optimized prompt to generate fine-tuning data for a smaller target model
6. Evaluate the fine-tuned smaller model against the prompted general model on your held-out test set

Here is a working DSPy pipeline illustrating this pattern. This uses the real DSPy 3.x API and is meant as a conceptual reference:

import dspy
from dspy.teleprompt import MIPROv2, BetterTogether

# Step 1: Configure the language model
lm = dspy.LM("anthropic/claude-sonnet-4-20250514", max_tokens=1024)
dspy.configure(lm=lm)


# Step 2: Define the Signature
# This replaces your hand-crafted prompt with a typed interface.
# The docstring and field descriptors are the only natural language you write.
class SupportTicketClassifier(dspy.Signature):
    """
    Classify a customer support ticket into the most relevant category.
    Return the category label and a one-sentence reasoning for the classification.
    """
    ticket_text: str = dspy.InputField(
        desc="Raw text of the customer support ticket"
    )
    category: str = dspy.OutputField(
        desc="One of: billing, technical, account, general"
    )
    reasoning: str = dspy.OutputField(
        desc="One sentence explaining why this category was chosen"
    )


# Step 3: Define the Module
class TicketClassifierModule(dspy.Module):
    def __init__(self):
        super().__init__()
        self.classifier = dspy.ChainOfThought(SupportTicketClassifier)

    def forward(self, ticket_text: str):
        return self.classifier(ticket_text=ticket_text)


# Step 4: Load training and validation sets
# Each Example has ticket_text (input) and category (ground truth label)
trainset = [
    dspy.Example(
        ticket_text="My invoice shows a double charge from last month.",
        category="billing"
    ).with_inputs("ticket_text"),
    # ... 150+ more examples
]
valset = [...]  # 50 held-out examples not used during optimization


# Step 5: Define the evaluation metric
def accuracy_metric(example, prediction, trace=None):
    return int(
        prediction.category.strip().lower() == example.category.strip().lower()
    )


# Step 6: Configure BetterTogether to chain prompt optimization with fine-tuning
prompt_optimizer = MIPROv2(
    metric=accuracy_metric,
    auto="medium",      # controls number of candidate prompts explored
    num_threads=4,
)

optimizer = dspy.BetterTogether(
    prompt_optimizer=prompt_optimizer,
    weight_optimizer="BootstrapFinetune",   # wraps LoRA fine-tuning
    metric=accuracy_metric,
)

# Step 7: Compile the module
# This runs prompt optimization, generates fine-tuning data,
# and fine-tunes the target smaller model in sequence
compiled_module = optimizer.compile(
    student=TicketClassifierModule(),
    trainset=trainset,
    valset=valset,
    target_model="meta-llama/Llama-3.1-8B-Instruct",
)

# Step 8: Save the compiled program as a versioned artifact
compiled_module.save("ticket_classifier_v1.json")

The SupportTicketClassifier Signature is the only thing you update when the task changes. Everything below it is handled by the optimizer. The compiled_module.save() call is important: it serializes the optimized program, including the discovered prompt text and few-shot examples, into a JSON artifact that can be version-controlled and deployed the same way you would deploy any model checkpoint.

For structured tasks like this, DSPy optimization typically delivers 15 to 30 percent accuracy improvement over hand-crafted prompts. When BetterTogether is used to produce a fine-tuned smaller model, inference cost drops 5 to 10x compared to a prompted frontier model. Those numbers are not guaranteed; they depend on task structure, data quality, and metric definition. But they are representative of what the literature reports and what I have seen on internal pipelines.

When to Use DSPy, and When Not To#

Most DSPy coverage soft-pedals the limitations. I want to be direct here because getting the scoping wrong is expensive.

Use DSPy when#

• The task is well-defined and repeatable. Classification, extraction, structured generation, question answering over a fixed schema. Something that runs hundreds or thousands of times per day with similar input structure. DSPy's optimization overhead amortizes over volume.
• You have or can collect ground truth labels. Fifty examples is enough to start with BootstrapFewShot. Two hundred gives reliable results with MIPROv2. Five hundred or more makes the BetterTogether fine-tuning path viable. Your existing production logs are usually the best source and most teams are not using them.
• You have a measurable metric for success. This is the most important criterion. Exact match, F1, an LLM-as-judge function, a custom classifier. Something that produces a score. If you cannot write metric(example, prediction) -> float, you are not ready for DSPy.
• You are considering fine-tuning anyway. DSPy's programmatic approach generates better fine-tuning data than manual labeling for many structured tasks, because the prompt optimizer automatically finds diverse, high-quality demonstrations. If a fine-tuning run is on your roadmap, running it through DSPy first is almost always worth the setup time.
• Model portability is on your roadmap. Want to test a cheaper model? Change the LM configuration, rerun the optimizer, evaluate on the held-out set. The task definition stays the same. Only the search reruns.

Do not use DSPy when#

• The task is genuinely novel and changes every time. Open-ended generation, creative writing, novel reasoning chains that adapt to unpredictable inputs. DSPy optimizes for a distribution. If there is no stable distribution of inputs, there is nothing to optimize toward.
• You have no ground truth or evaluation metric. This is a hard requirement, not a soft one. There is no workaround. You need something to optimize toward before DSPy can help.
• The pipeline runs infrequently. The optimization run itself costs compute and API calls. For a task that runs 50 times per day, the amortized cost of running MIPROv2 may never pay off versus spending an afternoon on careful manual prompt engineering.
• You are in the exploration phase. DSPy is production infrastructure, not a prototyping tool. If you are still figuring out whether an LLM can even do the task at all, raw prompting is faster, cheaper, and more informative. Come back to DSPy when you have validated the task is solvable and want to systematize the solution.

The single question that cuts through the decision: "Can I write an evaluation function for this task right now?" If yes, DSPy can probably improve it. If the answer is "I will know good output when I see it," you are not there yet.

DSPy vs the Alternatives#

The comparison with LangChain and LangGraph comes up constantly and is worth addressing directly. They solve different problems. LangChain and LangGraph are orchestration frameworks. They help you wire together stateful multi-step pipelines with memory, tool calls, and branching logic. DSPy is an optimization framework. It finds the best prompt for a given task node.

They are complementary. A DSPy-optimized module can run inside a LangGraph workflow. The DSPy module handles optimization of an individual LLM step. LangGraph handles how that step connects to the rest of the pipeline. If you are building the kinds of systems described in the agentic LLM workflow patterns post, DSPy sits below the orchestration layer and improves the individual nodes.

The OpenAI fine-tuning API comparison is more nuanced. If your entire stack is OpenAI and you do not need cross-model portability, it is simpler to start with. You do not need to learn DSPy's abstractions. The trade-off is flexibility and ecosystem lock-in. DSPy supports OpenAI, Anthropic, and any model accessible via LiteLLM, which includes hundreds of providers and local model endpoints. If model portability matters to you, and the LLM cost engineering post makes a strong argument for why it should, DSPy's abstraction layer pays for itself.

The summary: use DSPy for optimization of repeated structured tasks. Use LangGraph for orchestration. Use raw prompting for exploration. Use a prompt management platform for deployment governance. These are not competing choices. They are layers in a stack.

Getting Started: A Practical First Project#

The most common evaluation mistake is picking a task that is too complex or too open-ended. The right first project is boring on purpose.

Pick the right task. Classification or extraction tasks are ideal. Inputs are structured, outputs are constrained, and "correct" is well-defined. If you have a service that routes support tickets, extracts fields from documents, or classifies user intent, that is your first DSPy project. A task that already uses a hand-crafted prompt, runs at volume, and has clear ground truth is the exact right starting point.

Collect your evaluation set. Pull 100 to 200 examples from your production logs. You need both the input and the correct output for each example. Split them: 70 percent for training, 30 percent held out for evaluation. Never evaluate on examples you trained on. The evaluation setup usually takes longer than the DSPy code itself. That is not a problem. The evaluation setup is the valuable part.

Write the Signature. Define input fields and output fields. Add a plain-language description to the class docstring. Add short descriptors to each field. Spend real time on this. The Signature description is the signal DSPy has about what you want. A vague Signature produces a vague optimization.

Start with BootstrapFewShot, not MIPROv2. BootstrapFewShot is faster and requires less compute. It works by selecting the most useful few-shot demonstrations from your training set. Run it, evaluate on the held-out test set, and compare against your current hand-crafted prompt on the same test set. If the DSPy-optimized version is not better than your baseline, something is wrong with the metric or the data. You want to find that out before investing in a full MIPROv2 or BetterTogether run.

Evaluate rigorously. The number that matters is performance on the held-out test set, not on training examples. DSPy optimizers can overfit to the training distribution. The test set is your reality check.

Realistic time investment for a first working pipeline including evaluation setup: two to four hours for an engineer comfortable with Python and LLM APIs. That is a reasonable window for deciding whether DSPy fits your use case.

What This Means for Your Team's LLM Practices#

The implications of taking DSPy seriously go beyond the optimization runs themselves.

Prompts should be in version control with a change history that explains why. DSPy makes this natural. An optimized DSPy module is a serializable JSON artifact you can check in, diff, tag, and roll back. This is a structural improvement over a prompt in a Python string literal with a commit message that says "improved for better accuracy."

Evaluation sets should be a first-class deliverable. Not something you build after problems arise in production. When an engineer ships an LLM feature, the evaluation set should ship with it. Same discipline as shipping tests with code. DSPy makes this a hard requirement because the optimizer cannot run without it. That constraint, frustrating when you first hit it, is the discipline that makes the system work.

The prompt-as-code model. Treat LLM task definitions the same way you treat data models. Typed inputs and outputs. A schema reviewed when it changes. The actual prompt text is a generated artifact. You do not hand-edit generated artifacts.

The teams investing in this kind of discipline now are building a compounding advantage. Every model upgrade gets re-evaluated automatically against the existing metric. Every new task added to the pipeline gets optimization from day one. The infrastructure cost is paid once. The benefit recurs on every inference call.

That compounding effect is exactly what the agentic apps build and run cost post discusses in terms of where optimization investment pays off most at scale. An under-optimized prompt on a high-volume task adds up. A 20 percent improvement in token efficiency, multiplied across millions of calls, is a real number.

FAQ#

Does DSPy work with all LLMs?#

DSPy uses LiteLLM under the hood for model access, which means it supports OpenAI, Anthropic, Google, Cohere, Mistral, and most local model endpoints that follow the OpenAI API format. You configure the model once with dspy.configure(lm=dspy.LM("provider/model-name")) and the rest of the code is model-agnostic. Switching models to compare performance is a one-line change. Cross-model portability is one of the cleaner aspects of the API compared to provider-specific fine-tuning tools.

How much data do I need?#

Fifty labeled examples is enough to run BootstrapFewShot and get a meaningful signal. Two hundred or more is where MIPROv2 starts producing reliable optimization results. For the BetterTogether fine-tuning path, five hundred examples is a reasonable minimum, though the actual requirement depends on task complexity and target model size. Your existing production logs are almost always the best starting point. If you have been running a hand-crafted prompt in production for any length of time, you likely have more labeled data than you realize. You just have not organized it yet.

Does DSPy replace prompt engineers?#

No, and framing it as replacement misses the point. The job changes rather than disappears. Less time goes into writing and iterating on instruction text. More time goes into defining task schemas, writing evaluation metrics, curating training data, and analyzing edge cases in evaluation sets. Those activities are more valuable because they are reproducible and they compound across the whole pipeline. The prompt engineer who can write a good evaluation metric and design a representative training set is more valuable in a DSPy-oriented workflow, not less.

What is the risk of using an optimizer-generated prompt in production?#

The same risk that applies to any generated artifact: you need to test it on a genuinely held-out evaluation set before deploying it, and you need to monitor it after. The specific risk to watch for is overfitting to the training distribution. An optimizer can find prompts that perform very well on your training examples and poorly on inputs slightly outside that distribution. A held-out test set that was not used during optimization is the mitigation. DSPy makes it easier to do this correctly because the evaluation infrastructure is built into the framework. You still have to use it.

Is BetterTogether production-ready in 3.2.1?#

Treat it as early-access for now. The API is stable but BetterTogether is the newest part of the framework and has seen less production validation than the core optimizers. My recommendation is to run the fine-tuned model in shadow mode alongside your current general model for a validation period before cutting over. Shadow mode means both models run on every input and you compare results without serving the fine-tuned model's output. Do not replace a production general model with a fine-tuned one based solely on training and validation set metrics. The optimization quality is real. The operational maturity is still developing.

Can I use DSPy for multi-step pipelines rather than just single LLM calls?#

This is where the Module abstraction shows its value. You build a pipeline by composing Modules. When you run an optimizer on the top-level Module, it optimizes the entire chain. The optimizer can discover prompt variants at each step that work well together as a system, which is meaningfully different from optimizing each step in isolation. For multi-step structured tasks, the system-level optimization is one of the primary reasons to use DSPy. The agentic LLM workflow patterns post covers how these optimized nodes fit into larger agent architectures, and which patterns benefit most from having each node individually optimized.