Model Routing for Practitioners

I have reviewed a lot of production AI stacks over the past year. The pattern I see most often: one model, wired to everything. The same frontier model that handles complex multi-step planning also handles formatting a JSON response or classifying whether a user input is a question or a command. The team knows this is wasteful. They just haven't made time to fix it.

That is a mistake worth fixing. Not because it is theoretically inelegant, but because principled routing typically recovers 40 to 80 percent of your inference budget without touching quality. The math is hard to ignore once you work it out for your actual workload.

Model routing infographic showing task classification, cheap model path, frontier escalation path, quality checks, and budget monitoring.

Why Most Teams Do not Route#

A recent survey put the number at over 75 percent of production teams using multiple models. But almost none of them have routing logic that is principled, consistent, and monitored. Most have ad hoc decisions baked into prompts or picked by whoever wrote that part of the code. There is no coherent strategy.

The reason is measurement. Teams track cost per token. They do not track cost per successful completed task. That distinction matters more than most people realize.

An agent that runs cheaply per token but loops three times, retries twice on formatting failures, and eventually escalates to a stronger model is not actually cheap. A system that routes correctly the first time, completes the task on one call, and caches intelligently is dramatically cheaper even if the per-token rate is higher on some calls. Fix your cost metric first. Everything else follows.

The Routing Matrix#

Here is how I think about model selection across task categories.

Classification, routing, formatting, extraction. These tasks do not need frontier intelligence. Claude Haiku 4.5 at $1 per million input tokens and $5 per million output tokens handles them at 90-plus percent quality with well-crafted prompts. Gemini 2.5 Flash at $0.30 per million input and $2.50 per million output is even cheaper and faster. Both consistently hit under 600ms time to first token in benchmarks. For speed-sensitive pipelines where you need sub-second response on simple ops, Gemini 2.5 Flash is the default.

Standard generation, RAG, tool use. This is the midrange tier: GPT-4o or Claude Sonnet 4.6. Strong across domains, handles structured output reliably, good instruction-following. For tool-heavy workflows specifically, GPT-4o has strong structured output compliance which reduces the formatting failures that drive retry costs up.

Complex reasoning, multi-step planning, architectural decisions. This is where Claude Opus 4 and above earns its cost. Not every task needs it. Most do not. But when you need deep reasoning chains with minimal backtracking, it is the right tool.

Here is what those tiers actually cost once you hold the task shape constant.

Bar chart comparing the cost of running 100,000 identical tasks of 8,000 input and 800 output tokens across four models at published list prices, showing Gemini 2.5 Flash at $440, Claude Haiku 4.5 at $1,200, Claude Sonnet 4.6 at $3,600, and Claude Opus 4 at $18,000, a 41x spread between the cheapest and the frontier tier.

The routing question is always: what is the cheapest model that completes this task correctly on the first try?

Three Implementation Patterns#

Pre-request rules. Classify the incoming request with a lightweight model, then route based on the classification. This is the cheapest approach, the most predictable, and the right starting point for most teams. The classifier itself should run on Haiku or Gemini Flash. You are spending fractions of a cent to save dollars.

At-inference cascades. Start with a cheap model. If the response confidence is low or the output does not pass your quality checks, escalate to a stronger model. This pattern gives you the best cost-quality balance for workloads where task difficulty is genuinely variable. The tradeoff is latency: you take two model calls on hard tasks instead of one.

Post-response retry. Run with your primary model, detect failure through format validation or a lightweight quality scorer, then re-run with a stronger model. This is a safety net, not a primary routing strategy. Use it at the edges of your pipeline where unexpected inputs slip through your upstream classifier.

Most production systems end up combining these. A pre-request classifier handles the 70 percent of easy cases. Cascades manage the ambiguous middle. Post-response retry catches the stragglers.

Diagram comparing three routing implementation patterns as call sequences, showing that pre-request rules classify before spending and always cost two calls, at-inference cascades cost one call on easy tasks and two on hard ones with added latency, and post-response retry pays for a full second call every time its validator fires.

A Simple Routing Function#

Here is the pattern I use as a starting point. It is deliberately simple. Add complexity only when you have data showing the simple version is failing.

python code-highlight

from enum import Enum
from dataclasses import dataclass

class TaskTier(Enum):
    SIMPLE = "simple"       # classify, format, extract, route
    STANDARD = "standard"   # generate, RAG, tool use
    COMPLEX = "complex"     # planning, reasoning, architecture

@dataclass
class RoutingDecision:
    model: str
    tier: TaskTier
    rationale: str

ROUTING_TABLE = {
    TaskTier.SIMPLE: "claude-haiku-4-5",      # or gemini-2.5-flash for speed priority
    TaskTier.STANDARD: "claude-sonnet-4-6",
    TaskTier.COMPLEX: "claude-opus-4",
}

def classify_task(task_description: str, input_tokens: int) -> TaskTier:
    """
    Heuristic classifier. Replace with a trained model once you have labeled data.
    """
    simple_signals = [
        "classify", "format", "extract", "parse", "route",
        "yes or no", "true or false", "categorize", "tag"
    ]
    complex_signals = [
        "plan", "architect", "design system", "multi-step",
        "reason through", "analyze tradeoffs", "research"
    ]

    desc_lower = task_description.lower()

    if any(s in desc_lower for s in complex_signals) or input_tokens > 50_000:
        return TaskTier.COMPLEX

    if any(s in desc_lower for s in simple_signals) and input_tokens < 2_000:
        return TaskTier.SIMPLE

    return TaskTier.STANDARD

def route(task_description: str, input_tokens: int) -> RoutingDecision:
    tier = classify_task(task_description, input_tokens)
    model = ROUTING_TABLE[tier]
    return RoutingDecision(model=model, tier=tier, rationale=f"Classified as {tier.value}")

# Usage
decision = route("classify this support ticket as bug, feature request, or question", 150)
print(f"Model: {decision.model}")  # claude-haiku-4-5

Start with heuristics. Once you have a few hundred labeled routing decisions, train a small classifier using the MasRouter approach from arXiv 2502.11133: a learned routing model that predicts task difficulty from input features. That paper shows measurable improvements over static rules in multi-agent systems. The heuristic approach I have written above is your training data collection phase.

The Cache Interaction Problem#

There is a subtlety in routing that bites teams when they first implement it: prompt caching interacts badly with mid-conversation model switching.

When you route different turns of the same conversation to different models, you bust your cache. The long system prompt you've been caching for 80 percent of your requests no longer matches. For Manus AI's production workload, where agents average around 50 tool calls per task with roughly a 100:1 input to output token ratio, this matters enormously. Their inference cost is dominated by input token volume, not output. Losing cache hits on that input volume is expensive.

Side by side cost tables for a six turn conversation over a 20,000 token shared prefix, showing that switching models every turn misses the cache on all six turns and costs $0.360, while routing once per conversation writes the prefix once and reads it five times at a tenth of the input rate for $0.105, making the stable route 3.4x cheaper.

The solution is not to avoid routing. It is to route at the conversation level, not the turn level. Classify the task before the conversation starts, pick a model for the session, and stick with it. Reserve mid-conversation model switching for explicit escalation flows where cost savings justify the cache hit loss.

The Infrastructure Layer#

Routing logic does not live in your application code if you are doing this seriously. It lives in an AI gateway. Portkey and Helicone are the two I use most. Both support proxy-based routing with minimal setup, give you unified cost dashboards across providers, handle fallbacks when a provider has latency spikes, and manage credentials so you are not scattering API keys across your codebase.

The gateway is also where you connect routing decisions to your eval pipeline. When a routed call produces a low-quality output, that signal feeds back into your routing classifier training data. Without that loop, you are routing blind.

Where to Start#

Audit your current spend by task type. Not by model, by the actual tasks those model calls are performing. Most teams find that 40 to 60 percent of their frontier model calls are on classification, formatting, or extraction work that a cheap model handles just as well.

Then work through these in order:

Implement a pre-request classifier for your highest-volume simple tasks and route them to Haiku or Gemini Flash. Measure quality for two weeks before expanding.
Pick a cost metric that actually reflects your system: cost per successful completed task, not cost per token. Set up tracking for it.
Add an AI gateway in front of your model calls. Get unified observability before you add routing complexity.
Once you have labeled routing data from the heuristic approach, train a small classifier and evaluate whether it beats your rules.

Model routing is not glamorous engineering. But it is one of the highest-return things you can do in a production AI system. The teams that get this right spend their inference budget where it actually moves the needle.

How I Would Run This in Production#

I would not start by building a clever router. I would start by inventorying tasks. Label each model call by job: classification, extraction, rewriting, planning, code generation, tool selection, summarization, or final answer synthesis. Most teams discover that half their frontier calls are doing routine work.

Then build a static routing table before using a learned router. A static table is easier to reason about and easier to debug. Classification goes to a cheap model. High-risk planning goes to a stronger model. Ambiguous cases escalate. Once this works, you can add confidence-based routing.

Escalation rules are the important part. Cheap models are fine until they are not. Escalate on low confidence, schema repair loops, policy-sensitive content, user-visible high-stakes output, or disagreement between a cheap model and a verifier. Routing without escalation is just cost cutting with a nicer name.

Finally, test routing changes against the eval suite. A new routing policy is a behavior change. It should have a version, a rollout, and a rollback path. Teams often treat routing as infrastructure and forget that it changes product quality.

What I Would Measure#

Track cost per task type, quality score per task type, escalation rate, false cheap-pass rate, and latency by route. The false cheap-pass rate is the one to watch: cases where the cheap model was allowed through but a stronger review would have caught a problem.

I would also track cache impact. Routing can break caching if each model path receives a differently structured prompt. Cost savings from cheaper models can get eaten by poorer cache behavior if prompt layouts are unstable.

The business metric is not average cost per call. It is cost per successful workflow. A cheap model that causes retries, repairs, or user escalation may be more expensive than a stronger model used once.

Where This Connects#

This post is one piece of the production agent lifecycle. The adjacent pieces matter because the failure usually does not stay inside one layer. A tracing problem becomes an eval problem. An eval problem becomes an ownership problem. A routing problem becomes a cost and security problem if nobody can see what changed.

If you are using this as a checklist, read those posts as dependencies rather than as optional background. The stack only becomes reliable when the evidence loop, the release loop, and the security loop all point at the same production behavior.

FAQ#

What is model routing?#

Model routing is the practice of sending different tasks to different models based on difficulty, risk, cost, latency, and quality requirements. The goal is to spend frontier-model budget only where it matters.

Which tasks can usually use cheaper models?#

Classification, formatting, simple extraction, short summaries, routing decisions, and low-risk rewriting often work well on cheaper models. Complex planning, code changes, and high-stakes decisions usually need stronger models or review.

How do I know when to escalate?#

Escalate on low confidence, policy-sensitive content, schema repair loops, tool disagreement, customer impact, or cases where the cheap model output fails a verifier. Escalation rules should be explicit.

Can routing hurt quality?#

Yes. Poor routing can send hard tasks to weak models and create subtle quality loss. That is why routing policies need eval coverage, shadow testing, and production monitoring.

Does routing increase latency?#

Sometimes. A simple static route can reduce latency by using faster models. Multi-stage routing can increase latency if it runs classifiers and verifiers. Measure workflow latency, not just model latency.

Should the router itself be an LLM?#

Not at first. Start with rules and task labels. Move to an LLM router only when the routing decision is too nuanced for rules and you have evals to test router quality.

Implementation Review Checklist#

Before I would call this production-ready, I would ask five questions in the review. First, does the team know which artifact changed because of this lesson? A trace field, eval case, prompt file, routing rule, dashboard, or security policy should be visible in the repo or the runbook. If the only output is agreement, the lesson has not landed yet.

Second, can the team reproduce the failure mode in a lower environment? Production AI work gets messy when the only evidence is a screenshot, a Slack thread, or a vague user complaint. A reproducible example gives the team something concrete to test against after the fix.

Third, does the fix have an owner after the first merge? Most agent systems decay slowly. Datasets go stale, prompts drift, routing rules stop matching traffic, and permissions expand. The owner is the person who notices that drift before it becomes an incident.

Fourth, is rollback obvious? If a prompt, model, parser, evaluator, or security rule makes behavior worse, the team should know how to return to the last known-good version without rewriting the system under pressure.

Fifth, did the incident or improvement strengthen the loop? A good production process compounds. The trace becomes an eval. The eval becomes a gate. The gate becomes a safer release. That is the practical standard I would use for every idea in this series.

I would also ask whether the change makes the system easier to explain to a new teammate. Production AI stacks fail when the important behavior lives in someone s memory. If the reason for a prompt rule, routing threshold, eval example, or tool permission is not written down, the next person will eventually delete it while cleaning up what looks like accidental complexity.

The review should include one negative example. Show the input that used to fail, the trace or eval result that made the failure visible, and the current behavior after the change. This keeps the team grounded in evidence. It also stops the review from becoming abstract, which is where AI engineering discussions often drift.

For agent systems, I would check the cost impact and the quality impact together. A fix that improves quality by calling a frontier model three extra times may be correct for a regulated workflow and wrong for a low-margin support workflow. A cost reduction that removes useful context may look good for a week and then show up as worse user outcomes. The trade-off has to be explicit.

The last check is operational ownership. If the change creates a new dashboard, who looks at it? If it creates a new eval, who updates it? If it creates a new permission boundary, who approves exceptions? The difference between a strong production practice and a temporary cleanup is usually whether the maintenance path is obvious.

None of this needs to be heavy. A short pull request note, one linked trace, one eval case, and one owner are enough for many changes. The point is to leave a trail that future engineering work can build on. That is how AI systems become more reliable over time instead of slowly collecting unexplained rules.

Decision Record Template#

For a production team, I would capture the final decision in a small record with six fields: context, decision, alternatives rejected, evidence, owner, and review date. Context explains the failure or opportunity that triggered the work. Decision states the actual change in plain language. Alternatives rejected keeps the team from relitigating the same path later.

Evidence is the most important field. Link the trace, eval result, cost measurement, screenshot, support ticket, or security review that justifies the change. If there is no evidence, write that down too. It is better to be honest about a judgment call than to pretend the system proved something it did not.

Owner and review date keep the decision alive. Agent systems change quickly because models, providers, prompts, tools, and traffic all move. A decision that is correct in June can become wrong in September. The review date is not bureaucracy. It is a reminder that production AI choices age faster than normal application code.

This template is intentionally small because a large process will not survive contact with a busy engineering team. The goal is not documentation for its own sake. The goal is to leave enough context that the next person can understand why the system behaves this way and what evidence would justify changing it.

That small habit prevents future confusion.