Most AI teams I talk to have one of two problems. Either they have zero evals and are flying completely blind, or they have a solid offline eval suite and think that is enough. Both situations are dangerous. The second one is more dangerous because it comes with false confidence.
Here is the core thing to understand: offline evals test what you anticipated. Online evals catch what you did not. You need both running continuously. The moment you treat either one as optional, you are making a bet that you understand your users better than they actually behave.
Table of Contents#
- What Offline Evals Are Actually Good At
- What Offline Evals Quietly Miss
- What Online Evals Actually Catch
- What Online Evals Miss
- The Numbers That Matter
- Shadow Mode Is the Minimum Viable Safety Check
- The Deployment Gate Sequence
- Production Traffic as a Growing Eval Dataset
- Where to Start
- How I Would Run This in Production
- What I Would Measure
- Where This Connects
- FAQ
Most teams have neither. Let's fix that.
What Offline Evals Are Actually Good At#
Offline evals run against a fixed golden dataset before you ship anything. They are fast, deterministic, and cheap to run in CI. They are genuinely excellent at a specific set of things.
Prompt regressions after model upgrades are the most common thing they catch. You update from GPT-4o to a new model version, and suddenly your extraction prompt that worked 94% of the time is now hallucinating JSON keys. Your offline eval catches this in the PR before it ships. That is exactly the right place to catch it.
They are also good at controlled comparisons. If you want to know whether prompt variant A or prompt variant B performs better on your known test cases, offline evals give you an apples-to-apples answer. You control the inputs, so you are measuring the prompt, not the noise.
And they catch known bad patterns. If you've seen specific failure modes before, you can encode them as test cases. A customer service bot that previously started apologizing to users in overly formal Japanese when they asked about shipping costs, now has a test case for that. It will not regress.
What Offline Evals Quietly Miss#
The problems start when you start treating your golden dataset as a proxy for your actual user population. It is not.
Your golden dataset represents the edge cases and scenarios you thought to include when you built it. Real user behavior diverges from that almost immediately. Users phrase things in ways you did not anticipate. They combine intents in a single message. They paste in content from other applications. They write in languages you did not test. They are doing things at 2am that they do not do at 2pm.
System-level integration failures are nearly impossible to catch offline. Your prompt might score perfectly on a golden set while the tool-calling layer is silently dropping results under certain latency conditions. Offline evals do not see the full request path.
Tail cases that emerge at scale are the hardest. At 100 requests per day, you might never see the input pattern that causes your model to loop. At 10,000 requests per day, you will hit it weekly. Your eval set does not have that case, because you did not know to include it.
Distribution shift is the slow killer. User behavior evolves. The topics your users ask about six months from now will be different from what they asked about today. Your golden dataset ages. Your eval scores stay green while your actual production quality drifts.
What Online Evals Actually Catch#
Online evals instrument your production traffic. You sample requests, run judges or heuristics against the outputs, and aggregate the results as time-series metrics. This is where you see what is actually happening.
Production drift is visible here first. If your helpfulness score starts trending down over two weeks, you will see it in your online metrics before any user complaint reaches your support queue.
Abuse patterns show up. You see prompt injection attempts, jailbreak probes, and adversarial inputs that no one put in your golden set. Once you've seen them in online monitoring, they become candidates for offline regression tests.
Cost spikes from unexpected input distributions are something only online monitoring catches. If a new user segment starts sending inputs that are three times longer than your average, your token costs go up and your latency goes up. You cannot see this in offline evals.
Genuine user outcome quality, measured with production judges or thumb signals from real users, is available here and only here. A model that scores 91% on your golden set might produce outputs that real users find unhelpful 30% of the time. You will not know from offline evals.
What Online Evals Miss#
Online evals have a real weakness: attribution. When your production quality metric drops 3 points over a week, what caused it? You deployed a new prompt version on Monday. You updated a dependency on Tuesday. Your upstream API changed behavior on Wednesday. You onboarded a new customer segment on Thursday. Good luck isolating the cause without controlled experiments.
Isolated metric regressions are hard to trust. Production is a messy environment. If you do not control for confounders, a score change could be signal or noise. This is why A/B testing exists, and why proper A/B test sizing matters more than most teams realize.
Laid side by side, the two loops are each other's blind spot.
The Numbers That Matter#
Detecting a 2 percentage-point change against an 80% baseline requires roughly 6,400 samples per arm at alpha 0.05 with 80% power. That is around 12,800 total requests minimum. Underpowered A/B tests produce noise-driven decisions. If your system runs under 1,000 requests per day, you should not be making rollout decisions based on A/B tests until you've accumulated enough traffic. Just run 100% sampling and aggregate over longer windows.
Speaking of sampling rates: for general-purpose systems, 5-10% sampling for online evaluation is reasonable. For high-stakes domains like healthcare, legal, or finance, go to 20-30%. For systems under 1,000 requests per day, sample everything. The cost of sampling is low; the cost of missing something in a low-traffic system is high.
For input drift, Population Stability Index (PSI) is the right tool. PSI below 0.10 means your input distribution is stable. Between 0.10 and 0.25, investigate. Above 0.25, trigger re-evaluation or consider rollback. If the users sending requests now are fundamentally different from the users in your eval dataset, your eval scores are not meaningful. PSI makes that visible.
Shadow Mode Is the Minimum Viable Safety Check#
Before you run any canary, you should run shadow mode. Route production traffic to your candidate model in parallel with the live model. Users see the live model's output. You collect the candidate's output internally and diff them.
Gate on a hard diff rate below 1% before you advance to canary. If more than 1% of responses are substantially different from your live model, you need to understand why before you expose those responses to users. Shadow mode costs you some inference compute. The alternative is discovering behavioral changes after users see them.
The Deployment Gate Sequence#
This is what the full sequence looks like in practice:
PR review, then CI checks, then offline benchmark gate. If the benchmark regresses, the PR does not merge. That is the offline eval doing its job.
After merging, 24-hour shadow validation. Hard diff rate must be below 1%. If it passes, 1% canary for 24 hours. Then 5% for 48 hours. Then 25% for 72 hours. Then 50% for 72 hours. Then 100% with warm rollback ready.
LLM behavior is non-deterministic. You cannot fully predict production behavior from evals alone, no matter how good your golden set is. The canary stages are not optional bureaucracy. They are how you discover the behaviors that your evals did not catch.
Production Traffic as a Growing Eval Dataset#
Here is the feedback loop that makes both systems compound over time: every real edge case that online monitoring surfaces should become an offline regression test.
You see a failure mode in production. You label it. You add it to your golden set. Now your offline evals will catch regressions on that specific pattern forever. Over time, your golden set becomes a map of your actual failure surface, not just the failures you imagined before you had users.
LangSmith makes this concrete in one useful way: it shows cache-read tokens separately from total input tokens per invocation. That is the kind of observability that converts "we have caching enabled" into "we can see exactly which requests are hitting the cache and which are not." When you attach eval scores to span attributes and aggregate as time-series, you can alert on score degradation the same way you alert on p99 latency. The operational discipline for evals should look the same as the operational discipline for performance.
Tools worth knowing: Arize Phoenix is OpenTelemetry-native and works well for hybrid ML and LLM systems. LangSmith Engine automates the trace-to-dataset pipeline and is the natural choice if you are already in the LangChain ecosystem. Braintrust has the best trace-to-test pipeline if you are building outside LangChain and want to ship fast.
Where to Start#
If you have nothing right now, here is the sequence I would follow:
First, build a 200-case golden dataset from your most common and most failure-prone scenarios. Automate it to run in CI against every PR. This takes a week and will immediately start catching regressions.
Second, instrument production sampling at 10% using OpenTelemetry spans. Run a simple LLM judge against sampled outputs scoring for task completion and tone. Aggregate as daily time-series and alert when the score drops more than 5 points from a 7-day rolling average.
Third, implement shadow mode before your next model or prompt update. Do not skip this. The diff rate will tell you things that nothing else will.
Fourth, run your PSI calculation on input features weekly. If your inputs are drifting, your eval scores are drifting with them even if you do not see it.
None of this is complicated. The teams that operate reliably at scale are not doing magic. They are running two eval loops simultaneously and feeding one into the other. Start there.
How I Would Run This in Production#
I would wire the deployment sequence as a ladder, not a switch. A prompt or model change should pass offline evals, then shadow mode, then limited production traffic, then full rollout. Each stage catches a different class of failure. Skipping stages is how teams confuse a green test suite with production readiness.
Shadow mode is the most underused step. Run the new agent against real traffic without letting it act. Compare its decisions against the current system and human-reviewed outcomes. You will find distribution shift quickly because real inputs are messier than the examples your team remembered to write.
For online evals, I would sample intelligently. Do not judge every request at full depth. Judge all high-risk actions, a steady sample of normal traffic, and any request that triggers uncertainty signals such as retries, tool errors, unusually long reasoning, or policy boundary language.
The key is to feed online failures back into offline evals. If production finds a new pattern and the offline suite does not learn from it, the system stays stuck. Online evals should be a discovery mechanism for the next version of the CI gate.
What I Would Measure#
The dashboard needs live failure rate by task type, failure reason, model version, prompt version, customer segment, and traffic source. Aggregate quality scores are useful for executives but weak for debugging. The slices are where the signal lives.
I would measure disagreement between old and new systems during shadow mode. Disagreement is not automatically bad, but unexplained disagreement is risk. Review a sample before rollout and decide whether the new behavior is genuinely better or just different.
Alert fatigue is real. Alerts should fire on user-impacting quality drops, safety failures, and sharp drift, not every small judge-score movement. Online evals should make production calmer, not noisier.
Where This Connects#
This post is one piece of the production agent lifecycle. The adjacent pieces matter because the failure usually does not stay inside one layer. A tracing problem becomes an eval problem. An eval problem becomes an ownership problem. A routing problem becomes a cost and security problem if nobody can see what changed.
- Offline Evals as a CI Gate
- The Best Agent Evals Come From Production Failures, Not Design Sessions
- Your Agent Passes Every Test and Still Gets the Date Wrong
- Why Your Traces Are Lying to You
- Who Actually Owns Eval Quality
If you are using this as a checklist, read those posts as dependencies rather than as optional background. The stack only becomes reliable when the evidence loop, the release loop, and the security loop all point at the same production behavior.
FAQ#
What do online evals catch that offline evals miss?#
They catch distribution shift, model-provider changes, product changes, unexpected user behavior, long-tail tool failures, and quality drift that only appears under real traffic conditions.
Should online evals run on every request?#
Not always. Run full evaluation on high-risk actions and sample lower-risk traffic. Add targeted evaluation for uncertainty signals such as retries, fallback paths, tool errors, and unusually long reasoning traces.
What is shadow mode for AI agents?#
Shadow mode runs a new prompt, model, or agent policy against real inputs without letting it take action. You compare its outputs with the current production system before exposing users to the change.
How do online evals improve offline datasets?#
Every confirmed online failure should become either a new offline eval example, a new rubric clause, or a documented non-goal. That turns production surprises into future regression coverage.
Can online judges create privacy risk?#
Yes. Send only the minimum necessary data to the judge, redact sensitive fields, and avoid sending regulated content to providers that are not approved for that data. Treat judges as part of the production data path.
What is the biggest mistake with online evals?#
The biggest mistake is treating the score as the product truth. Online evals are signals. They need calibration, sampling discipline, human review for severe failures, and clear action paths.
Implementation Review Checklist#
Before I would call this production-ready, I would ask five questions in the review. First, does the team know which artifact changed because of this lesson? A trace field, eval case, prompt file, routing rule, dashboard, or security policy should be visible in the repo or the runbook. If the only output is agreement, the lesson has not landed yet.
Second, can the team reproduce the failure mode in a lower environment? Production AI work gets messy when the only evidence is a screenshot, a Slack thread, or a vague user complaint. A reproducible example gives the team something concrete to test against after the fix.
Third, does the fix have an owner after the first merge? Most agent systems decay slowly. Datasets go stale, prompts drift, routing rules stop matching traffic, and permissions expand. The owner is the person who notices that drift before it becomes an incident.
Fourth, is rollback obvious? If a prompt, model, parser, evaluator, or security rule makes behavior worse, the team should know how to return to the last known-good version without rewriting the system under pressure.
Fifth, did the incident or improvement strengthen the loop? A good production process compounds. The trace becomes an eval. The eval becomes a gate. The gate becomes a safer release. That is the practical standard I would use for every idea in this series.
I would also ask whether the change makes the system easier to explain to a new teammate. Production AI stacks fail when the important behavior lives in someone s memory. If the reason for a prompt rule, routing threshold, eval example, or tool permission is not written down, the next person will eventually delete it while cleaning up what looks like accidental complexity.
The review should include one negative example. Show the input that used to fail, the trace or eval result that made the failure visible, and the current behavior after the change. This keeps the team grounded in evidence. It also stops the review from becoming abstract, which is where AI engineering discussions often drift.
For agent systems, I would check the cost impact and the quality impact together. A fix that improves quality by calling a frontier model three extra times may be correct for a regulated workflow and wrong for a low-margin support workflow. A cost reduction that removes useful context may look good for a week and then show up as worse user outcomes. The trade-off has to be explicit.
The last check is operational ownership. If the change creates a new dashboard, who looks at it? If it creates a new eval, who updates it? If it creates a new permission boundary, who approves exceptions? The difference between a strong production practice and a temporary cleanup is usually whether the maintenance path is obvious.
None of this needs to be heavy. A short pull request note, one linked trace, one eval case, and one owner are enough for many changes. The point is to leave a trail that future engineering work can build on. That is how AI systems become more reliable over time instead of slowly collecting unexplained rules.