I used to think eval quality was an engineering problem. Build the harness, pick a judge model, write some prompts, ship it. The team that built the pipeline should own the pipeline. That felt obvious.
Then I read about what happened at Uber.
Table of Contents#
- The Question You are Not Asking About Your Eval Suite
- What BlackRock Got Right
- Harvey and Rogo Are Doing Something Different
- Amazon's Model: Ownership by Domain, Not by Platform
- On Rubrics: The Prometheus Finding
- The Human Loop That Cannot Be Automated
- The Adoption Gap Is Not a Capability Gap
- What to Actually Do About This
- How I Would Run This in Production
- What I Would Measure
- Where This Connects
- FAQ
Their conversational AI team had a session that looked fine on the surface. Users were getting answers. Nothing was crashing. But one metric crept up without anyone catching it for weeks: turns per session climbed from 4 to 20. That number, quietly rising, meant users were stuck in broken loops, re-asking the same questions, clarifying, getting confused, asking again. The model wasn't failing loudly. It was failing slowly, in a way that only made sense to someone who understood what a good conversation actually feels like.
The engineers had built a solid monitoring stack. They caught the signal eventually. But they had also been the ones maintaining the eval datasets, and those datasets had gone stale. Nobody had updated them in months. The golden examples reflected what the team thought "correct" looked like when they first built the system, not what users were actually running into in production.
After what they internally called the "pizza incident" (a specific broken interaction loop that sent turns-per-session through the roof), Uber moved eval ownership from engineers to conversational designers. People who had spent years listening to how humans actually talk. People who had strong opinions about what "resolved" versus "abandoned" sounded like. Now those datasets get updated weekly.
That surprised me. And the more I looked into it, the more I saw the same pattern everywhere.
The Question You are Not Asking About Your Eval Suite#
Uber developed four questions for evaluating whether an eval setup is actually trustworthy:
- Do you trust your eval results?
- If your eval flagged a regression, would that actually stop a deployment?
- When was your dataset last updated?
- How quickly can you test a newly released model?
Most teams I have talked to pass questions 1 and 4. They have some results, and they can swap in a new model and run it. Questions 2 and 3 are where it falls apart.
Question 3 is the one that stings. If your dataset was last updated six months ago, it is not a signal anymore. It is noise with false confidence attached. A stale dataset is worse than no dataset because it tells you something is fine when you have no idea whether it is.
The reason datasets go stale is not laziness. Engineers are busy with infrastructure, with latency, with the actual mechanics of running inference at scale. They are not the right people to keep asking "does this output actually reflect what a correct response sounds like in this domain?" That question requires domain knowledge, not platform knowledge.
What BlackRock Got Right#
BlackRock did something that sounds simple but runs against how most teams are built: they brought legal and compliance in at the PRD stage. Not as a final sign-off. Not as a gate at the end. As co-authors of edge cases.
The result was stark. Their lawyers generated roughly 10 edge cases for every 2 an engineer came up with. These were not synthetic gotchas. They were real scenarios drawn from years of watching what goes wrong in financial communication. The kinds of situations that an engineer, no matter how careful, would not think to construct because they had never seen them blow up in production.
Compliance approval time dropped from 8 weeks to 2 weeks. The mechanism is not complicated: when the people who understand domain failure modes are involved early, you stop discovering those failure modes during deployment.
Harvey and Rogo Are Doing Something Different#
The legal AI company Harvey and the financial AI company Rogo both hired ex-lawyers and ex-Goldman Sachs bankers. Not as consultants. Not as domain advisors. As core engineering functions.
These are people who understand what a correct output looks like because they spent years producing those outputs themselves. They can write rubrics that actually reflect what "accurate" means in their domain. They can spot when a model's answer is technically correct but practically wrong in a way that would never show up in a generic eval.
This is an expensive choice. It is also the only choice that scales. You cannot borrow domain expertise from a consultant for eval work. Eval is ongoing, weekly, tied to every model update and every new production pattern. You need people who understand the domain and are close enough to the system to update the evals when production drifts.
Amazon's Model: Ownership by Domain, Not by Platform#
Amazon's approach to eval ownership is horizontal. The customer service team owns intent eval. The shopping team owns tool-use eval. The multi-agent team owns coordination eval.
Each team owns the evals relevant to their domain because each team has the accumulated judgment to know what correct looks like in that context. The infrastructure for running evals is shared. The knowledge of what to put in those evals is not.
This is the insight that most teams miss. They centralize eval infrastructure and accidentally centralize eval content. The harness can be centralized. The golden datasets cannot be.
On Rubrics: The Prometheus Finding#
There is a 2024 paper from ICLR, the Prometheus project, with a finding that I keep coming back to. A fine-tuned 13B model with well-constructed custom rubrics achieved a Pearson correlation of 0.897 with human judgment. GPT-4 with generic prompting scored 0.882.
A smaller model with better rubrics beat a larger model with vague ones.
The implications for eval quality are not subtle. Judge capability matters less than rubric quality. And rubric quality is entirely determined by whether the person writing the rubric understands what failure looks like in the domain.
A good rubric has four properties. It is explicit: no ambiguity about what satisfies each criterion. It is structured: criteria are hierarchical, ordered by importance. It is decomposable: you can score each criterion independently without needing to read the whole output again. It is verifiable: every criterion can be checked from the output itself, without inferring intent.
Writing rubrics like that requires knowing the domain. You cannot write an explicit criterion for a failure mode you have never seen.
The Human Loop That Cannot Be Automated#
The teams running production AI at any real scale have a manual sampling practice that is not optional. Every week, someone sits down and reads real production traces. They label them by hand. They note where the model was accurate, where it was close but wrong in a way that matters, where it was confidently incorrect.
Those labeled traces feed back into the formal eval suite. The suite is not static. It is a living record of what production actually looks like.
This is not automatable. Not because the tooling does not exist, but because the judgment required to know what matters in a trace is the judgment that makes the whole system work. Automating it means substituting a model's opinion for a human's opinion, which is circular if you are trying to evaluate model quality.
The teams that do this well treat it like a sprint ceremony. Fixed time. Consistent participants. Clear handoff to whoever manages the eval suite.
The Adoption Gap Is Not a Capability Gap#
72% of organizations have deployed AI agents. Only 11% are running them at production scale. Only 6% say they fully trust their agents for core processes.
The primary barrier is not capability. The models are good enough. The infrastructure is mature enough. The barrier is rigorous ongoing evaluation frameworks.
That number, 6%, reflects what happens when teams treat eval as an engineering artifact rather than a domain knowledge artifact. You can get something deployed. Getting it trusted, and keeping it trusted as the system and the world change, requires people who understand what "correct" looks like well enough to keep telling the system so.
What to Actually Do About This#
If you are running AI systems in production right now, four things follow directly from everything above.
Audit your dataset last-updated date. If it is more than eight weeks old, you do not have a meaningful eval. You have a snapshot of assumptions from months ago. Schedule a dataset refresh before you schedule anything else.
Pull domain experts into rubric authoring, not just rubric review. Showing a lawyer a finished rubric and asking for sign-off is not the same as sitting with a lawyer and asking them to describe the last ten times a financial summary led to a bad outcome. The second conversation produces criteria you would never have written yourself.
Separate eval infrastructure ownership from eval content ownership. Engineers should own the harness, the runner, the scoring pipeline, the CI integration. Domain experts should own what goes in the golden datasets and how the rubrics are written. These are different jobs and confusing them is why datasets go stale.
Build a weekly trace-labeling practice with someone who understands the domain. One hour, fixed time, real production samples, hand-labeled. The labeled traces become eval cases. The practice keeps your eval suite connected to what production actually looks like rather than what you imagined it would look like when you shipped.
Split out by role, ownership looks like this:
The bottleneck in eval quality is almost never the platform. It is almost always the domain knowledge needed to know what you are measuring.
How I Would Run This in Production#
I would split ownership into three pieces. Engineering owns the harness, reliability, versioning, and CI integration. Domain experts own the expected behavior and the rubric. Product owns the decision about which failures matter enough to block release.
That split sounds bureaucratic until you see the alternative. When engineers own everything, the dataset drifts toward things engineers can easily test. It becomes heavy on schemas, short on judgment, and weak on domain nuance. The agent passes the suite and still gives an answer no domain expert would accept.
The operating rhythm matters. I would run a weekly eval review with one engineer, one domain expert, and one product owner. Review new production failures, retire obsolete cases, inspect judge disagreements, and update rubric language. Keep it short. The point is continuity, not ceremony.
For regulated workflows, compliance should be involved before release, not after. The best edge cases often come from people who know where the organization has been burned before. Waiting until sign-off turns compliance into a blocker. Involving them in dataset design turns them into a source of coverage.
What I Would Measure#
The first metric is dataset freshness: how many examples were added, edited, or retired in the last thirty days. A stale dataset is usually a stale product assumption. The second is reviewer coverage: how many examples have domain-owner approval rather than engineer-only approval.
I would also track disagreement between engineers and domain reviewers. That disagreement is valuable. It shows where the product definition is fuzzy. If the team cannot agree on what correct means, the model will not magically infer it.
Finally, measure adoption. A beautiful eval process that engineers ignore is theater. The harness should show failures in the tools developers already use: pull requests, deploy dashboards, and incident reviews.
Where This Connects#
This post is one piece of the production agent lifecycle. The adjacent pieces matter because the failure usually does not stay inside one layer. A tracing problem becomes an eval problem. An eval problem becomes an ownership problem. A routing problem becomes a cost and security problem if nobody can see what changed.
- Offline Evals as a CI Gate
- Why You Need Offline and Online Evals
- The Best Agent Evals Come From Production Failures, Not Design Sessions
- Treating Prompts Like Code
If you are using this as a checklist, read those posts as dependencies rather than as optional background. The stack only becomes reliable when the evidence loop, the release loop, and the security loop all point at the same production behavior.
FAQ#
Should engineers own eval datasets?#
Engineers should own the infrastructure, but domain experts should own the meaning of correctness. The dataset needs both technical reliability and domain judgment.
Who writes the rubric for an LLM judge?#
A domain expert should define the quality criteria with engineering help. Engineers can translate criteria into prompts and structured outputs, but they should not invent domain standards alone.
How often should eval datasets be reviewed?#
Weekly is a practical default for active systems. Low-volume systems can review less often, but every production incident or product policy change should trigger a dataset review.
What should product managers own?#
Product managers should own the release threshold and trade-offs. They decide which failures block launch, which are warnings, and which are acceptable limitations for the current product stage.
How do compliance teams fit into eval ownership?#
Compliance should contribute edge cases, policy constraints, and unacceptable outcomes before release. That turns compliance knowledge into test coverage instead of late-stage approval friction.
What is a sign that ownership is broken?#
If nobody can explain why an example is in the dataset, or if every example was written by the same engineer months ago, ownership has probably drifted away from the people who understand the task.
Implementation Review Checklist#
Before I would call this production-ready, I would ask five questions in the review. First, does the team know which artifact changed because of this lesson? A trace field, eval case, prompt file, routing rule, dashboard, or security policy should be visible in the repo or the runbook. If the only output is agreement, the lesson has not landed yet.
Second, can the team reproduce the failure mode in a lower environment? Production AI work gets messy when the only evidence is a screenshot, a Slack thread, or a vague user complaint. A reproducible example gives the team something concrete to test against after the fix.
Third, does the fix have an owner after the first merge? Most agent systems decay slowly. Datasets go stale, prompts drift, routing rules stop matching traffic, and permissions expand. The owner is the person who notices that drift before it becomes an incident.
Fourth, is rollback obvious? If a prompt, model, parser, evaluator, or security rule makes behavior worse, the team should know how to return to the last known-good version without rewriting the system under pressure.
Fifth, did the incident or improvement strengthen the loop? A good production process compounds. The trace becomes an eval. The eval becomes a gate. The gate becomes a safer release. That is the practical standard I would use for every idea in this series.
I would also ask whether the change makes the system easier to explain to a new teammate. Production AI stacks fail when the important behavior lives in someone s memory. If the reason for a prompt rule, routing threshold, eval example, or tool permission is not written down, the next person will eventually delete it while cleaning up what looks like accidental complexity.
The review should include one negative example. Show the input that used to fail, the trace or eval result that made the failure visible, and the current behavior after the change. This keeps the team grounded in evidence. It also stops the review from becoming abstract, which is where AI engineering discussions often drift.
For agent systems, I would check the cost impact and the quality impact together. A fix that improves quality by calling a frontier model three extra times may be correct for a regulated workflow and wrong for a low-margin support workflow. A cost reduction that removes useful context may look good for a week and then show up as worse user outcomes. The trade-off has to be explicit.
The last check is operational ownership. If the change creates a new dashboard, who looks at it? If it creates a new eval, who updates it? If it creates a new permission boundary, who approves exceptions? The difference between a strong production practice and a temporary cleanup is usually whether the maintenance path is obvious.
None of this needs to be heavy. A short pull request note, one linked trace, one eval case, and one owner are enough for many changes. The point is to leave a trail that future engineering work can build on. That is how AI systems become more reliable over time instead of slowly collecting unexplained rules.