A tax software founder told a story recently that I have not been able to put down. His product does AI data entry for tax firms. They pushed accuracy past 80% against an industry baseline of 50 to 60%, which is a genuinely large jump, and a lot of their customers still complained. Same model, same stack, different reactions across users. So they dug in.
The unhappy customers were not getting worse AI. They were reverse-engineering everything the product produced. The team had not removed work from the customer's day. They had changed its shape. And that reshaping is the thing almost nobody building AI products is measuring, because everyone is staring at accuracy instead.
The verification burden is the work a person must do to confirm an AI system's output is correct before they rely on it. It matters because that cost falls on the user, not the model, and it does not shrink when accuracy rises. It is distinct from accuracy, which measures how often the system is right, and from trust calibration, which measures whether the user checks at the right moments. Accuracy is measured on the model. The verification burden is paid by the human.
That is the whole argument, and here is the short version if you read nothing else. Accuracy is a lagging indicator. It is the score you get after the game is already over, and it tells you almost nothing about whether your product actually helped the person using it. The number that decides that is how expensive it is for the accountable human to check the output before they rely on it, which I will call cost-to-verify. Raising accuracy from 80% to 85% removes a fifth of the errors but none of the checking, because the user still cannot tell in advance which outputs are the wrong ones, so they inspect all of them. Most teams have never measured cost-to-verify, do not have a dashboard for it, and are surprised when better accuracy scores do not translate into happier customers. This post is about why that happens and what to measure instead.
Table of Contents#
- What is the verification burden?
- Why does higher accuracy not reduce complaints?
- What did the developer study actually measure?
- Why can users not see the errors they are accountable for?
- Do citations remove the verification burden or hide it?
- When do explanations actually help?
- What should you measure instead of accuracy?
- FAQ
What is the verification burden?#
Think about what an AI feature actually promises. It promises to take a task off your plate. You delegate the work, the system does it, you get the result back. The pitch is that the agent does the job while you do something else.
But in any setting where the human is still accountable for the outcome, the delegation is incomplete. A tax preparer signs the return. A doctor owns the diagnosis. An engineer merges the code. The AI produced a draft, and the accountable human still has to decide whether that draft is right. That deciding is the verification burden, and it is real work, not a rounding error on top of the real work.
Here is the trap. A chat interface and a citation trail both feel like the fix for this. They look like transparency. They look like you are handing the user everything they need to trust the output. In practice they do the opposite. They hand the verification burden back to the user with extra steps. Chat is synchronous, so the user cannot leave and go do their job while it runs. Citations mean the user now has to open each source and confirm it says what the model claims it says. You have not removed the checking. You have decorated it.
Why does higher accuracy not reduce complaints?#
Because complaints are not really about accuracy. They are about who holds the bag when something is wrong.
When the user remains accountable for every output, every error is catastrophic, not statistically. It does not matter that the system is right 80% of the time if the user has to inspect 100% of the outputs to find the 20% that are wrong. The accuracy went up and the amount of checking did not go down, because you cannot know in advance which cases are the wrong ones. So the user audits everything, the same way they would audit a new junior hire, and the product that promised to save them time has instead handed them a review queue.
This is why pushing accuracy from 80% to 85% often does nothing for satisfaction. You removed a quarter of the errors and none of the verification. The reviewing was the expensive part, and the reviewing is untouched. I have made a version of this argument before in the context of why the observability gap breaks production agents, and it rhymes here: the thing that hurts is not the model's failure rate, it is that the human cannot see where the failures are without redoing the work.
What did the developer study actually measure?#
The best hard evidence for this does not come from tax software. It comes from a randomized controlled trial of experienced software developers, and it is worth being precise about what it found and what it did not.
In July 2025, METR published a randomized controlled trial of 16 experienced open source maintainers working on 246 real tasks, in mature repositories they had spent an average of about five years inside. Before starting, the developers expected AI to speed them up by around 24%. After the study, they estimated it had sped them up by about 20%. According to METR, allowing AI actually increased their completion time by 19%. They were slower, and they did not notice they were slower. The screen recordings show where the time went: active coding fell from roughly 37% of the session to around 24%, and three new activities appeared in its place, reviewing the AI output, prompting the AI, and waiting on it. The work did not disappear. It moved into verification and orchestration.
Two honest qualifiers, because this number gets misquoted constantly. First, this was early-2025 tooling, and METR's own February 2026 follow-up with a larger sample found the slowdown had shrunk to roughly 4% with a confidence interval spanning zero, and they believe developers are likely being sped up now. So the 19% is not a timeless fact about AI coding. It is a snapshot. Second, this is software, and expert maintainers on codebases they know intimately are close to the worst case for AI assistance by construction. The durable finding is not the headline slowdown. It is the shape of the reallocation. Reviewing, prompting, and waiting are real cost centers, and they are exactly the costs a naive accuracy metric cannot see.
Why can users not see the errors they are accountable for?#
This is the part that should worry anyone shipping a professional AI tool. The user cannot reliably catch the errors, and their confidence goes up anyway.
A CHI 2025 experiment from Microsoft Research had people make decisions with and without an LLM search tool. The LLM roughly halved task time. On the four straightforward tasks, accuracy was statistically indistinguishable between the two groups. But on the one task engineered to trip the model up, decision accuracy collapsed from 93% in the traditional group to 47% in the LLM group. People took the wrong answer and ran with it. The mechanism is the alarming bit: of the participants who got a wrong answer from the model, essentially none issued a follow-up query to double-check. They asked once, believed it, and moved on. Now the punchline. Those same users rated their overall experience higher, 4.41 against 3.10, and rated the tool's reliability no differently at all, with a p-value of 0.91, despite having been actively misled. This is one engineered task in a low-stakes lab setting, so treat it as a mechanism rather than a field failure rate. But the mechanism is the point. Satisfaction and reliability ratings did not track correctness.
Put that next to the METR result and a pattern falls out. In one study, professionals were slowed down and believed they had been sped up. In the other, users were misled and rated the tool just as reliable. Customer satisfaction and self-reported productivity are not proxies for accuracy. They are not even weakly correlated with it in these studies. If you are steering your product by CSAT or by how much users say they love the AI, you are steering by an instrument that does not point at whether the thing works.
There is a deeper reason this does not train away. The human factors literature has a name for it, automation bias, and the canonical review by Parasuraman and Manzey is blunt about the scope. Automation bias produces both errors of omission and errors of commission, it occurs in expert participants as well as novices, it shows up in teams as well as individuals, and it cannot be prevented by training or instructions. Read that last clause again, because it quietly guts the standard mitigation stack. "Train the professional, tell them to verify, add a second reviewer" is the exact plan that both compliance frameworks and vendor human-in-the-loop marketing lean on, and the research says the first two do not eliminate the problem. A 2024 preregistered meta-analysis in Nature Human Behaviour covering 106 experiments found that human-AI combinations performed significantly worse on average than the better of human alone or AI alone, with the losses concentrated in decision-making tasks. The imperfect aid does not just leave the residual errors. It induces a new class of error that the unaided human would not have made.
Do citations remove the verification burden or hide it?#
Citations are the industry's favorite answer to this problem, and the evidence on them is uncomfortable. They do not discharge the verification duty. They make it look discharged.
Start with behavior. A 2025 study on citations and trust found that of the answers that carried at least one reference, only 9.77% were actually checked, and only 42.1% of participants checked even a single citation. So most citations are never opened. Now layer on quality. A 2026 audit of production search-augmented LLMs across five providers, which its authors call verified misguidance, found that 30.6% of citations distort the source they point at. The link works, the source is real, and the claim sitting on top of it is still wrong. This is not a new or shrinking problem. Back in 2023, an evaluation of generative search engines found only about half of generated sentences were fully supported by their own citations. Three years and several model generations apart, the same order of magnitude.
Combine those two facts and the citation model looks worse than useless in the accountable case. Most users do not check the citation. Of the ones who do, a meaningful fraction are checking a source that misrepresents the claim. A working link is not evidence that the claim above it is true. It is a trust signal that the user reads as verification without performing verification. And here is the sharper finding, from a pre-registered CHI 2025 study on reliance: the presence of an explanation increases reliance on both correct and incorrect answers. When the model was wrong, participants who saw an explanation were more likely to go along with the wrong answer, not less. The explanation surface raises trust whether or not the explanation is any good.
When do explanations actually help?#
Here is where I want to be careful, because "citations are theater, rip them out" is the lazy conclusion and it is wrong. The research does not support it. It supports something more precise and more useful.
Explanations help when, and only when, they make checking cheaper than doing. That is the verifiability hypothesis from Fok and Weld, published in AI Magazine in 2024, and it reframes the whole problem in a way that gives you something to build toward. Their analogy is NP-complete problems: some things are hard to produce but cheap to verify, the way a completed Sudoku is tedious to solve but instant to check. If a task has that shape, an explanation that surfaces the certificate lets the human verify fast, and the human plus AI genuinely beats either alone. Their review of around 24 prior studies found only about five that achieved this complementary performance, and those five were precisely the ones where the explanation lowered the cost of verification. The problem, in their words, is that most tasks fundamentally do not allow easy verification, regardless of the explanation method.
An empirical study backs this up. Stanford researchers ran five studies with 731 participants and showed that people make a cost-benefit choice about whether to engage with an AI explanation at all. When they made verifying cheaper, overreliance dropped below the no-explanation baseline. When they made it more expensive, overreliance went up. So the explanation is not the lever. The cost of verification is the lever, and the explanation only matters insofar as it moves that cost. A citation that requires opening a PDF and reading three pages has raised the cost of verifying, not lowered it. A confidence highlight that points directly at the one number the model is unsure about has lowered it.
What should you measure instead of accuracy?#
Cost-to-verify. Make it a first-class product metric and instrument it the way you currently instrument accuracy.
Concretely, that means measuring things like the median time from output to accept, the percentage of outputs a user accepts without editing, and how much of the output a user actually inspects before signing off. Those numbers tell you whether you are removing work or reshaping it. A product can improve its accuracy every quarter while its cost-to-verify stays flat, and that product will keep collecting complaints, because the thing the user pays for has not changed. This is the same instinct behind eval-driven development for agents, just pointed at the human's workload rather than the model's outputs. You cannot improve what you refuse to measure.
Now the part that is genuinely contested, which is where I will plant a flag. There is evidence that you can lower cost-to-verify, and that doing it may make your users slightly less happy in the short term. In the same Microsoft Research experiment, a second condition highlighted the model's per-token confidence, showing low-confidence spans in red. On the hard task, accuracy more than doubled, from 26% to 58%, because users saw the flagged spans and issued follow-up queries exactly when the answer was shaky. It worked. And the same intervention slightly lowered the users' ratings of the tool's reliability and their overall experience. Making users appropriately skeptical made them trust the tool a little less. You can fix the verification problem, or you can win the satisfaction survey, and in the short run you may not get both. That is an uncomfortable trade, and I have not seen a single vendor say it out loud, which is exactly why it is worth saying.
The following table sorts the three levers most teams reach for by what they actually do to the verification burden.
| Lever | What it looks like | Effect on cost-to-verify | Effect on measured satisfaction |
|---|---|---|---|
| Raise model accuracy | Better model, more training, RAG | Little to none: user still checks everything | Flat, because the checking did not change |
| Add citations and chat | Source links, conversational UI | Often raises it: user opens sources, waits on chat | Rises, because the surface looks trustworthy |
| Surface targeted uncertainty | Confidence highlights, plan-then-approve gates | Lowers it: user checks the flagged spans only | May dip, because the tool looks less confident |
The uncomfortable reading of that table is that the two levers which raise satisfaction do nothing for the real burden, and the lever that lowers the real burden costs you a little satisfaction. Optimizing the survey and optimizing the user's actual workload point in different directions, and most roadmaps are quietly optimizing the survey.
There is a design direction that follows from all of this, and it connects to how the best agent products are already being built. Instead of dumping a finished output and asking the human to audit it, decompose the task into independently checkable atoms, diff the new output against a known-good prior, and flag only the spans where the model is unsure. Let the human verify the 5% that is risky instead of re-reading the 100%. That is the same principle behind treating each recurring edge case as a reusable skill the system learns rather than a model failure to paper over. The goal is not a more accurate dump. It is a smaller thing to check.
None of this means accuracy does not matter. It means accuracy is necessary and radically insufficient, and that if the user still holds the bag, you have to design for the holding, not just the producing. The founder who told the tax story reached that conclusion from complaints. The research reached it from controlled experiments. They agree, and the agreement is the signal.
FAQ#
What is the verification burden in AI products?#
It is the work a human must do to confirm an AI output is correct before relying on it. In any workflow where the person remains accountable for the result, delegation to the AI is incomplete, because the human still has to check. That checking is real work, it falls on the user rather than the model, and unlike accuracy it does not automatically shrink as the model improves.
Why does improving accuracy not reduce user complaints?#
Because the user cannot know in advance which outputs are the wrong ones, so they inspect all of them regardless of the model's hit rate. Moving accuracy from 80% to 85% removes a fifth of the errors but none of the inspection. The expensive part, reviewing every output, is untouched. Complaints track the verification burden, and accuracy gains alone do not lower it.
Do citations make AI outputs more trustworthy?#
They make outputs feel more trustworthy without reliably making them more correct. Studies show most citations are never opened, only about 9.77% of cited answers get checked, and around 30% of citations in production systems distort the source they point at. Worse, the mere presence of an explanation raises user reliance on wrong answers as well as right ones. A working link is a trust signal, not proof the claim is accurate.
What metric should replace accuracy for agentic products?#
Cost-to-verify. Instrument the median time from output to acceptance, the share of outputs accepted without edits, and how much of each output the user inspects before signing off. These reveal whether you are removing work or reshaping it. Accuracy measures the model. Cost-to-verify measures the human's remaining workload, which is what the customer is actually paying you to reduce.
Can you design AI products so verification is cheaper?#
Yes, and the evidence is fairly direct. Decompose a task into independently checkable pieces, diff new output against a trusted prior, and flag only the low-confidence spans so the human checks the risky 5% rather than re-reading everything. In one controlled study, surfacing per-token confidence more than doubled accuracy on a hard task by prompting users to double-check exactly where the model was unsure.
Is it true that making verification easier can lower satisfaction?#
In the short term it can. The same study that doubled accuracy by highlighting model uncertainty also found users rated the tool slightly less reliable and less enjoyable. Showing users where the AI is unsure makes them appropriately skeptical, which reads as the tool being less confident. You may have to choose, at least initially, between fixing the real problem and winning the satisfaction survey.
Does human-in-the-loop review solve the verification problem?#
Not on its own. The automation bias literature finds that the classic mitigations, training the reviewer, instructing them to verify, and adding a second reviewer, do not eliminate the problem. Automation bias appears in experts and in teams, and it survives explicit instructions to check. Human oversight helps most when the product actively lowers the cost of verifying, not when it simply asks a human to look.