Five agents appeared on the screen.
One evaluated cost. One looked for technical risk. Another represented the user. A fourth challenged the consensus. A fifth merged the conclusions into a final verdict.
Their disagreement looked intelligent. The answers they disagreed about looked different from what any one of them had said alone.
The final answer looked better than anything a single model would have produced.
I almost accepted the architecture immediately.
I have been building production agents long enough to know that a convincing demo can hide expensive assumptions. So before I built anything, I asked a different question. Not "does the swarm work?" but something simpler and harder.
What is the cheapest thing this swarm has to beat?
That question changes the whole picture.
An agent swarm is a group of language model agents that analyze the same decision from different assigned roles, then reconcile their conclusions. A well-built swarm can catch errors a single model misses on genuinely hard, contested decisions. It differs from a single agentic loop in that the value is supposed to come from structured disagreement between independent agents, not from one model reasoning longer. That "supposed to" is the whole question.
Table of Contents#
- The demo that convinced me
- The question nobody measured
- The boring baseline that changed the result
- When disagreement becomes real
- Why agents can spread mistakes
- The swarm I would actually build
- FAQ
The demo that convinced me#
There is a genre of build post going around right now that goes like this. Stop asking one model to evaluate a decision, because it gives you one averaged, cautious view. Instead, spin up a swarm of agents with conflicting roles. One that thinks only about money, one about technical risk, one about the user. Let them argue, add a devil's advocate to attack the consensus, and merge it all into a verdict better than any single agent could produce. The posts come with clean code and a lot of confidence.
The pattern is real and the intuition is mostly right. When a decision is genuinely contested, structured disagreement between independent agents can surface errors that a single model misses. That is not nothing.
The founding academic version of this is not the product-launch example from the build posts. It is Du, Li, Torralba, Tenenbaum and Mordatch, "Improving Factuality and Reasoning in Language Models through Multiagent Debate," published in 2023 and later at ICML 2024. Multiple model instances propose answers, read each other's responses, and revise over several rounds. The Du et al. results show GSM8K math accuracy rising to 85.0% from 77.0% for a single agent, and MMLU on a 100-question slice reaching 71.1% versus 63.9%. Arithmetic jumped to 81.8% from 67.0%.
Those are real numbers. I do not want to wave them away. Debate does something.
The question the build posts almost never ask is: better than what?
The question nobody measured#
While building one of these architectures, I kept coming back to something that bothered me.
The swarm was calling the model five times. The orchestrator was calling it twice more for assignment and merge. The devil's advocate added another round. At the end of all of that, the answer was good. But I had no idea whether it was good because the agents were collaborating, or simply because I had called the model more times.
Those are two very different things. If it was the latter, there was a much cheaper way to get the same result.
So I removed the roles.
No financial expert. No user advocate. No devil's advocate. No structured debate.
I kept the same model and matched the compute budget. I asked the same question several times independently and took the majority answer.
This pattern has a name. It is called self-consistency. You sample the same model multiple times independently, then take the majority answer. No roles, no debate, no orchestrator, no merge prompt. Just the same model, a few times, with a vote at the end.
That is the control group your swarm has to beat. And the expensive pipeline now had a very boring competitor.
The boring baseline that changed the result#
When I ran self-consistency against the swarm at matched compute, the swarm did not win. Not clearly. Not reliably.
Then I found that researchers had already run almost exactly this comparison, at scale.
Huang and colleagues at Google DeepMind and the University of Illinois published "Large Language Models Cannot Self-Correct Reasoning Yet" in 2023, presented at ICLR 2024. Section 4 of that paper is titled, in plain words, "Multi-Agent Debate does not Outperform Self-Consistency."
Their critique of the Du et al. debate results is surgical. The original debate comparison was three agents over multiple rounds against a single model response. That is not a fair comparison, because the debate setup was spending several times the compute. When Huang et al. gave self-consistency the same budget, sampling the model as many times as the debate used agent-turns and then voting, majority voting matched or beat the debate. The gain was not coming from agents collaborating. It was coming from taking more samples.
This is not one contrarian result. Smit and colleagues, in "Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs" at ICML 2024, benchmarked debate against self-consistency and ensembling across accuracy, cost, and time. Their finding, quoted directly, is that "multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling."
The uncomfortable summary is that a large fraction of the reported multi-agent magic is ensembling wearing a costume.
If you have not compared your swarm against self-consistency at matched compute, you do not actually know that the swarm is doing anything. You know it is doing something expensive. This is the trap I keep seeing teams walk into when they build multi-agent systems by default: the architecture looks sophisticated, the demo is convincing, and nobody ran the boring one-model control that would have told them whether any of it was necessary.
| Baseline | Relative cost | What it gives up | When to reach past it |
|---|---|---|---|
| Single model, one call | 1× | Multiple samples | When a rough answer is enough |
| Self-consistency (N samples, majority vote) | N× | Orchestration overhead | Most decisions. This is the baseline. |
| Homogeneous swarm (same model, N roles) | 5–20× | Usually loses to self-consistency at equal compute | Rarely justified on current evidence |
| Cross-model swarm (different models, blind) | 5–20× | Cost | High-stakes, genuinely contested decisions |
When disagreement becomes real#
The story does not end with self-consistency winning. And I would not write this if agent swarms had no use at all.
The durable gains in the research appear under a specific condition the default build post ignores: genuine heterogeneity.
The single most striking result in the entire Du et al. paper was not same-model debate. It was cross-model. When they had ChatGPT and Bard debate on a set of grade-school math problems, the pair solved 17 of 20. Each model alone solved between 11 and 14. Two genuinely different models caught each other's mistakes in a way that neither could catch alone. That is a heterogeneity effect, and it is a different mechanism from asking one model to wear five hats.
The counter-evidence on cosmetic personas is blunt. A 2026 paper titled "The Cost of Consensus" put homogeneous debate swarms under real cognitive load, on harder benchmark variants like GSM-Hard and MMLU-Hard. The swarms degraded into what the authors call "systematic groupthink and contextual fragility," and isolated self-correction beat unguided homogeneous debate. When a build post shows you five roles that are all the same base model with different system prompts, the research says that arrangement collapses precisely when the decision gets hard. Which is the only time you needed the swarm.
The problem was never multiple agents. The problem was multiple copies of the same judgment pretending to be independent.
Cost is also a first-class design input here, not an afterthought. A pipeline with five experts, a debate round, a devil's advocate, and a merge is easily ten to twenty times the tokens of a single self-consistency pass that might match it. That multiplier makes sense for a decision where being wrong is catastrophic and rare, like a large irreversible architectural choice. It does not make sense for a decision you make a hundred times a day. This is the same discipline I argued for in why you should count the cost before running ten agents: the number of agents is a budget line, not a free knob.
Why agents can spread mistakes#
The build posts get one part right, and here the evidence backs the intuition rather than undercutting it. Experts must analyze independently, because the moment one agent sees another's conclusion, it starts to conform.
This is not a stylistic preference. It is a measured failure mode.
Language models tuned with human feedback are sycophantic. They adjust their answers toward what they think the other party believes. Anthropic's work on sycophancy established the single-model version, where a model will abandon a correct answer when a user pushes back. In a multi-agent setting this becomes inter-agent sycophancy, and it is worse, because the pressure is now coming from a peer that sounds just as confident.
Wu and colleagues at McGill and Mila, in "Can LLM Agents Really Debate?" from late 2025, are the clearest source for this mechanism. They found that "when agents share similar training or biases, debates can reinforce incorrect beliefs," and that "an eloquent but incorrect agent can sometimes sway others." Related work shows that debate accuracy can actually decrease over rounds, with agents shifting from correct answers to incorrect ones under peer pressure.
Read that carefully, because it inverts the naive picture. The default mental model is that agents talking to each other error-correct. The evidence says that agents that share training and can see each other frequently error-amplify. A confident wrong answer propagates through the group. Parallel, blind execution stops that from happening, and it is a good reason to build it that way from the start.
The devil's advocate question follows from the same principle. Adding an agent whose only job is to attack the consensus feels obviously useful. It is useful, but only under a condition most implementations quietly violate.
Self-Refine and Reflexion, both from 2023, are the canonical wins for self-critique. The problem Huang et al. also demonstrated is that those gains often depended on an oracle: some external signal telling the system whether it was actually right. Strip the oracle away and ask a model to re-examine its own reasoning against nothing but itself, and the intrinsic self-correction tends to flatten or degrade the answer.
Constitutional AI, Anthropic's method where a model critiques its outputs against a written set of principles, works. But notice why: the critique has an external anchor, the constitution. That is the whole distinction. A critic helps when it brings information the generator did not have. A different model, a scoring rubric, a retrieval step, or an executable test all qualify. A same-model devil's advocate with no external ground truth is close to the intrinsic self-correction case that does not reliably help.
| Approach | Does it beat the cheap baseline? | The condition that decides it |
|---|---|---|
| Same-model debate, homogeneous roles | Often no, loses to self-consistency at equal compute | Only with heavy tuning, and it collapses under load |
| Cross-model debate | Yes, on the evidence | Genuinely different models, not one model in costumes |
| Same-model self-critique, no oracle | No, can degrade reasoning | Needs an external anchor to add value |
| Critic with rubric, tool, or second model | Yes | The critic must bring information the generator lacks |
The swarm I would actually build#
I still believe in agent swarms. I have built them in production. They solve real problems.
I just no longer treat them as the starting architecture.
The first thing I build now is the boring baseline. Before writing the orchestrator, I run self-consistency on the actual decisions: sample the best single model several times, take the majority, record the accuracy and the cost. That is the number the swarm has to beat. If I skip it, I will never know whether the elaborate pipeline is earning its keep. And given the research, the base rate says it probably is not. This is the same instinct behind building evals from real production decisions rather than trusting that a fancier architecture is better because it feels more sophisticated.
If and only if self-consistency is not good enough, I reach for the swarm. And I build it under the conditions the evidence supports. Use genuinely different models where possible, not one model wearing personas. Keep the experts blind to each other so a confident wrong answer cannot propagate. Give any critic an external anchor, a rubric or a tool or a different model, so it is contributing information rather than re-reading itself. Reserve the whole apparatus for decisions where the stakes justify ten to twenty times the token cost: contested, high-value, low-frequency calls, not routine ones.
There is a broader caution worth holding here. The largest meta-analysis on combining decision-makers, Vaccaro, Almaatouq and Malone in Nature Human Behaviour in 2024, looked at 106 studies and 370 effect sizes of human and AI teams. On average the combinations performed significantly worse than the better of the two alone, and the losses were concentrated specifically in decision-making tasks. That is about human-AI teams, not agent-agent teams, so it does not transfer cleanly. But the mechanism it identifies, that combining two decision-makers of unequal skill often drags the result toward the weaker one, is exactly the failure mode a homogeneous agent swarm reproduces. More deliberators is not automatically better judgment. It is a hypothesis you have to test, on your data, against the cheap baseline.
Most decisions do not need five opinions. They need one opinion measured properly.
When the decision is expensive, irreversible, or genuinely contested, then structured disagreement earns its cost. I happily pay for it in those cases.
But every swarm I build now starts with the same question: can one model and a boring baseline already solve this?
If the answer is yes, I stop there.
FAQ#
What is an agent swarm in AI?#
An agent swarm is a set of language model agents that each analyze the same problem from an assigned role or perspective, then reconcile their conclusions into one decision. The idea is that structured disagreement between independent agents surfaces errors and angles a single model would miss. In practice its value depends heavily on the agents being genuinely different and on the decision being hard enough to justify the extra cost.
Does multi-agent debate actually improve accuracy?#
Sometimes, but often not once you compare fairly. Multi-agent debate improves over a single model response on several reasoning benchmarks. However, Huang et al. (ICLR 2024) showed it does not reliably beat self-consistency, which is majority voting over independent samples of one model, at equal compute. Much of the measured gain comes from spending more compute, not from agents collaborating. Always compare against the compute-matched baseline.
What is self-consistency and why does it matter here?#
Self-consistency samples the same model multiple times independently and takes the majority answer. It is the cheap baseline any multi-agent system must beat to justify its complexity. It matters because studies repeatedly find that debate and swarm architectures fail to outperform self-consistency when both are given the same compute budget, which means a lot of multi-agent systems are effectively doing expensive ensembling.
Why should agents in a swarm not see each other's answers?#
Because language models are sycophantic and conform to confident peers. When agents share training data and can see each other, a confident but wrong answer tends to propagate rather than get corrected. Research on inter-agent sycophancy shows debate accuracy can decrease over rounds as agents shift from correct to incorrect under peer pressure. Keeping agents blind and running them in parallel preserves the independence that makes disagreement informative.
When is building an agent swarm worth it?#
When three conditions hold. The decision is genuinely contested and high-stakes, so being wrong is costly and rare. You can use real heterogeneity, ideally different models rather than one model with different personas. And any critic in the loop has an external anchor like a rubric, a tool, or a second model. For routine, high-frequency decisions, a single model with self-consistency is usually the better trade.
Does adding a devil's advocate agent help?#
Only if it brings information the generator lacks. A critic anchored to an external signal, a scoring rubric, an executable test, retrieval, or a different model, can genuinely improve the output. A same-model critic asked to re-examine its own reasoning against nothing external tends to add variance without reliably catching errors, the same way intrinsic self-correction degrades reasoning in the Huang et al. results.
Is per-model heterogeneity better than role prompting?#
On the current evidence, yes. The strongest debate result in the founding paper came from two different models, ChatGPT and Bard, catching each other's errors, not from one model playing multiple roles. Homogeneous swarms with cosmetic personas have been shown to collapse into groupthink under cognitive load. If you want durable gains from a swarm, invest in genuinely different underlying models or genuinely enforced, externally grounded roles.