Where multi-agent systems have delivered measurable gains

What actually improved with multiple agents?

Giving a difficult job to several AI agents does not guarantee a better answer. They can adopt one another’s mistakes, repeat the same searches, or spend their compute budget communicating. Yet research has improved code correctness, research evaluations, answer quality, and the novelty of ideas, and produced scientific hypotheses for experimental validation. Which jobs benefited, and what did the systems separate?

This article uses multi-agent to mean multiple LLM calls that maintain separate information or histories, produce proposals or perform work, and pass their results to other workers or a synthesis process. They may use the same model or different models. Simple voting and predefined divisions of work remain comparison points. Since papers use “agent” differently, autonomous tool use and the generation of answer candidates are distinguished.

My interpretation of the successful cases is that work with independently searchable information, stages that can be isolated and integrated while preserving dependencies, verification separated from generation, or diverse candidates that can be selected and combined offers a concrete reason to test multiple agents. Calling a task complex does not establish what an additional worker will contribute.

AgentCoder separates code generation from test design

AgentCoder makes the effect of separation unusually tangible. One LLM writes code; another designs tests from the problem without seeing the complete generated implementation. A Python script executes the tests and returns failures to the programmer. The paper’s three-agent terminology therefore does not mean three conversational LLMs.

In Table 7, separating the workers raises pass@1 from 71.3% to 79.9% on HumanEval and from 79.4% to 89.9% on MBPP compared with generating code and tests in the same conversation. These are gains of 8.6 and 10.5 percentage points. Pass@1 evaluates the one code candidate finally submitted after the system’s internal refinement; it does not mean a single LLM call.

The design aims to prevent assumptions in the implementation from shaping its tests. If the code overlooks empty input, tests generated in the same history may inherit that omission. A separate worker can search the specification for boundary cases independently. Execution then turns criticism into a concrete failure that the programmer can address.

The design lesson I would adopt is separation of generation and verification, coupled with execution feedback. Prompts, context separation, and iteration also contribute, so the result does not isolate headcount alone. Function-level benchmarks do not establish success on changes to a large repository, and a test designer can still invent an incorrect expected result.

CAID improves development with interdependent stages

AgentCoder’s function-level results cannot simply be extended to long development jobs. For dependent stages, I would start with one agent retaining the working history. The OneFlow study reports across seven benchmarks that a single agent executing the roles of homogeneous workflows sequentially can match their performance. More stages or role names alone do not justify multiple agents.

CAID offers an improvement from that starting point. A manager tracks dependencies and delegates executable parts to isolated workspaces. Workers verify and submit changes for integration; tasks with unmet dependencies wait. The overall job remains dependent even though selected parts can run independently.

With Claude Sonnet 4.5 on PaperBench Code-Dev, the single-agent score is 57.2, shared-workspace multi-agent execution scores 55.5, and CAID with git worktree isolation scores 63.3. These are code-development evaluation scores, not rates of fully reproducing experimental results. Isolation and explicit integration distinguish this result from simply increasing headcount.

This is not an equal-compute comparison. Iteration limits differ between the single agent and the manager–worker configuration, and CAID costs more and takes longer in this setting. Quality gains should be separated from speed gains.

I would consider this pattern where one manager can retain global dependencies while partial changes are isolated, checked, and integrated. It does not establish the same case for a process whose every stage must be handed off serially. Knowing which changes have actually been integrated is a condition of this successful design.

Parallel investigation helps when the work can be divided

CAID selects independently executable work while preserving dependencies. Research and analysis can instead divide the collection of an artifact’s ingredients. Revenue, costs, and market conditions can be examined separately before they are combined into one analysis, reducing the need for every worker to share the entire evolving history.

Kim and colleagues’ 2026 study evaluates 260 configurations across six benchmarks, five architectures, and three LLM families. On Finance Agent, centralized coordination raises mean success from 34.9% to 63.1%: 28.2 percentage points, or an 80.8% relative improvement. Each model–architecture cell uses 50 questions, with matched tools and system-wide compute ceilings.

All multi-agent architectures in the same study lose performance on PlanCraft. The authors contrast decomposable financial analysis with sequential planning that must track conditions changed by previous actions. My reading is that the ability to produce intermediate results independently is a better starting point for delegation than difficulty alone.

An implementation example comes from Anthropic’s technical report on Research. An Opus 4 lead with parallel Sonnet 4 workers reportedly outperforms a single Opus 4 by 90.2% on an internal research evaluation. Its example asks for the board members of Information Technology companies in the S&P 500. Investigations can proceed company by company, giving workers distinct assignments.

This is a vendor report rather than a shared, peer-reviewed benchmark comparison, and it does not establish a 90.2% gain at equal cost. Its estimate of roughly 15 times the tokens compares multi-agent systems with ordinary chats, not with a single agent. I would test this pattern for valuable research with separable companies or document collections and checkable records. It does not justify the same architecture for a short factual lookup.

Divisions that helped computer use and sequential execution

Long computer-use tasks can divide capabilities rather than isolate code changes. Agent S2 separates high-level planning, actions, and precise identification of screen targets. It updates plans from observations after actions and uses specialized grounding capabilities. On OSWorld with a 50-action limit, Claude Computer Use with Claude 3.7 Sonnet scores 26.0%, compared with Agent S2’s 34.5%.

The system also uses UI-TARS, text recognition, and structural application interfaces for grounding. It demonstrates a combination of specialist capabilities, not an isolated effect of multiplying the same LLM. I would consider this pattern where reading the screen or executing precise actions is a demonstrated bottleneck.

Another study changes handoff history without changing the model. An information-bottleneck analysis connects object search and subsequent action through a short relay in the ALFWorld household simulator. In the zero-shot setting with Qwen2.5-7B, success is 36.6% for ordinary single-agent execution and 54.5% for the multi-agent setup. It also includes a single-context control following the same stages and matches a maximum total of 50 steps across 134 tasks.

This supports removing irrelevant search history while retaining needed objects and locations. However, with the stronger 4-bit-quantized Qwen3.5-27B in the same setting, ordinary single-agent execution scores 90.3% versus 89.5% for multiple agents. Tokens and costs are not matched. Before adding workers, I would also test whether managing one agent’s history achieves the same effect.

Changing history or specialization still leaves actions dependent on earlier results. M1-Parallel retains those dependencies within each team while running several complete attempts concurrently. On GAIA validation, returning the first answer from three teams reports a 1.6–1.8× speedup with maintained completion, at 1.7–1.8× inference cost. Its comparator, Magentic-One, is already multi-agent; this does not demonstrate superiority over a single agent.

A separate mode waits to combine answers and execution logs, increasing solved tasks but generally also increasing latency. Early stopping and aggregation serve different objectives and should not be combined into one claim of higher speed and accuracy. The first returned answer is not selected by a correctness oracle.

I would compare this pattern for research with alternative solution paths where reduced waiting justifies extra cost. Replicating an entire workflow that commits orders or updates records could repeat those actions. A design for parallel attempts must also determine which one result is allowed to drive writes.

Mixture-of-Agents combines different answers

Where M1-Parallel combines records from complete execution paths, another approach combines answer candidates themselves. Mixture-of-Agents, or MoA, has several models propose answers and subsequent layers rewrite them using earlier responses. Its agents primarily propose and aggregate text; they are different from autonomous browser workers.

A six-proposer, three-layer system reports a 65.1% length-controlled win rate on AlpacaEval 2.0, compared with 57.5% for the May 2024 GPT-4o, a difference of 7.6 percentage points. This measures preference against benchmark reference answers; it does not mean 65.1% factual accuracy. Different constituent models and call counts also prevent this comparison from establishing coordination efficiency alone.

A separate diversity comparison in Table 3 fixes Qwen1.5-110B-Chat as the aggregator in a two-layer, six-proposal setup. Six proposals from the same model score 56.7%, while proposals from different models score 61.3%. These are different settings from the three-layer headline result. Holding proposal count constant supports investigating diversity, although model sizes and computational costs are not equalized.

I connect this result to work where synthesis can reuse complementary explanations or perspectives. Combining drafts to repair omissions is one candidate. If every proposer repeats the same false fact, aggregation adds no independent evidence; checking the original sources remains necessary.

Creative exploration shows gains in novelty

Finding a single correct answer and widening a field of ideas need different outcome measures. Hu and colleagues’ 2026 preprint compares human and LLM teams on six tasks spanning social challenges, business problems, and hypothetical situations. Five human judges rate novelty and usefulness, with their product defining creativity.

For GPT-4.1, the mean rises from 0.253 for a single agent to 0.298 for that model’s best multi-agent configuration. The best configurations also improve on single-agent baselines for two o3 reasoning settings. These are selected best configurations, not evidence that arbitrary teams achieve the same gain. Before rating, an LLM harmonizes the wording of ideas to reduce presentation differences.

The advantage over human teams is driven mainly by novelty, with comparable usefulness. That is a result about exploration during ideation. The study does not measure implementation success, customer adoption, or business performance. An application to planning should separately measure the diversity of proposals and their usefulness after selection.

For this purpose, forcing early agreement could remove the differences the system was intended to produce. I would test independent proposal generation followed by selection against constraints, evaluating the sequence together with the chosen model and discussion protocol.

Co-Scientist takes proposals into experimental validation

Co-Scientist goes beyond rating an idea by connecting proposals to experiments. The 2025 preprint, “Towards an AI co-scientist,” became “Accelerating scientific discovery with Co-Scientist” in its June 2026 revision. Built on Gemini, it uses multiple workers to generate, critique, and refine hypotheses against researchers’ objectives and prior evidence.

The system repeatedly compares and improves hypotheses through tournament-style evaluation. The revised paper reports assistance in identifying drug-repurposing candidates and synergistic combinations for acute myeloid leukemia, validated in vitro. The achievement includes producing proposals that researchers can test outside the LLM system.

Experimental results do not, by themselves, prove that an equally funded single agent could not have produced them. Additional computation, hypothesis selection, scientific guidance, and experimental design all contribute. In vitro validation must also be distinguished from clinical efficacy.

The adoption candidate I draw from this is work where evaluating proposals is expensive enough to make broad generation and criticism worthwhile before an experiment. Useful output is a testable candidate and its external validation result, rather than the volume of discussion.

Measure debate’s additional value against voting

Comparing candidates can help without making longer debate beneficial. Du and colleagues’ early study reports gains in reasoning and factuality when models propose answers and reasoning, then revise through discussion. Debate or Vote later separates components across seven language benchmarks and finds that majority voting explains most gains commonly attributed to debate.

Together, these studies motivate separate measurement of the benefit from multiple candidates and the extra benefit from communication. Self-Consistency, which samples reasoning paths from one model and aggregates answers, supplies a comparison. Voting or Consensus? also shows sensitivity to decision protocols and reports that more discussion before voting can reduce performance.

When adopting debate, I would specify what another worker’s message should add: a counterexample, a different source, or an executed test failure. Those can justify updating a candidate. Restating the same speculation adds no material for deciding whether it is correct.

Handoffs and verification also affect outcomes. The MAST failure analysis organizes over 1,600 traces from seven frameworks into 14 failure modes involving design, inter-agent misalignment, and verification. Independent work is insufficient if the receiving worker loses the sources or assumptions needed to use it.

Which jobs should be tested first?

Transferring these results into practice requires matching a single agent’s concrete shortcoming with the contribution of an additional worker. The following candidates are my synthesis of the research, not measured results for each business workflow.

For work where each action determines the next, start with a single agent executing the same stages. Then ask whether CAID-style dependency management can isolate changes or whether Agent S2-style specialists address a real bottleneck. Passing each stage to a different worker can otherwise lose needed history while adding handoff overhead.

A comparison can start with a single agent, repeated generation or voting from one model, and a system with separate roles on the same task set. Match models, tools, system-wide token or spending ceilings, and trial counts. Separate an equal-budget comparison from an experiment aimed at reducing elapsed time. This is a proposed evaluation approach, not a procedure shared by every cited study.

As AI Agents That Matter argues, accuracy and cost should be evaluated together. Include failures and retries when measuring cost per task, median and slow-case latency, and human correction time. A modest accuracy gain provides a weaker adoption case if checking the output creates more work.

For AgentCoder, can another worker test conditions the generator missed? For CAID, can parts of dependent development be isolated, verified, and integrated? For parallel research, can workers investigate different sources and retain their provenance through synthesis? For a Co-Scientist-like system, can proposals move into external experiments? Moving to multiple agents is justified when an additional worker addresses such a concrete shortcoming and its contribution can be demonstrated against a single system doing the same job.