Source-linked AI summary
Multi-Agent Teams Hold Experts Back
Aneesh Pappu, Batu El, Hancheng Cao, Carmelo di Nolfo, Yanchao Sun, Meng Cao, James Zou
TL;DR
The paper asks whether self-organizing LLM teams can harness heterogeneous expertise when coordination is unconstrained, addressing a gap left by fixed roles, workflows, and aggregation rules. Using human-inspired decision tasks and frontier ML benchmarks, it finds that teams consistently fail to match their best member—even when the expert is identified—because they compromise instead of leveraging expertise, although this behavior improves adversarial robustness.
Problem
Existing work often engineers coordination through fixed roles, workflows, or aggregation rules, leaving open whether unconstrained self-organizing LLM teams can achieve strong synergy by matching or exceeding their strongest member.
Method
The paper evaluates self-organizing teams across organizational-psychology decision tasks and five frontier ML benchmarks, including conditions that reveal expert identity and ablations separating expert identification from leveraging.
Results
LLM teams consistently underperform their best member, with relative synergy gaps of 6.3%–41.1% on ML benchmarks; leveraging, not identification, is the primary bottleneck, while compromise correlates negatively with performance.
Takeaways & Limitations
Self-organizing LLM teams do not reliably harness differential expertise, while their consensus-seeking behavior provides robustness to adversarial agents and creates a trade-off with effective expertise utilization.
Takeaways & Limitations
The attribution of consensus-seeking to alignment is correlational, and the evaluation covers only a small subset of tasks and five ML benchmarks compared with more complex real-world collaboration.
Abstract
from arXiv · showhide
Multi-agent LLM systems are increasingly deployed as autonomous collaborators, where agents interact freely rather than execute fixed, pre-specified workflows. In such settings, effective coordination cannot be fully designed in advance and must instead emerge through interaction. However, most prior work enforces coordination through fixed roles, workflows, or aggregation rules, leaving open the question of how well self-organizing teams perform when coordination is unconstrained. Drawing on organizational psychology, we study whether self-organizing LLM teams achieve strong synergy, where team performance matches or exceeds the best individual member. Across human-inspired and frontier ML benchmarks, we find that -- unlike human teams -- LLM teams consistently fail to match their expert agent's performance, even when explicitly told who the expert is, incurring performance losses of up to 41.1% on ML benchmarks. Decomposing this failure, we show that expert leveraging, rather than identification, is the primary bottleneck. Conversational analysis reveals a tendency toward integrative compromise -- averaging expert and non-expert views rather than appropriately weighting expertise -- which increases with team size and correlates negatively with performance. Interestingly, this consensus-seeking behavior improves robustness to adversarial agents, suggesting a trade-off between alignment and effective expertise utilization. Our findings reveal a significant gap in the ability of self-organizing multi-agent teams to harness the collective expertise of their members.
1. Introduction
The paper tests whether unconstrained, self-organizing LLM teams can match their strongest member and finds that they generally cannot. Their main failure is leveraging expertise: teams compromise between views, with dilution worsening as teams grow, although consensus improves adversarial robustness.
- Evaluation criterion: Strong synergy requires teams to match or exceed their strongest member, providing a model-agnostic test of whether interaction leverages rather than dilutes expertise.The paper applies this criterion to self-organizing teams because coordination must emerge through interaction.
- Evaluation settings: The evaluation spans organizational-psychology decision tasks and frontier ML benchmarks to study self-organization under both controlled and realistic conditions.The human-inspired tasks include NASA Moon Survival, Lost at Sea, and Student Body President; ML benchmarks include MMLU Pro, GPQA Diamond, HLE, MATH-500, and SimpleQA.
- Main finding: LLM teams consistently underperform their best member by 6.3%–41.1% on frontier ML benchmarks, unlike human teams that match expert performance when expertise is revealed.The comparison spans human-inspired tasks and modern ML benchmarks.
- Failure decomposition: Controlled ablations identify expert leveraging, rather than expert identification, as the primary bottleneck even when teams are explicitly told who the expert is.Teams fail to harness expert knowledge despite receiving expert-identity information.
- Failure mechanism: Conversational analysis finds integrative compromise: agents average expert and non-expert views instead of appropriately weighting superior knowledge, and compromise correlates negatively with performance.This behavior is associated with underperformance rather than effective expertise use.
- Trade-off: Expertise dilution worsens with team size, while consensus-seeking also provides robustness to adversarial team members, creating a trade-off between expertise use and manipulation resistance.The same consensus behavior that limits expert utilization can filter adversarial input.
2. Background and Related Work
The paper differs from much prior multi-agent work by studying heterogeneous frontier models in unconstrained deliberation rather than fixed routing or task decomposition. It evaluates whether such teams achieve strong synergy against their best member, drawing on human teamwork research where revealed expertise supports expert-level group performance.
- Model heterogeneity: The study uses heterogeneous frontier models with distinct training procedures and differential expertise, rather than mainly using copies of one model or role personas.This addresses model heterogeneity as a distinct axis of multi-agent collaboration.
- Coordination structure: Unlike systems with learned topologies or structured routing, the paper studies unconstrained group deliberation with open communication and no pre-specified roles.This setting more closely mirrors human team interaction.
- Evaluation goal: The paper evaluates synergy by comparing team output with the best individual member, rather than only against external benchmarks or the average member.This reframes evaluation around whether collaboration harnesses the strongest available expertise.
- Research distinction: The paper’s central distinction is between optimizing task decomposition and aggregation versus asking whether self-organizing teams can achieve synergistic performance beyond their strongest member.This positions strong synergy as the paper’s research goal.
- Human-team background: Human teams can match or exceed their best member when expertise is identifiable and solution validity is demonstrable, while unknown expertise usually yields only weak synergy.When expertise is explicitly revealed, human teams reliably reach expert-level performance.
- Task foundations: The study uses canonical intellective tasks, including NASA Moon Survival, Lost at Sea, and Student Body President, to test whether LLM teams defer to knowledgeable members similarly to human teams.These tasks come from organizational-psychology research on expertise and hidden information.
3. Setup
The study evaluates four-agent LLM teams across human psychology tasks and frontier ML benchmarks under controlled expertise distributions and information conditions. It measures whether teams can leverage expertise by comparing team performance with the strongest individual and decomposing performance gaps into identification and leveraging components.
- Experimental overview: Experiments use teams of four models discussing for four rounds, with final answers determined by majority vote after discussion.The setup also varies whether teams know which agent is the expert.
- Tasks and benchmarks: The evaluation combines NASA Moon Survival, Lost at Sea, and Student Body President with MMLU Pro, SimpleQA, GPQA Diamond, HLE, and MATH-500.The two task categories support controlled expertise manipulation and testing on heterogeneous frontier models.
- Expertise distribution: Expertise is either concentrated in one team member or distributed mutually exclusively across multiple members.For ML benchmarks, the best-performing model can vary by problem, producing question-level concentrated expertise and task-level distributed expertise.
- Human psychology tasks: Ranking-task performance uses L1 error, the sum of absolute position differences from the ground-truth ranking.NASA Moon Survival and Lost at Sea compare against NASA and US Coast Guard expert rankings, while Student Body President uses the objectively best ranking from complete information.
- Performance gaps: The relative synergy gap measures team underperformance against the best individual, with 0% indicating matched expert performance and larger positive values indicating greater underperformance.For accuracy tasks it compares the best individual’s performance with team performance; for error-based tasks, the numerator is reversed because lower error is better.
- ML benchmark conditions: The At Least One Correct upper bound estimates performance when the team perfectly identifies and leverages the agent with the correct answer on each ML problem.The expert may change across problems, and the Reveal Expert condition identifies the most expert agent rather than guaranteeing that agent has the correct answer.
4. Experiments
Across human psychology tasks and frontier ML benchmarks, LLM teams consistently underperform their strongest member, even when experts are explicitly identified. The experiments attribute this gap primarily to ineffective expertise leveraging, with larger teams worsening dilution while consensus-seeking offers adversarial robustness.
- Human Psychology Tasks: Revealing the expert provides only modest improvement over allowing teams to identify expertise themselves, indicating that leveraging—not identification—is the primary bottleneck.The same conclusion appears in concentrated-expertise results and the Lost at Sea figure.
- Human Psychology Tasks: LLM teams consistently underperform their best member across concentrated and distributed expertise settings, despite prompts instructing them to defer to identified experts.This pattern holds across all three human psychology tasks.
- Machine Learning Benchmarks: 6.3%–41.1% relative synergy gaps occur across frontier ML benchmarks, and no tested coordination protocol reaches the At Least One Correct upper bound.The benchmarks include MMLU Pro, SimpleQA, GPQA Diamond, HLE text-only, and MATH-500; results use 100 problems per benchmark.
- Expertise Dilution with Team Size: Larger teams show statistically significant positive correlations between team size and synergy gap across tasks and information conditions, including Reveal Expert.The effect persists across model compositions and indicates expertise dilution as teams grow.
- Robustness to Adversarial Team Members: The same dilution mechanism that limits expert knowledge leveraging also produces minimal performance degradation under adversarial input across team sizes and configurations.The authors present this as a trade-off between harnessing differential expertise and filtering adversarial contributions.
- Conversational Analysis: Integrative compromise correlates with larger synergy gaps, whereas epistemic deference correlates with better performance when non-experts yield to experts.For NASA Moon Survival, integrative compromise correlates with the synergy gap at r = 0.55, p < 0.001, while epistemic deference correlates with performance at r = −0.44, p = 0.007.
5. Discussion
The discussion attributes teams’ expertise failures to consensus-seeking that undermines expertise leveraging, while noting that the alignment attribution remains untested and the evidence has limited scope.
- Consensus-seeking rather than expertise leveraging is the paper’s central tension in explaining team underperformance.The authors hypothesize, but do not establish, that alignment procedures may partly encourage this behavior.
- Consensus-seeking can filter adversarial input while preventing teams from harnessing differential expertise.This creates a trade-off between robustness to manipulation and effective expertise utilization.
- LLM teams’ integrative compromise suggests a failure to let knowledgeable members demonstrate solutions to less-informed members.The paper links this failure to the demonstrability condition required for effective group performance on intellective tasks.
- The paper’s broad limitation is that its alignment explanation is correlational and its evaluation covers only a small subset of possible tasks and deployments.The authors do not compare aligned models with base counterparts, and real-world collaboration may be more complex.
6. Conclusion
The conclusion reports persistent strong-synergy failures alongside weak synergy and adversarial robustness, leaving contextual expertise leveraging as an open challenge.
- 6.3%–41.1%: LLM teams underperform the best individual even when told who the expert is.They instead engage in integrative compromise, and this failure worsens with team size.
- Teams consistently achieve weak synergy by matching or outperforming the average member despite failing to match the best member.Consensus-seeking also provides robustness to adversarial input.
- Contextual expertise leveraging without sacrificing robustness remains an open training challenge.
Impact Statement
The paper cautions that unconstrained multi-agent collaboration may underperform a strong individual, and describes a standardized deliberation protocol designed to measure and test expertise leveraging.
- Multi-agent teams may consistently underperform a single strong expert when expertise is asymmetrically distributed.The impact statement highlights domains such as medical diagnosis, legal analysis, and financial decision-making.
- The protocol collects independent opinions, randomizes discussion order, runs four discussion rounds, and uses majority vote for the final team answer.
- Randomized speaking order mitigates position bias, while individual baselines support synergy measurement.
- Majority voting gives no single agent structural advantage over the team answer, and unstructured discussion preserves self-organization.
- The protocol is held constant across tasks and experimental conditions.
- The experiments tested prompts discouraging compromise and emphasizing expert deference, yet synergy gaps persisted.
- Expert-reveal prompts explicitly identify specialized agents and instruct teams to prioritize their relevant input.The concentrated-expertise prompt directs teams to adopt the expert ranking as a baseline and make only justified refinements.
- Human-task results report team and expert ranking errors alongside absolute and relative synergy gaps, with lower ranking error indicating better performance.
C. Team Performance: Human Psychology Tasks
Across human psychology tasks, teams persistently underperform the best individual across expertise arrangements while generally matching or exceeding the member average.
- Teams’ expertise leveraging gap persists across all human-task team compositions, including concentrated and distributed expertise.
- Teams consistently underperform the best individual whether expertise is concentrated in one agent or distributed across multiple agents.The figures compare No Information, Expert Not Mentioned, Reveal Expert, and Best Individual conditions; lower ranking error is better.
- Human-task performance is reported using team and expert ranking errors, absolute synergy gaps, and relative synergy gaps.Ranking error is measured as L1 distance from the ground-truth ranking, with lower values indicating better performance.
- Teams achieve weak synergy by matching or outperforming the average of individual member performances despite failing to achieve strong synergy.Figures 8–10 use the member average as the comparison baseline.
D. Full Information Ablation: Isolating Communication Noise
The Full Information ablation separates communication noise from expertise asymmetry by giving every agent the complete expert information before identical deliberation. Teams of four experts match a single expert, whereas concentrating expertise in one agent produces a large gap, isolating expertise distribution as the cause.
- A team of four experts performs comparably to a single expert, showing that the discussion protocol does not meaningfully degrade performance.The team performs slightly better than a single expert on two of three human psychology tasks.
- Replacing three experts with non-experts under the same protocol and team size collapses performance, isolating expertise asymmetry rather than communication overhead.The cited NASA Moon Survival comparison is 16.22 →25.35.
- The ablation uses four agents, four rounds, three seeds, and three GPT-4o-mini/Claude 3.5 Haiku team compositions.
- The ML benchmark setup compares team performance under Expert Not Mentioned and Reveal Expert conditions against individual, Best Individual, and At Least One Correct baselines.The baselines are used to assess whether heterogeneous models can leverage differential expertise.
E.2. Weak Synergy Analysis
Teams achieve weak synergy on ML benchmarks by outperforming their members’ average, but they do not achieve strong synergy against the At Least One Correct upper bound. Additional protocols and prompt optimization likewise leave teams below that upper bound.
- Teams exceed the member-average accuracy on every ML benchmark but remain below the At Least One Correct upper bound.Thus, the teams achieve weak synergy without achieving strong synergy.
- Every tested coordination protocol and information condition remains below the At Least One Correct upper bound.The comparison includes Debate, opt-out, and CoT+MV.
- Even after GEPA prompt optimization, teams fail to reach At Least One Correct on any ML benchmark.On NASA Moon Survival, optimization narrows the gap from 23.2 to 16.8, but the team still trails the single expert at 13.6.
- GEPA converges on extreme-deference prompts that treat the expert as correct by default and override that answer only under an exacting evidentiary bar.The resulting behavior effectively suppresses deliberation rather than improving expertise use through discussion.
- The optimized results indicate that team deliberation, rather than Reveal Expert prompt phrasing, is the bottleneck.
H.1. Correlation Analysis
Larger teams increasingly underperform their expert across tasks and information conditions. The effect persists when the expert is identified and across team compositions, consistent with expertise dilution driven by compromise pressure and coordination costs.
- The downward synergy-gap trend remains across homogeneous and heterogeneous Anthropic/OpenAI team compositions.The visualization includes 100% Anthropic, 50% Anthropic/50% OpenAI, and 100% OpenAI configurations.
- The synergy gap is defined as Best Individual Score minus Team Score, with negative values indicating that the team underperformed its best individual.
- All team-size correlations with the strong synergy gap are significant and positive across the three tasks and information conditions.Positive correlations mean that larger teams underperform the expert by greater margins; all p < 0.05.
- The expertise dilution effect persists in the Reveal Expert condition, showing that identifying the expert does not ensure effective expertise use.
- The proposed mechanisms are increasing pressure toward consensus and greater coordination costs as team size grows.Both mechanisms may prevent teams from properly processing and leveraging expert contributions.
- Scaling team size may actively harm performance when expertise asymmetries exist.
I. Adversarial Robustness
Adversarial-member experiments test whether teams resist an agent instructed to worsen performance. Teams show minimal degradation, suggesting that consensus-seeking dilutes adversarial input while also diluting expert input.
- Teams show minimal performance degradation when one member receives the worst possible ranking and is instructed to sabotage performance.The experiments use the no-info setting, where only the adversarial agent receives special information.
- Consensus-seeking appears to protect teams from adversarial members when their rankings diverge substantially from the majority.
- The same mechanism that dilutes adversarial input also prevents teams from effectively leveraging expert knowledge.This creates a trade-off between robustness to adversaries and expertise utilization.
J. Epistemic Deference Analysis Methodology
The appendix defines a role-specific coding scheme for epistemic behaviors and applies it to team transcripts after the expert is revealed. It then correlates coded event frequencies with team performance across three tasks.
- Coding scheme: The scheme distinguishes non-expert deference or compromise from expert persistence or flexibility.Epistemic Deference and Integrative Compromise apply to non-experts; Strategic Persistence and Epistemic Flexibility apply to experts.
- Analysis pipeline: Transcript extraction records full deliberations, speaker identities, and turn numbers before automated role-specific coding.Gemini 3.0 Pro codes each turn, and event frequencies are counted for ED, IC, SP, and EF.
- Analysis pipeline: Pearson correlations relate coded event frequencies to the performance gap between team score and best individual score.The analysis uses the difference between Team Score and Best Individual Score as its performance-gap measure.
- Validity controls: Four validity gates restrict coding to post-reveal disagreements, enforce role-appropriate labels, and exclude system messages and experimental instructions.The Reveal, Discrepancy, Role-Code Lock, and Experimenter gates constrain which transcript turns enter analysis.
- Task coverage: Lost at Sea analysis uses n=30 conversations, while Student Body President uses n=28 after two exclusions for data quality.Lost at Sea correlations do not reach statistical significance; Student Body President exhibits the strongest correlation effects across event types.