PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity

Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, Shangding Gu

arXiv:2602.03794v1cs.AIcs.LG

TL;DR

Multi-agent systems can improve difficult-task performance, but it remains unclear why adding agents saturates and why diversity helps. The paper develops an information-theoretic framework and a label-free effective-channel metric, finding that heterogeneous systems can achieve comparable or better performance with far fewer agents.

  • Problem

    The paper addresses the question of why homogeneous MAS scaling has diminishing returns and why heterogeneity provides additional gains.

  • Method

    The paper derives architecture-agnostic information-theoretic bounds and introduces K* to quantify effective channels without ground-truth labels.

  • Results

    Heterogeneous configurations consistently outperform homogeneous scaling, with 2 diverse agents matching or exceeding 16 homogeneous agents.

  • Takeaways & Limitations

    MAS gains depend on allocating diverse, non-redundant information channels rather than merely increasing raw agent count.

  • Takeaways & Limitations

    The theoretical analysis relies on idealized sufficiency, conditional-independence, and uniform-coverage assumptions that may not hold for real-world agents.

Abstract

from arXiv · show

LLM-based multi-agent systems (MAS) have emerged as a promising approach to tackle complex tasks that are difficult for individual LLMs. A natural strategy is to scale performance by increasing the number of agents; however, we find that such scaling exhibits strong diminishing returns in homogeneous settings, while introducing heterogeneity (e.g., different models, prompts, or tools) continues to yield substantial gains. This raises a fundamental question: what limits scaling, and why does diversity help? We present an information-theoretic framework showing that MAS performance is bounded by the intrinsic task uncertainty, not by agent count. We derive architecture-agnostic bounds demonstrating that improvements depend on how many effective channels the system accesses. Homogeneous agents saturate early because their outputs are strongly correlated, whereas heterogeneous agents contribute complementary evidence. We further introduce $K^*$, an effective channel count that quantifies the number of effective channels without ground-truth labels. Empirically, we show that heterogeneous configurations consistently outperform homogeneous scaling: 2 diverse agents can match or exceed the performance of 16 homogeneous agents. Our results provide principled guidelines for building efficient and robust MAS through diversity-aware design. Code and Dataset are available at the link: https://github.com/SafeRL-Lab/Agent-Scaling.

1. Introduction

The paper asks why adding homogeneous agents yields diminishing returns and argues that diversity sustains gains by providing complementary, non-redundant information through more effective channels.

  • Scaling and diversity: Homogeneous agent scaling improves accuracy initially but exhibits rapidly collapsing marginal gains as agent count increases.This saturation is attributed to increasingly redundant trajectories and highly correlated outputs.
  • Scaling and diversity: Diversity in models, prompts, personas, or tools introduces more complementary information and can outperform homogeneous systems with fewer agent calls.The paper motivates this pattern through experiments showing sustained improvements from heterogeneous configurations.
  • Empirical findings: Two diverse agents can match or exceed sixteen homogeneous agents across seven benchmarks.This empirical comparison demonstrates the reported efficiency advantage of heterogeneous configurations.
  • Information-theoretic framework: The framework bounds MAS performance by intrinsic task uncertainty and links improvement to the number of effective channels rather than raw agent count.It explains saturation through correlated outputs that reduce effective channels.
  • Information-theoretic framework: K* quantifies non-redundant information sources in agent outputs without requiring ground-truth labels.The metric is introduced as a way to measure effective channels and support diversity-driven system design.

2. Related Works

Prior work studies information flow and multi-agent coordination, but this paper addresses the lack of a unified theoretical account of diminishing returns and diversity in MAS.

  • Information-theoretic analysis: Information-theoretic studies of LLM reasoning characterize information flow but do not explain diversity constraints or derive performance bounds.The paper positions its contribution as explaining diminishing returns through formal limits.
  • LLM-based multi-agent systems: Existing MAS designs include independent sampling, voting, debate, role-playing, centralized orchestration, and hybrid coordination frameworks.These approaches use communication, coordination, or aggregation to perform compound inference.
  • Scaling and diversity: Empirical studies report limited benefits from naively scaling homogeneous agents across voting, debate, and broader coordination mechanisms.Related evidence spans multiple MAS workflows rather than a single architecture.
  • Scaling and diversity: Related work finds that diversity can improve success rates and heterogeneous ensembles, while its benefits vary with disagreement and task complexity.These findings motivate analyzing diversity as a central factor in MAS performance.

3. Problem Formulation

This section formalizes LLM-based multi-agent systems, defines usable evidence as information about the answer extracted beyond the input, and shows why intrinsic uncertainty and non-redundant evidence constrain scaling. It also distinguishes agent count from effective information contributions across workflows.

  • 3.1. LLM-based Multi-Agent Systems: An LLM-based multi-agent system uses configured agents that execute calls through a workflow and aggregate outputs into a final answer.Configurations specify backbone model, prompt or persona, decoding strategy, and tool access.
  • 3.1. LLM-based Multi-Agent Systems: The analysis distinguishes N agents from n agent calls, with multi-round workflows such as debate producing n = N × R calls.This distinction focuses analysis on extracted information rather than which agent produced it.
  • 3.2. Usable Evidence and Information Budget: Usable evidence measures how much observing the transcript reduces uncertainty about Y beyond the information already contained in X.The transcript may include outputs that depend on the input and preceding outputs.
  • 3.2. Usable Evidence and Information Budget: The incremental contribution of call i is Δi = I(Zi; Y | X, Z<i), representing new information given all previous outputs.The chain-rule decomposition makes cumulative usable evidence depend on incremental contributions.
  • 3.2. Usable Evidence and Information Budget: Performance depends on non-redundant evidence rather than total calls; correlated outputs make Δi diminish rapidly, while heterogeneous agents provide complementary coverage.Figure 3 contrasts redundant homogeneous behavior with more diverse information processing from heterogeneous configurations.
  • 3.2. Usable Evidence and Information Budget: No MAS can extract more information about Y than the intrinsic task uncertainty H(Y | X), so scaling benefits plateau near this ceiling.Homogeneous systems may saturate earlier because redundant evidence reaches the ceiling less efficiently.
  • 3.3. Agent Configuration Types: Agent diversity varies backbone models, prompts or personas, decoding strategies, and tool access, represented as configuration types for each call.Each call is associated with a type b(i) from the configuration-type set.
  • 3.4. Type-Dependent Ceilings Across workflows: Across parallel and sequential workflows, achievable information gain is controlled by instantiated configuration types and composition rather than raw call count n.Debate is a sequential-interaction case and inherits the same ceiling; this motivates the effective channel count.

4. Why Diversity Matters

The paper explains MAS scaling through effective channels and complementarity: performance depends on how many non-redundant information sources are accessed and how much fresh evidence they provide. This framework accounts for diminishing returns in homogeneous systems and formalizes the advantage of heterogeneous designs.

  • 4.1. Effective Channels: From Compute to Usable Evidence: An effective channel represents one independent source of task-relevant information; nearly identical reasoning contributes one channel despite multiple calls.Genuinely different reasoning paths can contribute multiple channels.
  • 4.1. Effective Channels: From Compute to Usable Evidence: The complementarity rate α measures how likely a new effective channel is to uncover previously missing task-relevant evidence.Higher α indicates fresher evidence, while lower α indicates greater overlap with existing information.
  • 4.1. Effective Channels: From Compute to Usable Evidence: An effective-channel representation maps the full transcript Z1:n to K aggregated channels Ẑ1:K, where K counts non-redundant information sources.The aggregation map ϕ may be lossy.
  • 4.1. Effective Channels: From Compute to Usable Evidence: Information recovery is governed by αK: increasing K helps when added channels provide complementary evidence, not merely more outputs.The product αK is identified as the fundamental quantity governing information recovery.
  • 4.1. Effective Channels: From Compute to Usable Evidence: The residual uncertainty satisfies E[H(Y | X, Ẑ1:K)] / H(Y | X) ≤ (1 − α)^K ≤ e^−αK, yielding fast-then-slow gains as K grows.The marginal benefit is largest at small K and decays as effective channels accumulate.
  • 4. Why Diversity Matters: The effective channel count K, rather than raw calls n, governs how much usable evidence a MAS can extract under fixed intrinsic uncertainty and structural ceilings.The central scaling question is whether additional calls increase K(n).
  • 4.2. K as the State Variable of MAS Scaling: The heterogeneity advantage bound compares homogeneous and heterogeneous designs through their effective channel counts and coverage rates under matched compute.The heterogeneous lower bound H(Y | X)(1 − e^−αheterogKheterog) exceeds the corresponding homogeneous guarantee.
  • 4.2. K as the State Variable of MAS Scaling: Heterogeneous configurations recover more task-relevant information under matched compute because they increase αK through more non-redundant channels or higher complementarity.Table 1 isolates persona assignments while sharing the same base model pool.

5. Experiments

Experiments across diverse benchmarks and workflows show that homogeneous scaling saturates quickly, while progressively enriched diversity improves performance, reduces redundancy, and can achieve comparable accuracy with far fewer agents. Effective-channel diversity, especially the combination of model and persona diversity, tracks these gains.

  • Experimental setup: Across seven benchmarks, the experiments test Vote and Debate workflows with agent counts from 2 to 16 under matched compute budgets.The setup spans reasoning, knowledge, deterministic, and ambiguous tasks, using three open-source LLMs and progressively enriched diversity layers.
  • Finding 1: Homogeneous scaling saturates: Homogeneous accuracy improves only at small agent counts, after which marginal gains rapidly collapse and sometimes performance degrades as N increases.The observed saturation is attributed to highly correlated outputs that fail to increase the effective channel count.
  • Finding 2: Diversity beats scale: Model diversity and persona diversity each provide non-trivial gains, while their combination consistently performs best for both Vote and Debate.Performance improves as diversity is enriched from L1 through L4 across the evaluated settings.
  • Finding 2: Diversity beats scale: Heterogeneous configurations reduce redundancy and improve success because they provide more complementary information than homogeneous agents.Higher similarity is associated with lower performance, redundancy increases with agent count, and redundancy decreases monotonically from L1 to L4.
  • Finding 3: Effective channels: K* consistently increases from L1 to L4 under both Vote and Debate, and its increase is accompanied by higher accuracy in most cases.The reported relationship is positive but not universal: an L2 Debate anomaly shows K* increasing while accuracy slightly decreases.
  • Design guidelines: The experiments motivate diversity-aware design guidelines: match diversity to task type, favor diversity of correct reasoning paths, and right-size agent count.The paper reports stronger K*–accuracy prediction on reasoning tasks than on knowledge-heavy tasks and emphasizes correct-path dominance.

6. Conclusion

The paper concludes that raw agent-count scaling has diminishing returns, whereas heterogeneity improves performance by adding diverse, non-redundant information. K* captures effective channels and highlights the importance of balancing correct-path diversity against redundancy.

  • Conclusion: Increasing agent count alone yields diminishing returns, while heterogeneity improves performance by introducing more diverse, non-redundant information.The conclusion applies this contrast to multi-agent scaling broadly within the paper’s supported scope.
  • Conclusion: K* measures effective channels without labels and indicates that gains depend on the balance between correct-path diversity and redundancy.The paper frames effective information allocation, rather than raw computational power, as the central scaling challenge.

Impact Statement

The framework explains MAS scaling through information-theoretic bounds, while identifying idealized assumptions and scope limits for the theory, metric, and experiments.

  • Theoretical Contributions and Scope: The framework’s theoretical analysis assumes sufficient, conditionally independent latent evidence and uniform, independent coverage probabilities.Real-world agents may have more complex dependency structures.
  • Limitations of K*: K* measures semantic diversity in embedding space rather than task-relevant information diversity.Its accuracy correlation varies by task type, and task-adaptive diversity metrics remain an open problem.
  • Empirical Scope: The empirical study evaluates 7B–8B open-weight models across seven benchmarks, focusing on vote and debate mechanisms.Generalization to larger or closed-source models and more complex workflows requires further investigation.

A.2. Finite Information Budget (Upper Bound)

The finite information budget bounds total MAS information by intrinsic task uncertainty, while redundancy determines how quickly additional agent outputs stop contributing new evidence.

  • Finite Information Budget: Total extracted information is bounded by H(Y | X), and each successive incremental contribution Δi converges to zero.This establishes an architecture-independent ceiling on information gained from expanding the transcript.
  • Homogeneous and Heterogeneous Scaling: Homogeneous configurations quickly saturate because repeated samples produce redundant evidence, whereas heterogeneity mitigates overlap and sustains incremental information.The analysis applies to parallel voting and, through the chain rule, to sequential interaction and debate.
  • Redundancy and Saturation: Previous outputs reduce a new output’s incremental information through redundancy, quantified by I(Zi; Z<i | X).Repeated calls with similar reasoning trajectories therefore leave less additional evidence to accumulate.
  • Debate: Debate inherits the same ceiling H(Y | X) as other sequential MAS processes when agents remain redundant.Thus, interaction alone does not remove the finite information budget.
  • Effective Channels: Under the evidence-coverage model, each effective channel independently covers latent evidence with probability α, formalizing complementarity.The model assumes sufficient latent evidence, conditional independence, and coverage that reveals an evidence bit completely.

A.5.3. RESIDUAL CONTRACTION AND SATURATED LOWER BOUND

The residual-contraction analysis predicts rapid early improvement followed by saturation as effective channels accumulate, and defines K* as a label-free embedding-based channel proxy.

  • Residual Contraction: The normalized residual uncertainty is at most (1 − α)^K and therefore at most e^−αK.This yields the characteristic rapid-then-saturating improvement curve as effective channel count increases.
  • Heterogeneity Advantage: Heterogeneity has a strictly higher information-recovery guarantee when αheterogKheterog exceeds αhomogKhomog.The guarantee follows because 1 − e^−t increases strictly with t.
  • Definition of K*: K* is computed from normalized output embeddings through a cosine-similarity Gram matrix and entropy-based spectrum normalization.It is intended as a label-free proxy for effective channels.
  • Properties of K*: K* equals 1 for collinear embeddings and n for orthogonal embeddings with equal norm.The metric is continuous in the embedding matrix and invariant to output permutation.

B.1. Closed-Source Model Experiments

Closed-source experiments on Formal Logic confirm that heterogeneity improves over homogeneous baselines, while the size and scaling pattern of the benefit varies across model families.

  • Heterogeneity Results: Heterogeneity improves over homogeneous baselines for all five evaluated models in at least one interaction mechanism.The reported result indicates that the advantage is not limited to open-source models.
  • Heterogeneity Results: gpt-5-mini improves from 0–6% homogeneous accuracy to 35–55% with heterogeneous prompting, with ΔHet of +39–41%.Models with weaker homogeneous baselines benefit more from heterogeneity than stronger-baseline models.
  • Scaling Trends: gpt-4.1-mini’s heterogeneous debate accuracy drops from 50.79% to 42.86% when scaling from N = 2 to N = 16, giving ΔN = −7.93.This is an example of negative scaling under heterogeneous settings for a strong base model.

B.2. Robustness to Embedding Model Choice

The effective-channel metric K∗ remains robust across embedding models: the models largely agree on relative rankings, and both metrics correlate positively with task accuracy.

  • The comparison spans seven datasets, agent counts N ∈ {2, 4, 8, 12, 16}, and multiple interaction mechanisms.K∗ is recomputed with gte-Qwen2-1.5B-instruct and compared against NV-Embed-v2.
  • ρ = 0.91 average Spearman correlation, with over 95% of matched pairs showing ρ > 0.5 across embedding-model rankings.Robustness is assessed through relative ordering rather than raw K∗ values because embedding dimensionality changes absolute values.
  • Both embeddings produce K∗ metrics that positively correlate with task accuracy.The reported correlations are r = 0.40 for NV-Embed-v2 and r = 0.23 for gte-Qwen2.

B.3. Is K∗More Than a Proxy for Scale and Configuration?

K∗ captures information about output diversity beyond agent count and configuration labels, with correctness-conditioned diversity providing the strongest explanatory signal. Additional ablations show that heterogeneous configurations and model mixtures outperform homogeneous or single-model alternatives.

  • R2 = 0.062 for the baseline using only agent count and configuration labels, indicating limited predictive power.The baseline regression predicts task accuracy from N and configuration labels alone.
  • ∆R2 = +0.147 when K∗ is added, showing that effective channels explain variance beyond scale and configuration.The incremental gain indicates that K∗ captures structural information about output diversity.
  • ∆R2 = +0.331 for correctness-conditioned K∗c, more than doubling the incremental gain from K∗.Adding K∗w changes the gain only from +0.331 to +0.334, indicating negligible additional improvement.
  • Correct-path diversity is more predictive of performance than total channel count or diversity among incorrect responses.The analysis identifies the number of distinct ways agents reach the correct answer as the strongest supported signal.
  • z-scores above 5 with p < 10^-3 show that effective-channel correlations lie far outside permutation-based null distributions.The randomization tests preserve the marginal accuracy distribution while destroying structural association with K∗.
  • At N ≥ 4, heterogeneous model mixtures outperform the best single model, reaching up to +14.28% absolute accuracy at N = 8.The advantage persists under heterogeneous prompting, supporting synergy across models rather than model selection alone.
Loading 2602.03794v1…