PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

Carolina Fortuna, Blaz Bertalanic

arXiv:2609.31563v1cs.AIcs.MA

TL;DR

Multi-agent systems are often expected to improve as teams grow, but the paper frames scaling as dependent on task structure. It applies Steiner’s taxonomy and finds distinct patterns for disjunctive and Fermi tasks, including gains constrained by modal answers and shared item-level bias.

  • Problem

    Multi-agent LLM systems are often expected to improve as team size increases, but the scaling behavior may depend on task structure.

    Source: §1 · p. 1

  • Method

    The paper applies Steiner’s taxonomy to LLM benchmarks and formalizes independently sampled agents’ behavior, including large-team limits across task types.

    Source: §1 · p. 2, §2.4 · p. 4

  • Results

    On disjunctive tasks, pass@N grows with team size while plurality converges to the model’s modal answer; on Fermi estimation, item-level bias explains 87% of squared log-error, limiting averaging’s error reduction to 6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For disjunctive tasks, the pass@N–plurality gap identifies process loss for verifiers, reward models, or structured deliberation to recover; reducing Fermi bias requires changing estimates, not merely adding samples.

    Source: §4 · p. 9

    Fermi findings rest on one benchmark with substantial label noise, and all tested models are open-weight with at most 20B parameters.

    Source: §4 · p. 9

Abstract

from arXiv · show

Multi-agent LLM systems are often expected to improve as team size increases, yet the scaling behavior may depend on task structure. Our central contribution is to introduce Steiner's taxonomy of group tasks as a framework for analyzing multi-agent LLM scaling and focusing the analysis on disjunctive and compensatory tasks. We model independently sampled agents as conditionally independent given the item, which yields their large-team limits: plurality voting converges to the model's modal answer, and averaging converges to the model's item-level bias. Across selected representative benchmarks, 13 open-weight models, and teams of up to 30 agents, we find qualitatively different scaling behavior. On disjunctive tasks, the probability that at least one agent is correct grows by 5-20 points with team size, but plurality voting over agents that answer directly realises almost none of this potential, as the model predicts to within 0.5 points on average. Multi-round revision raises accuracy considerably, yet the gain is nearly the same with one peer as with 29. In contrast, scaling provides little benefit on Fermi estimation, despite its natural suitability for aggregation: item-level biases shared across the samples of a model account for about 87% of the squared error, so averaging reduces error by only about 6%. Combining model families helps on Fermi estimation but does not surpass the strongest member on disjunctive tasks. These results show that task structure, together with the mechanism combining member outputs, is a fundamental determinant of team scaling.

1 Introduction

The paper introduces Steiner’s task taxonomy as a framework for analyzing multi-agent LLM scaling. Across disjunctive and compensatory tasks, the reported benefits depend on how member outputs are combined and on shared item-level behavior.

1 Introduction

  • Steiner’s taxonomy links collective success or process loss to task structure, and this paper applies it to LLM scaling while formalizing agent behavior and large-team limits.Source: §1 · pp. 1–2
  • Prior work reports sampling, voting, and debate gains on disjunctive benchmarks, with gains sometimes saturating or reversing as calls grow (Wang et al., 2023; Du et al., 2024).Improvements have also been reported for Fermi and guesstimate tasks (Kalyan et al., 2021; Liu et al., 2025; Chuang et al., 2025).Source: §1 · p. 2
  • Earlier scaling analyses found only heterogeneous teams escaping hard ceilings, while homogeneous teams saturated (Bertalanič & Fortuna, 2026; Yang et al., 2026).Source: §1 · p. 2
  • On disjunctive tasks, pass@N grows with team size, but plurality voting converges to the model’s modal answer; revision gains are nearly unchanged from one peer to 29.The conditional-independence model predicts plurality voting to within 0.5 points.Source: §1 · p. 2
  • About 87% of squared log-error is item-level bias on compensatory Fermi tasks, so averaging reduces error by only about 6%; implausible reference labels explain part of cross-model correlation.Source: §1 · p. 2
  • Heterogeneous teams improve over their average member, but not the strongest member on disjunctive tasks; five 7B–8B models outperform their members on Fermi estimation.Source: §1 · p. 2

2 Taxonomy and Theoretical Scaling Framework

The section uses Steiner’s taxonomy to frame multi-agent scaling by how a task combines members’ contributions. Its theory distinguishes disjunctive success, which can approach the share of items with a correct modal answer, from compensatory aggregation, whose error reduction is limited by shared bias and correlated errors.

2.1 Steiner’s Task Taxonomy and Modern LLM Benchmarks

  • Steiner task categories describe how a system combines member contributions, not an intrinsic property of a benchmark; this section focuses on disjunctive and compensatory tasks.The taxonomy table categorizes tasks according to Steiner’s framework.Source: §2.1 · p. 2, Table 1 · p. 3

2.2.1 The Pure Disjunctive Case

  • With an ideal verifier and independent agents, disjunctive success is 1 − (1 − p)^N, the probability that at least one agent answers correctly.The paper calls the empirically estimated share of items with at least one correct agent pass@N.Source: §2.2.1 · p. 3

2.2.2 Majority Voting and the Condorcet Jury Theorem

  • For uncorrelated agents with individual accuracy above 0.5, majority-vote error is bounded by exponential suppression as team size increases.Without an oracle verifier, multi-agent systems aggregate discrete candidate answers through majority or plurality voting.Source: §2.2.2 · pp. 3–4

2.3 Modeling Compensatory Tasks through Geometric Aggregation

  • For continuous estimates, the team aggregates members’ values geometrically, equivalent to averaging their signed log-errors.The paper evaluates these estimation errors in logarithmic space.Source: §2.3 · p. 4

2.3.1 The Statistical Principle of Team Superiority

  • When estimates are unbiased and independent, averaging reduces error variance to σ^2/N, so the team’s standard error shrinks with team size.These benefits rely on zero-mean individual errors and uncorrelated judgments.Source: §2.3.1 · p. 4

2.4 Effective Team Capacity and Scaling Limits Across Task Types

  • For independently sampled agents from one model, outputs are independent conditional on the item, but item difficulty induces correctness correlation across items.The paper interprets this correlation as heterogeneity in item difficulty, rather than interaction between agents.Source: §2.4 · p. 4
  • Under conditional independence, disjunctive accuracy approaches the share of items with a correct modal answer, while compensatory error reduction remains limited by item-level bias and effective team capacity.For positive error correlation, effective capacity is bounded by 1/ρ.Source: §2.4 · pp. 4–5

3 Evaluation of Disjunctive vs. Compensatory Scaling

Across 13 open-weight models, scaling differs by task: disjunctive pass@N grows while direct plurality saturates, and RealFP averaging yields little improvement. The section traces these limits to answer concentration and shared item-level estimation bias, while documenting gains from deliberation and heterogeneous teams.

3.1 Experimental Setup

  • Evaluation covers 13 open-weight models, team sizes up to 30, and one to three communication rounds, with agents answering independently before seeing peers’ responses in later rounds.The experiments compare homogeneous and heterogeneous teams.Source: §3.1 · p. 5

3.2 Empirical Scaling Dynamics on Disjunctive vs Compensatory Tasks

  • On disjunctive tasks, pass@30 reaches 54.4% on GSM8K (vs 34.3% single-agent), while RealFP within-order accuracy reaches just 33.4% (vs 31.6%) under averaging.Accuracy on RealFP means an estimate within one order of magnitude.Source: Figure 1 · p. 6, §3.2 · p. 6
  • Even on 385 label-clean RealFP items, averaging raises accuracy only from 40.9% at N=1 to 43.6% at N=30, while pass@30 reaches 70.8%.The gap between potential and realized Fermi-team performance therefore remains on the label-clean subset.Source: §3.2 · p. 6

3.3 The Anatomy of the Disjunctive Ceiling

  • Round-1 plurality gains only +0.3 points on GSM8K from N=2 to 30; the model predicts observed accuracy within 0.48 points on average.Plurality voting converges to the model’s modal answer for each item.Source: §3.3 · p. 7
  • Round-3 accuracy peaks at intermediate team sizes, then declines 0.4–1.1 points toward N=30 on mathematical and knowledge tasks.The largest declines occur on multi-step symbolic tasks, where an incorrect intermediate result shared by peers can act as an attractor during revision.Source: §3.3 · p. 7

3.4 The Anatomy of the Compensatory Ceiling

  • Item-level bias accounts for 87% of mean squared log-error, so even an infinite homogeneous team could remove at most about 13%.Independent samples cluster around each model’s item-specific estimate, leaving its systematic error untouched by averaging.Source: §3.4 · pp. 8–9
  • Models with near-zero global mean error can still be biased on individual items, where their independent samples cluster around an item-specific estimate.The root-mean-square item-level bias averages 2.01 orders of magnitude.Source: §3.4 · pp. 8–9
  • A heterogeneous 7B–8B pool reduces average-member error by 25.8% and outperforms its best member alone, consistent with partly unaligned model biases.Its error is 1.45, compared with 1.80 for the best member.Source: §3.4 · pp. 8–9, Table 3 · p. 9

4 Discussion

The discussion identifies different routes to improvement for disjunctive and compensatory tasks, and lists limits on the experiments’ prompting regime, benchmarks, and model scope.

4 Discussion

  • On disjunctive tasks, the gap between pass@N and plurality accuracy identifies process loss that verifiers, reward models, or structured deliberation could recover.Reasoning before answering or combining model families can raise the voting ceiling, within the bound given by Eq. 3.Source: §4 · p. 9
  • On compensatory tasks, reducing item-level bias requires changing estimates, for example by combining models with unaligned biases, rather than adding samples.The paper evaluates prompting interventions and a fine-tuned model in Appendix F.Source: §4 · p. 9
  • The Fermi findings rest on one benchmark with substantial label noise and two prompt families.The study also uses answer-first prompts, open-weight models of at most 20B parameters, and leaves additive, conjunctive, and discretionary tasks unstudied.Source: §4 · p. 9

Use of Large Language Models

The authors describe generative AI assistance across methodological feedback, analysis, and writing, while stating that they manually reviewed the work and recomputed reported numbers.

Use of Large Language Models

  • Generative AI tools provided simulated-review feedback on methodology and helped refine theoretical analysis, including the voting-gain bound, error decomposition, and parts of proofs.Source: Use of Large Language Models · p. 10
  • The tools also assisted with analysis code, bootstrap intervals, equation validation, label checks, and sensitivity analyses.Source: Use of Large Language Models · p. 10
  • The authors manually revised the text and references, reviewed AI-generated code, and recomputed all reported numbers from raw generation logs.Source: Use of Large Language Models · p. 10

A Detailed Steiner Taxonomy Mapping on all 5 task categories

The appendix maps five Steiner task types by how member contributions combine, then contrasts ideal scaling in disjunctive selection and compensatory aggregation under independence and unbiasedness.

A Detailed Steiner Taxonomy Mapping on all 5 task categories

B.1 The Scaling Dichotomy Under Ideal Assumptions

  • Under ideal independence and unbiasedness, both disjunctive and compensatory tasks suppress large-error probability exponentially, but respond differently to correlation and bias.For compensatory tasks, the standard error falls with team size, and sub-Gaussian errors give an exponentially decreasing probability of exceeding a fixed tolerance.Source: §B.1 · p. 15

B.2 Correlated and Biased LLM Ensembles: Breakdown Across Task Types

The classical models assume independent and unbiased members, but samples from one LLM violate both assumptions at the item level. The consequences differ across Steiner’s task classes.

  • The classical models in Sections 2.2 and 2.3 assume independent and unbiased members.Source: §B.2 · p. 15
  • Ensembles of samples from the same LLM violate both assumptions at the level of individual items.Source: §B.2 · p. 15
  • The consequences of these violations differ across Steiner’s task classes.Source: §B.2 · p. 15

B.2.1 Disjunctive Tasks Under Correlation and Bias

For same-model samples, item difficulty creates correlated correctness, and plurality voting converges to whether the correct answer is modal. Concentrated probability on one wrong answer can reinforce it, while verifiable rationales offer a route for deliberation.

  • With positive pairwise correlation, collective vote-count variance is inflated by 1+(N −1)ρ, corresponding to effective team size Neff(N) = N/(1 + (N −1)ρ).Source: §B.2.1 · p. 15
  • Plurality accuracy converges to the fraction of items on which the correct answer is modal.For independent samples from one model, correctness correlation arises from heterogeneous item difficulty.Source: §B.2.1 · p. 15
  • When incorrect answers are dispersed but correct derivations converge, the correct answer can remain plurality even when pk < 1/2; concentrated probability on one wrong answer instead reinforces it.Measurements report intra-class correlations between 0.53 and 0.97.Source: §B.2.1 · pp. 15–16
  • Because mathematical reasoning steps can be checked, an agent can present a correct step that peers are able to verify.Source: §B.2.1 · p. 16

B.2.2 Compensatory Tasks Under Correlation and Bias

On compensatory estimation tasks, shared item-level bias and correlated errors limit what averaging or revision can achieve. The proof section states the large-team arguments for oracle accuracy, plurality voting, attainable voting gain, and compensatory error.

B.2.2 Compensatory Tasks Under Correlation and Bias

  • Averaging samples of one model cannot reduce item-level bias, and expected team error is invariant to ensemble size under the stated bias condition.Source: §B.2.2 · p. 16
  • With equicorrelated errors, the variance of the average approaches a non-zero floor.Source: §B.2.2 · p. 16
  • Signed log-error intra-class correlation is 0.70–0.95, while revision leaves Fermi accuracy essentially unchanged, between −0.6 and +0.7 points.Source: §B.2.2 · p. 16

B.3 Proofs

  • For oracle accuracy, independent outputs imply probability (1 −pk)^N that none is correct, yielding the large-team limit Prk(pk > 0).Source: §B.3 · p. 16
  • For plurality voting, the proof uses itemwise comparisons with competing answers to establish convergence to the stated limit.Source: §B.3 · p. 16
  • For binary outcomes, the attainable voting gain is bounded using item-level success probabilities and their variance across items.Source: §B.3 · p. 16
  • The compensatory decomposition separates common item bias from sampling noise; equicorrelated errors yield variance σ2(1 + (N −1)ρ)/N.Source: §B.3 · p. 16

C Experimental Details: Models, Prompts, and Data Integrity

The appendix specifies the experimental protocol, model and prompt setup, and data-integrity checks. It also presents randomly selected traces illustrating deliberation mechanisms and shared bias in revision.

C Experimental Details: Models, Prompts, and Data Integrity

C.1 Example Traces

  • Example traces were randomly drawn using fixed-seed selection rules, with outputs abbreviated but otherwise reproduced verbatim.Source: §C.1 · p. 18
  • In the disjunctive trace, one agent adopts a peer’s correct argument while the other revises its answer in light of its own reasoning.The selected case begins with both agents incorrect and ends with a correct plurality answer in Round 3.Source: §C.1 · p. 18
  • In the compensatory trace, agents share an item-level bias, and revision moves the estimate closest to the reference toward the majority, increasing team error.Source: §C.1 · p. 18

D Additional Scaling and Disjunctive Results

Additional analyses test the disjunctive scaling model and voting-gain bound, examine how sampling temperature affects gains, and report how deliberation peaks are measured.

Prediction of plurality accuracy

  • Across 650 configurations, resampled plurality accuracy differs from observed accuracy by 0.48 points on average, validating the conditional-independence prediction.The largest deviations occur on MATH-500, where equivalent answers written in different forms are merged before voting but the estimated distributions use answer strings.Source: D Additional Scaling and Disjunctive… · p. 20, Figure 2 · p. 20

Validation of the voting-gain bound

Sampling temperature on disjunctive tasks

  • As temperature rises from 0.2 to 1.0, pass@30 increases by 9.3, 8.3, and 14.1 points across three tasks, while plurality accuracy remains within 2.3 points of single-agent accuracy.The added answer diversity falls on incorrect answers, leaving most items’ modal answer unchanged; process loss grows with temperature.Source: D Additional Scaling and Disjunctive… · p. 21, Table 5 · p. 21

Statistics of the deliberation peak

  • Table 6 reports the optimal Round-3 team size, peak accuracy, accuracy at N = 30, post-peak decay, and net deliberation gain, with paired bootstrap intervals.Because optimal team size is selected as the maximum, the reported decay is biased downward.Source: D Additional Scaling and Disjunctive… · p. 19, Table 6 · p. 21

E Additional Compensatory Results

Additional compensatory analyses validate the bias-and-variance error decomposition, test its sensitivity to clipping and temperature, and examine interventions intended to reduce shared bias or diversify errors.

E Additional Compensatory Results

  • For ten models, predicted and observed mean squared error differ by 0.6% on average, and the curves approach the bias floor within five to ten agents.The maximum relative difference is 3.0% across 90 out-of-sample points; only N = 30 points are in-sample.Source: E Additional Compensatory Results · p. 22, Figure 4 · p. 22
  • Across clipping thresholds, averaging reduces error by only 4–8%, while item-level bias accounts for 86–94% of mean squared error.Without clipping, a small number of extreme items dominate the mean; clipping therefore makes the reported ceiling conservative rather than creating it.Source: E Additional Compensatory Results · p. 22
  • Raising temperature increases within-item estimate dispersion by 30–70%, but accuracy differences remain within roughly ±4-point intervals and aggregates do not improve.The intra-class correlation remains between 0.83 and 0.98 as unanimous teams fall from 33% to 4%.Source: E Additional Compensatory Results · p. 23, Table 8 · p. 23
  • Across the ten models, mean off-diagonal correlation of item-level biases is 0.63, indicating substantial cross-model bias similarity.Source: Table 10 · p. 24

F De-Saturating Compensatory LLMs: Interventions and Specialist Models

  • The intervention battery targets correlated errors and systematic bias using heterogeneous models, distinct decomposition strategies, estimation templates, and log-space calculation or grounding approaches.The five-model heterogeneous team uses one run at T = 0.4 on 529 RealFP items.Source: F De-Saturating Compensatory LLMs:… · p. 24, §F.1 · pp. 24–25

F.1 The Heterogeneous Intervention Battery

  • Structured scaffolding reaches 36.3% accuracy at N = 5 versus 33.1% for the baseline, but their intervals overlap; every logcalc variant lowers accuracy.The logcalc and grounding conditions cover only 180 and 57 items, respectively, so they are not comparable with the other cells.Source: §F.1 · p. 25, Table 11 · p. 25

F.2 Evaluation of a Specialist Model

The specialist-model comparison finds no clear general accuracy advantage for fine-tuning; the observed gain is limited to scaffold prompting on SciOly.

F.2 Evaluation of a Specialist Model

  • At the shared 16k-token budget, the fine-tuned model was not more accurate on RealFP with PoT (+0.4 points, 95% CI [−3.6, 4.4]) or scaffold (+0.4, [−3.9, 4.9]), or SciOly with PoT (+0.7, [−2.6, 3.7]).Source: §F.2 · p. 25
  • With scaffold prompting on SciOly, the fine-tuned model is 5.7 points more accurate than its base model, with a 95% interval of [0.7, 10.4].Source: §F.2 · p. 25
  • The paired comparisons use only items both models answered under the same 16k-token budget, because lower-budget controls were truncated on about half the items.Source: §F.2 · p. 25
  • PoT yields the highest accuracy for both models: about 50% within one order of magnitude on RealFP and 76% on SciOly.Source: §F.2 · p. 25
Loading 2609.31563v1…