PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models

Leonardo Ferreira, Gardenia Liu, Kaden Zheng

arXiv:2609.35875v1cs.AIcs.MA

TL;DR

Prior work leaves open whether cognitive diversity drives multi-agent debate’s gains over single-agent inference. Across small open-weight models, the paper tests three forms of diversity against matched-budget sampling controls and finds no advantage for debate at equal generation counts. Debate’s gains over single-agent inference therefore do not establish a benefit from deliberation over sampling.

  • Problem

    Prior debate work treats agents as symmetric peers, leaving open whether cognitive diversity drives reported gains over single-agent inference.

    Source: §1 · p. 1

  • Method

    The study varies persona, temperature, and model diversity across small open-weight models and pairs debate configurations with generation-budget-matched majority-vote controls.

    Source: §1 · p. 1, §8 · p. 9

  • Results

    At matched generation counts, debate ties or loses to sampling on every task, despite beating single-agent inference where tasks have headroom.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The results position budget-matched, contamination-checked comparisons as the baseline future debate mechanisms should beat.

    Source: §8 · p. 9

    The findings are established for small open-weight models, and each task cell rests on 100 problems or 40 biography subjects, with per-task significance patterns shifting between seeds.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Multi-agent debate (MAD) reportedly improves reasoning and factuality over single-model inference, but prior work treats agents as symmetric peers, leaving open what drives the gains. We test the hypothesis that cognitive diversity among agents is the driver, in the setting where the question is still measurable: small open-weight models with benchmark headroom. Across 23 models from eleven vendor families, five tasks, and 5,500+ debate and control runs, we vary diversity along three axes - personas, sampling temperature, and model identity - pairing every debate configuration with a generation-budget-matched majority-vote control. The hypothesis is rejected on every axis. Debate beats single-agent inference (3--7 points where tasks have headroom) but at matched budget conditions it ties or even loses to self-consistency sampling at 1.6$\times$ the wall-clock and 3.4$\times$ the token cost. Persona prompting reduces accuracy and a dose-response experiment over each model's full combinatorial persona space shows the cost is a persona tax, not a diversity tax: redundant personas hurt most, while maximally-diverse teams recover part of the loss. Furthermore, mixed-model teams lose to majority votes over their own rosters, with accuracy tracking member capability rather than heterogeneity, and nearly all of debate's benefit comes from the first exchange of answers. We further identify a pervasive measurement hazard in which debate transcripts silently overflow serving context windows, whose correction alone moves our debate-versus-sampling comparison from $-1.8$ points to parity. Our results recast reported MAD gains as an ensemble-sampling effect and provide the budget-matched, contamination-checked baseline bar that future debate mechanisms should be required to clear.

1 INTRODUCTION

The paper tests whether cognitive diversity explains multi-agent debate’s gains, using small open-weight models where benchmark headroom makes the question measurable. It finds that diversity does not explain the gains under matched-budget comparisons.

1 INTRODUCTION

  • The study tests persona, temperature, and model diversity as candidate drivers of debate effectiveness.Persona diversity uses distinct reasoning-style prompts; temperature diversity varies sampling temperatures; model diversity uses distinct base models.Source: §1 · p. 1
  • The evaluation spans 23 small open-weight models from eleven vendor families and more than 5,500 debate and control runs across five tasks.The models retain benchmark headroom, with solo accuracies ranging roughly from 30% to 92%.Source: §1 · pp. 1–2
  • +3.3 to +6.6 points: debate beats single-agent inference where tasks have headroom, but at matched budgets it ties or loses to sampling.The comparison is reported at 1.6× the wall-clock and 3.4× the total token count.Source: §1 · p. 2

2 RELATED WORK

Prior work reports debate gains but has questioned whether they reflect deliberation or sampling, and whether model heterogeneity is especially beneficial. This paper targets several unresolved comparisons with a broad, budget-matched evaluation.

2 RELATED WORK

  • Prior critiques argue that homogeneous debate may reflect sampling effects and often fails to beat matched self-consistency or majority voting.Other work models debate as a martingale over initial answers, while plain sampling-and-voting scales independently.Source: §2 · p. 2
  • Model heterogeneity is among the most consistently reported reasons for debate gains, while scalable-oversight work finds clearer wins under information asymmetry.The sampling literature also develops adaptive stopping, reliability-weighted accumulation, and ranked-ballot aggregation.Source: §2 · p. 2
  • The paper addresses gaps in budget-matched debate-versus-sampling tests, persona selection and dose-response, roster-matched heterogeneous teams, and context-window measurement.Small open-weight models provide benchmark headroom and support thousands of matched controls.Source: §2 · p. 2

3 METHODS

The methods compare iterative debate with matched sampling controls while varying persona, temperature, and model identity. Persona diversity is selected and measured in each participant model’s representation space.

3.1 MULTI-AGENT DEBATE PROTOCOL

  • After agents independently answer in round one, they see other agents’ prior-round responses in later rounds, then answers are aggregated by majority vote or judged independently.Majority vote is used for discrete tasks; generation tasks are judged independently.Source: §3.1 · p. 2

3.2 DIVERSITY CONDITIONS

  • Diversity conditions include homogeneous debate, optimized persona teams, temperature variation, and mixed-model teams, each paired with matched-budget controls.Persona teams use extreme-tier system prompts and are selected by a diversity objective; self-consistency controls use 9, 15, or 21 independent samples with majority voting, matching the generations in three-round debates. Mixed-model teams are stratified by capability and compared with voting over their own rosters.Source: §3.2 · p. 3

3.3 DIVERSITY METRIC

  • Persona diversity is scored by the determinant of the cosine-similarity matrix for normalized persona embeddings in each debating model’s representation space.The team score represents the Gram volume spanned by its personas; exhaustive search selects combinations, and alternative metrics preserve the reported ladder direction.Source: §3.3 · p. 3

4 EXPERIMENTAL SETUP

The experiment uses a diverse pool of open-weight models and five tasks, with fixed paired problem sets and bootstrap uncertainty estimates. The task suite covers mathematical and knowledge questions as well as factual biography generation.

4.1 MODELS

  • The evaluation includes 23 open-weight models across eleven vendor families and three architecture types, spanning 0.6B–109B total parameters.Model-diversity teams draw from the 19 models available when team sampling was frozen; four later additions join single-model conditions.Source: §4.1 · p. 4
  • Biography outputs are evaluated by three independent judges using claim-level supported, contradicted, or unverifiable labels against reference articles.A supplementary judge shares a vendor with participants and is reported separately to measure the judge–participant-coupling concern.Source: §4.1 · p. 4

4.2 TASKS

  • Five tasks cover arithmetic, grade-school math, academic multiple choice, commonsense questions, and factual biographies.Math tasks use exact numerical matching with 10^-4 tolerance; biography evaluation reports claim precision and a complementary reference-fact recall direction.Source: §4.2 · p. 4
  • All conditions answer the same fixed problem sets, enabling paired comparisons, with bootstrap confidence intervals estimated according to the sampled unit.Debate-versus-control deltas are paired by problem, team-sampled comparisons are clustered over teams, and correlations are reported within model strata.Source: §4.2 · p. 4

5 RESULTS

Across five results subsections, debate’s gains over solo inference do not translate into an advantage over budget-matched sampling; persona prompts reduce accuracy, mixed-model debate trails voting over the same rosters, and context-window overflow can distort comparisons.

5.1 MAIN RESULTS: DEBATE HELPS ONLY AS MUCH AS ITS COMPUTE BUDGET

  • +3.3pp on GSM8K, +6.6pp on MMLU, and +5.7pp on CSQA versus Solo; against SC-9, debate is indistinguishable on math, GSM8K, and MMLU, while sampling wins CSQA by 2.3pp.At fixed budget, debate costs 1.6× the wall-clock and 3.4× the total tokens.Source: §5.1 · pp. 4–5, Table 1 · p. 5

5.2 PERSONAS TAX SMALL MODELS AND DIVERSITY IS NOT THE CULPRIT

  • Maximally diverse persona teams reduce accuracy versus plain debate, while the dose-response attributes the larger penalty to having personas, especially redundant ones.From the most redundant to the most diverse rung, accuracy is flat to mildly increasing; within-model answer disagreement remains unchanged across rungs.Source: §5.2 · pp. 5–6, Figure 2 · p. 6
  • Temperature spread changes self-consistency accuracy by −0.11 to −0.41pp across 92 model-task cells, with all p ≥0.34.No task moves more than 1.5pp, so this tested temperature variation does not account for the debate-versus-ensemble comparisons.Source: §5.2 · p. 6

5.3 HETEROGENEOUS-TEAM DEBATE AT A MATCHED COMPUTE BUDGET

  • Mixed-model debate loses to majority-vote ensembles built from the same rosters by 1.04pp (95% CI [−1.61, −0.52]) across 320 team-task cells.Team accuracy follows member capability rather than heterogeneity; matched ensembles also reproduce and exceed the rescue of weaker teams.Source: §5.3 · pp. 6–7, Table 2 · p. 7

5.4 BIOGRAPHY GENERATION

  • On biography generation, debate reaches 55.6% claim precision versus 54.6% for solo generation, while nine independently sampled biographies reach 64.6%.Persona teams reach 51.8% claim precision; the sampling comparison uses the same vendor-balanced judge panel.Source: §5.4 · p. 7

5.5 CONTEXT-WINDOW CEILINGS

  • Context-window overflow can silently produce empty final answers; correcting affected runs shifts the debate-versus-sampling comparison from −1.84pp to −0.06pp.All headline numbers use context-corrected runs.Source: §5.5 · p. 8

6 DISCUSSION

The discussion attributes persona penalties to the cost of adopting cognitive frames, not to excessive diversity, and finds that voting captures the benefits of model heterogeneity without a debate advantage.

6 DISCUSSION

  • Most redundant persona teams hurt more than maximally distinct teams on every task, while increasing diversity weakly improves accuracy.Communication breakdown between mutually alien personas therefore cannot explain the largest penalty.Source: §6 · p. 8
  • Persona prompting imposes a capacity tax that is largest on knowledge retrieval and shrinks as model capability rises.The penalty is CSQA −7.3pp at rung 0; its association with capability is Spearman +0.46, p = 0.027.Source: §6 · p. 8
  • Redundant personas add correlated bias, while diverse personas can decorrelate damage; the penalty persists and deepens on knowledge tasks with five or seven agents.The embedding-space account is tentative: its association is confounded with capability in a 23-model sample.Source: §6 · p. 8
  • Model diversity is harmless but not helpful: voting captures mixed-roster benefits, while deliberation layered on top subtracts slightly.The authors leave open whether larger models can turn diversity into deliberative gains.Source: §6 · p. 8

7 LIMITATIONS AND FUTURE WORK

The authors limit their conclusions to small open-weight models and identify constraints in the persona intervention, evaluation, and seed-level evidence.

7 LIMITATIONS AND FUTURE WORK

  • Findings are established for small open-weight models; larger models may adopt unconventional frames without the tax, but Llama-4-Scout is only one-model evidence.Its maximally diverse teams beat matched sampling on MMLU (87.0 vs. 83.0) and CSQA (91.0 vs. 89.0); budget accounting could differ for frontier models.Source: §7 · pp. 8–9
  • Seed sensitivity remains: task-level significance patterns shift between seeds, and each cell rests on 100 problems (40 biography subjects).The headline mixed-model deficit replicates at a second problem seed (−0.93 vs. −1.04pp); the homogeneous grid’s second-seed slice correlates 0.978 with the first.Source: §7 · p. 9
  • The persona intervention is prompt-based, with determinant-selected rungs; task-aligned personas, fine-tuned styles, and tool or information asymmetries remain untested.These alternatives could produce cognitive diversity that prompting cannot.Source: §7 · p. 9
  • Structured single-agent methods, objectively verifiable domains such as code generation, and domains valuing novelty or synthesis remain untested.The authors identify tree-of-thoughts search and iterative self-refinement as mechanisms for future budget-matched comparisons.Source: §7 · p. 9
  • Biography conclusions inherit judges’ disagreement because human claim-level annotation was unavailable.The authors expose disagreement through vendor-balanced scoring, per-judge splits, and κ matrices rather than resolving it.Source: §7 · p. 9
  • Future work should learn team composition rather than fix it, though integrating matched provenance and architecture comparisons requires resampling the team space.Source: §7 · p. 9

8 CONCLUSION

Across the tested settings, cognitive diversity does not explain debate’s gains: those gains largely arrive with the first exchange and do not survive budget matching. The conclusion also identifies transcript overflow as a measurement hazard and releases reproducibility materials.

8 CONCLUSION

  • Across 23 open-weight models, five tasks, and 5,500+ runs, the cognitive-diversity hypothesis fails on every tested axis.Persona prompting reduces accuracy, while temperature diversity changes nothing and mixed-model teams gain no benefit beyond majority voting.Source: §8 · p. 9
  • Debate’s gains over single-agent inference are an ensemble-sampling effect: nearly all arrive with the first exchange, and none survives budget matching despite 1.6× wall-clock and 3.4× token cost.Source: §8 · p. 9
  • Transcript overflow can manufacture an apparent debate deficit; correcting it moves the comparison to parity, while voting over independent samples matches or exceeds debate accuracy without exchange.Debate is net-corrective, with helpful flips outnumbering harmful ones roughly two to one (7.6% vs. 4.0% of agent–problem pairs).Source: §8 · p. 9
  • The conclusion favors retaining serial iteration while dropping inter-agent transcripts, and offers the budget-matched, contamination-checked grid as a baseline for future debate mechanisms.Source: §8 · p. 9

REPRODUCIBILITY STATEMENT

  • The authors release per-file scored outputs, judge-verdict tables, persona ladders, and a fleet manifest, with published teams reproducible by re-enumerating each model’s persona space.Figures and tables are generated by deterministic scripts with fixed bootstrap seeds.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The paper reports human-directed use of AI for research infrastructure, describes its model pool and experimental design, and provides per-model accuracy tables for the core conditions.

AI USE STATEMENT

  • A Claude agent reviewed analysis code and handled campaign orchestration, scheduling, failure diagnosis, recovery, and data verification under continuous human direction; the authors retain responsibility for research decisions and interpretations.Source: AI USE STATEMENT · p. 10

A THE MODEL POOL

  • Table 3 lists the 23-model participant pool and lineage classes; four models joined after mixed-model team sampling was frozen and appear only in single-model conditions.Source: Table 3 · p. 12

B EXPERIMENTAL DESIGN AND INFRASTRUCTURE

C PER-MODEL RESULTS

D MIXED-MODEL ARMS BY TASK

Mixed-model results vary by task and team composition, with voting often matching or outperforming debate. Team capability tracks accuracy, and most gains from additional debate rounds occur at the first exchange.

D MIXED-MODEL ARMS BY TASK

  • On math, GSM8K, and CSQA, debate is negative where significant; on MMLU, weak-duplicated quads significantly beat matched sampling by +2.80 pp.Source: Table 10 · p. 16, D MIXED-MODEL ARMS BY TASK · p. 16
  • Accuracy-weighted voting widens the ensemble lead over debate from −1.04 to −1.35pp on core triples and from −2.16 to −3.40pp on pairs; multiclass log-odds weighting also beats debate on triples and quads.The weighted-voting comparison uses the same stored generations; seed-43 replication reproduces the widened lead.Source: Table 11 · p. 16, D MIXED-MODEL ARMS BY TASK · p. 16
  • Accuracy is monotone with team capability composition in both debate and ensemble arms; significant debate deficits concentrate in mid-to-weak teams, while all-strong teams show a +0.44 edge for debate.Source: D MIXED-MODEL ARMS BY TASK · p. 16, Table 12 · p. 17
  • Nearly all debate-round gains arrive by r=2; rounds 3–5 are flat, and CSQA declines beyond r=2.The all-weak composition rises from 61.5 to 74.2 after the first exchange, then erodes; round columns are not budget-matched.Source: D MIXED-MODEL ARMS BY TASK · p. 16, Table 13 · p. 17

E BIOGRAPHY EVALUATION

Biography evaluation shows that sampling’s margin over solo is driven mostly by multiplicity, while judge-vendor differences qualify interpretation. Context-window overflow can also depress debate accuracy as empty final rounds appear.

E BIOGRAPHY EVALUATION

  • The biography panel balanced the two near-duplicate OpenAI judges into one vote before averaging with Gemma.Their per-cell persona effects correlated at r=0.95, so a plain three-judge mean would have weighted OpenAI two-to-one over Google.Source: E BIOGRAPHY EVALUATION · p. 17, Table 14 · p. 18
  • The sampling control’s +10.0pp margin over solo split into +2.0pp from temperature spread and +8.0pp from multiplicity; the debate comparison pooled three biographies per subject.The passage identifies the debate comparison as the robust one.Source: E BIOGRAPHY EVALUATION · p. 17
  • Claim precision assessed only written claims, so declining unknown subjects could flatter honest abstainers over confabulators.The LFM family declined up to 10 of 40 subjects in some cells.Source: E BIOGRAPHY EVALUATION · p. 17

F THE CONTEXT-WINDOW CURVE

  • The Qwen3.5 thinkers’ accuracy fell by half at 16K and recovered monotonically to 131K as empty final rounds vanished; models fitting their windows stayed flat.The passage attributes the change to a serving artifact, not a change in ability.Source: F THE CONTEXT-WINDOW CURVE · p. 19, Figure 3 · p. 19

G BENCHMARK SLICES: MMLU-PRO, GSM-SYMBOLIC, AND GPQA DIAMOND

Across the benchmark slices, debate’s gains over solo are shared by sampling, and a seeming GPQA advantage disappears after correcting a budget cap. The remaining sections document costs, protocol choices, robustness checks, and practical implications.

G BENCHMARK SLICES: MMLU-PRO, GSM-SYMBOLIC, AND GPQA DIAMOND

  • MMLU-Pro scores were about 15 points below plain MMLU across all 23 models, while GSM-Symbolic results stayed within a point of GSM8K and debate and sampling were at parity (+0.35pp).GSM-Symbolic perturbs GSM8K surface forms; the authors report that contamination does not appear to drive the results.Source: G BENCHMARK SLICES: MMLU-PRO,… · p. 19, Table 17 · p. 20
  • After full-budget reruns, solo scored 49.1, debate 54.3, and SC-9 54.9 on GPQA Diamond; debate −SC-9 = −0.6pp [−2.2, +1.0].The earlier seven-model pilot lead was four points; the scaled 23-model lead was +3.2pp and not statistically significant before the budget correction.Source: G BENCHMARK SLICES: MMLU-PRO,… · pp. 19–20
  • Debate still beat solo by +5.2pp, but sampling did too; sampling edged debate on MMLU-Pro by 2.0pp across all 23 models.The weak-duplicated-quad MMLU cell remained the sole positive result for deliberation.Source: G BENCHMARK SLICES: MMLU-PRO,… · p. 20

H WALL-CLOCK AND TOKEN COST

  • Debate re-read growing peer transcripts each round, making prompt-side consumption roughly ten times the control’s and its total token bill 2.3–4.1× across tasks, while accuracy tied or lagged.Persona debate added a further 1.26× wall-clock over plain debate.Source: H WALL-CLOCK AND TOKEN COST · p. 21, Table 18 · p. 21

I PROTOCOL DETAILS

  • Debate rounds prepend peers’ full previous responses, while comparisons reused uniformly sampled problems drawn with fixed seed 42 (seed 43 for the replication slice).The single-agent n=1 arm instead receives a double-check prompt; majority-vote ties use the first-encountered parseable answer.Source: I PROTOCOL DETAILS · p. 21

J DIVERSITY-METRIC ROBUSTNESS

  • Across all 23 models, the most-redundant teams remained most redundant and MaxDet teams remained near-optimal when re-ranked by MaxMin distance.The table reports combo-space and ladder rank correlations plus MaxMin percentiles for rung-0 and MaxDet teams.Source: J DIVERSITY-METRIC ROBUSTNESS · p. 21, Table 19 · p. 22

K PRACTICAL IMPLICATIONS

  • At fixed budgets, practitioners are advised to prefer sampling-plus-voting, pool different small models and vote, and avoid personas; richer debate mechanisms must clear this reproducible baseline.The authors also recommend two debate rounds, sizing context for agents × rounds × per-turn budget, and vendor-balanced panels for long-form evaluation.Source: K PRACTICAL IMPLICATIONS · p. 22

L THE PERSONA SET

  • The complete candidate pool contained 50 extreme personas, each injected verbatim as the agent’s system prompt; persona teams were selected from that tier.The paper also describes a moderate tier of 50 personas used for early calibration only.Source: L THE PERSONA SET · p. 22
Loading 2609.35875v1…