PaperlayerView exact revision on arXiv

Source-linked AI summary

When Order Matters: First-Speaker Bias and Mitigation through Personality in Sequential Multi-Agent Debate

Duofeng Xu, Bryan Hooi, Dandan Qiao

arXiv:2609.38964v1cs.AI

TL;DR

Sequential debate may favor earlier speakers, reducing the stronger agent’s influence when it speaks after weaker agents. The paper compares speaking orders and tests Big Five personality prompts in that disadvantaged setting, finding that low agreeableness on the strong agent can restore influence and improve accuracy.

  • Problem

    It is unclear whether sequential debate systematically favors earlier speakers and whether personality-induced behavior can mitigate the resulting disadvantage for a stronger agent speaking last.

    Source: §1 · p. 1

  • Method

    The authors compare mirrored speaking orders, then test agreeableness and extraversion prompts on the strong or weak side in the strong-agent-last setting.

    Source: §1 · p. 1

  • Results

    In W-W-S, low agreeableness on the strong agent raises strong-agent influence by 9.55 percentage points and final accuracy by 1.64 percentage points; extraversion effects are less systematic.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    Debate outcomes depend not only on model capability but also on speaking order and interaction behavior, with low agreeableness on the stronger agent helping restore some lost influence.

    Source: §1 · p. 2, §6 · p. 9

    The study uses a controlled three-agent team with two instances of one weak model, and its personality prompts represent induced behavior rather than stable human-like traits.

    Source: LIMITATIONS · pp. 9–10

Abstract

from arXiv · show

Multi-agent debate (MAD) is often used to improve large language model (LLM) reasoning, but sequential debate is rarely a neutral aggregator of agents' opinions. We show that sequential MAD suffers from a pronounced first-speaker bias: agents disproportionately shape the final answer when they speak first. As a result, placing a stronger model after weaker ones can substantially offset its reasoning advantage. We then focus on the disadvantaged strong-agent-last setting and ask whether personality prompting can mitigate this imbalance. Drawing on the Big Five model, we study agreeableness and extraversion as behavioral interventions applied to either the strong or weak side. We find that their effects are trait-specific. Influence consistently shifts in the direction of lower agreeableness, and assigning low agreeableness to the stronger agent helps restore its lost influence and improves final accuracy. Extraversion, by contrast, produces less systematic changes in influence and accuracy, with its clearest effect appearing in agents' verbosity. These findings show that effective MAD design depends not only on model capability, but also on how speaking order and induced interaction behavior shape the debate process.

1 INTRODUCTION

This paper tests whether sequential debate systematically favors earlier speakers and whether personality prompts can rebalance influence when the stronger agent speaks last. It finds a first-speaker bias and trait-specific intervention effects.

1 INTRODUCTION

  • The first speaker exerts disproportionate influence, and placing the stronger agent after weaker agents sharply reduces its influence and debate accuracy.The authors vary only speaking order while holding models, questions, and protocol fixed.Source: §1 · p. 1
  • The study asks whether speaking order affects final answers and whether agreeableness- or extraversion-based interventions mitigate the strong agent’s disadvantage when speaking last.The authors treat personality prompts as controlled behavioral interventions and focus on traits associated with accommodating or resisting others’ views and expressing one’s own reasoning.Source: §1 · pp. 1–2
  • In the strong-agent-last setting, lower agreeableness shifts influence toward the less agreeable side, while extraversion has less systematic effects on influence and accuracy.The largest accuracy improvement occurs when the strong agent is less agreeable and the weak agents remain unprompted; extraversion’s clearest effect is on justification length.Source: §1 · p. 2
  • The paper argues that model capability, speaking order, and induced interaction behavior jointly shape collective debate outcomes.It frames personality-guided interventions as a way to redistribute influence and, under suitable configurations, improve final reasoning performance.Source: §1 · p. 2

2 RELATED WORK

Prior work shows that debate structure and agent interaction can shape collective reasoning, while personality prompts can alter LLM behavior. The paper addresses open questions about speaking order in sequential debate and whether personality changes influence and performance.

2.1 MULTI-AGENT DEBATE FOR LLM REASONING

  • Prior MAD work studies varied protocols and finds that performance depends on both participating models and how their interaction is organized.Examples include confidence-aware debate, opposing viewpoints, specialized roles, judge-based selection, and sequential task decomposition.Source: §2.1 · p. 2

2.2 INTERACTION DYNAMICS IN LLM DEBATE

  • Debate can involve conformity, copying, premature consensus, and asymmetric influence, but speaking-order effects that alter interaction history remain unsystematically studied.The paper distinguishes changing the order of speakers, which changes subsequent responses, from changing the positions of viewpoints within an update context.Source: §2.2 · p. 3

2.3 PERSONALITY IN LLM AGENTS

  • Although prompted Big Five traits can systematically change LLM behavior, their effects on influence and reasoning performance in sequential debate remain unclear.The authors focus on agreeableness, linked to accommodation, and extraversion, linked to assertiveness and social expressiveness.Source: §2.3 · p. 3

3 EXPERIMENTAL DESIGN

The experiments use controlled mixed-capability debate teams across four multiple-choice benchmarks. They compare mirrored speaking orders, then test personality prompts in the strong-agent-last order.

3.1 TASKS AND EVALUATION DATA

  • Evaluation covers MMLU-Pro, GPQA Diamond, SuperGPQA, and AQuA-RAT, using 200 sampled questions from three benchmarks and all 198 GPQA Diamond questions.The shared multiple-choice format supports consistent answer extraction and comparison.Source: §3.1 · p. 3

3.2 DEBATE PROTOCOL AND AGENT ROLES

  • Each debate uses two instances of one weak model and one relatively stronger model, with sequential responses over four rounds and a final-round majority answer.The three-agent team supports majority voting while holding the weak side as a controlled unit.Source: §3.2 · p. 4

3.3 MODEL COMBINATIONS

  • Teams are formed from seven LLMs across four model families into ten weak–strong combinations, with the strong model more accurate on standalone benchmark questions in every pair.Combinations include within-family and cross-family pairings.Source: §3.3 · p. 4

3.4 SPEAKING ORDER AND PERSONALITY MANIPULATION

  • The order comparison uses W-W-S and S-W-W, while personality interventions test high or low agreeableness or extraversion on either side under W-W-S.The two weak agents stay adjacent and share the same assignment; other trait dimensions are held neutral, and prompts are treated as behavioral configurations rather than stable personalities.Source: §3.4 · pp. 4–5

3.5 EXPERIMENTAL CONFIGURATIONS

  • Configurations vary model combination, speaking order, and personality assignment, while holding protocol, task prompts, and answer extraction fixed across four benchmarks.The design aims to isolate speaking-order and personality effects, with standalone responses collected as accuracy baselines and pre-interaction references.Source: §3.5 · p. 5

4 METRICS AND ANALYTICAL FRAMEWORK

The analysis measures collective accuracy and which side prevails when strong and weak agents initially disagree. Matched comparisons estimate speaking-order and personality effects across benchmark–model cells.

4.1 METRICS

  • Final debate accuracy is the percentage of questions answered correctly by the final-round majority, with no-majority outcomes counted incorrect.The study records each agent’s answers and textual justifications; 99.9% of debates reached a final-round majority.Source: §4.1 · p. 5, §4 · p. 5
  • Influence is measured only when strong and weak standalone answers disagree and the final answer matches one of them; the adopting side’s share defines its influence.Strong- and weak-agent influence sum to 100% by construction; about 45% of questions qualify, using standalone answers under the corresponding personality condition as the reference.Source: §4.1 · pp. 5–6

4.2 ANALYTICAL STRATEGY

  • Matched comparisons aggregate outcomes across 40 model-combination–benchmark cells and test paired percentage-point differences using paired, signed-rank, and sign tests plus a mixed-effects model.Speaking-order comparisons use unprompted S-W-W versus W-W-S; personality comparisons compare prompted W-W-S conditions with matched unprompted W-W-S baselines.Source: §4.2 · p. 6

5 RESULTS

Speaking order shapes both influence and accuracy in sequential debate. In the strong-agent-last setting, low agreeableness on the strong agent improves its influence and accuracy, while extraversion has less systematic effects and most clearly changes verbosity.

5.1 FIRST-SPEAKER BIAS

  • Strong-agent influence is 21.01 percentage points higher in S-W-W than W-W-S, while accuracy rises from 66.43% to 68.78% (2.34 percentage points).The comparison uses the no-personality baselines aggregated across benchmark datasets and model combinations.Source: §5.1 · pp. 6–7, Figure 1 · p. 6, Table 1 · p. 7

5.2 EFFECTS OF AGREEABLENESS

  • With low agreeableness on the strong agent and no prompts on the weak agents, influence rises 9.55 percentage points and accuracy rises 1.64 percentage points versus the W-W-S baseline.Most agreeableness manipulations shift influence toward the side with lower agreeableness; high agreeableness on the strong agent produces the largest reductions in its influence and accuracy.Source: §5.2 · p. 7, Table 2 · p. 7
  • Greater strong-agent influence does not reliably improve accuracy: ah_ah_al reaches 71.43% influence but 67.23% accuracy, versus 68.38% and 68.08% for n_n_al.The authors note that weak agents can contribute useful information, so effective mitigation requires retaining that contribution rather than maximizing strong-agent influence.Source: §5.2 · p. 8

5.3 EFFECTS OF EXTRAVERSION

  • Extraversion shifts strong-agent influence significantly in all four manipulations, but its effects do not follow a consistent high-versus-low pattern.Low extraversion on the weak agents improves accuracy by 0.78 percentage points, less than the 1.64-point gain for n_n_al; the sign test is only marginal (p = 0.0939).Source: §5.3 · pp. 8–9, Table 3 · p. 8
  • High-extraversion prompts generally produce longer justifications than low-extraversion prompts, whereas agreeableness does not systematically change justification length.The justification-length comparison is aggregated across benchmark datasets and model combinations in the W-W-S order.Source: Figure 2 · p. 8, §5.3 · p. 9

6 CONCLUSION

The paper finds that sequential debate’s outcomes depend on speaking order and interaction behavior, not only model capability. Low agreeableness on the strong agent helps restore influence and improve accuracy, while extraversion has less systematic effects.

6 CONCLUSION

  • Speaking earlier gives agents more influence; placing the stronger agent last reduces its influence and final debate accuracy relative to placing it first.The paper identifies this as first-speaker bias in sequential debate.Source: §6 · p. 9
  • Low agreeableness on the stronger agent, with weaker agents unprompted, restores some lost influence and yields the largest accuracy improvement among personality interventions.Further increasing strong-agent influence does not necessarily produce additional performance gains.Source: §6 · p. 9
  • Extraversion produces less systematic influence changes; its clearest and most consistent effect is on how agents express their reasoning.Source: §6 · p. 9

LIMITATIONS

The paper’s limitations concern the scope of its debate setting, model pool, tasks, and personality prompts. The appendices also report single-agent stability under personality prompting and statistically significant order differences.

LIMITATIONS

  • The study uses three agents—two instances of a weak model and one strong model—and does not capture larger teams or distinct models on the weak side.The controlled design supports comparisons of speaking order and influence.Source: LIMITATIONS · p. 9
  • The model pool comprises seven LLMs across four families and ten weak–strong combinations; broader architecture and training-paradigm coverage could strengthen generalizability.Source: LIMITATIONS · p. 9
  • The experiments use multiple-choice reasoning benchmarks, which may not capture interaction dynamics in open-ended writing, planning, or coding tasks.The setting enables clear measurement of final accuracy and answer-level influence.Source: LIMITATIONS · p. 9
  • Personality manipulations are prompt-induced behavioral interventions, not evidence of stable human-like LLM personalities; prompt wording may affect results, and the alternative-prompt check covers one model combination.The alternative-prompt check broadly reproduces the main directional pattern.Source: LIMITATIONS · p. 10

A MODELS AND STANDALONE PERFORMANCE

B STATISTICAL TESTS FOR FIRST-SPEAKER BIAS

C CAPABILITY GAP MODERATION

The analysis tests whether the strong–weak capability gap moderates speaking-order effects. A larger gap is associated with greater strong-agent influence and accuracy under W-W-S, but only the influence disadvantage of speaking last becomes smaller as the gap grows.

C CAPABILITY GAP MODERATION

  • The gap is the difference in no-personality standalone accuracy between the strong and weak models, analyzed by model pair, dataset, and speaking order.It is mean-centered before estimation.Source: C CAPABILITY GAP MODERATION · p. 16
  • Under W-W-S, each additional percentage point of capability gap is associated with a 0.90 percentage point increase in strong-agent influence and a 0.37 percentage point increase in final accuracy.Source: C CAPABILITY GAP MODERATION · p. 16
  • 21.01 percentage points: at the average capability gap, S-W-W increases strong-agent influence relative to W-W-S; the influence advantage of S-W-W shrinks as the gap grows.The interaction is negative and significant (β3 = −0.385, p = .019).Source: C CAPABILITY GAP MODERATION · p. 16, Table 7 · p. 17
  • 2.34 percentage points: at the average capability gap, S-W-W improves final accuracy relative to W-W-S, with no evidence that this advantage changes as the gap increases.The interaction is small and not statistically significant (β3 = 0.053, p = .501).Source: C CAPABILITY GAP MODERATION · pp. 16–17, Table 7 · p. 17

D ADDITIONAL AGREEABLENESS COMBINATIONS

In an additional W-W-S configuration, giving the strong agent low agreeableness and the weak agents high agreeableness further increases strong-agent influence but lowers accuracy. Agreeableness does not consistently change justification length, and an alternative personality prompt broadly reproduces the main directional patterns.

D ADDITIONAL AGREEABLENESS COMBINATIONS

  • 71.43% versus 68.38%: assigning low agreeableness to the strong agent and high agreeableness to weak agents increases strong-agent influence over low-agreeableness strong agent with unprompted weak agents, but accuracy falls from 68.08% to 67.23%.The result indicates that greater strong-agent influence does not translate monotonically into higher accuracy; the paper suggests mitigation requires countering, not maximizing, its positional disadvantage.Source: Table 8 · p. 17, D ADDITIONAL AGREEABLENESS COMBINATIONS · p. 17

E AGREEABLENESS AND JUSTIFICATION LENGTH

F PROMPT TEMPLATES

  • Each agent prompt combines a dataset-specific role, optional personality prompt, task question, and a debate update prompt after the first response.Task wording and answer extraction are fixed within each benchmark; the first-stage variation is speaking order, and the second-stage variation is personality assignment under W-W-S.Source: F PROMPT TEMPLATES · p. 18, §F.7 · p. 20

F.1 DATASET-SPECIFIC ROLE PROMPTS

  • Agents receive dataset-specific expert roles; the multiple-choice question template adapts answer-option counts by benchmark while keeping its wording and answer format fixed.Agents return an answer and brief justification.Source: §F.1 · p. 18, §F.4 · p. 19, §F.6 · p. 20

F.3 PERSONALITY-SPECIFIC INSTRUCTIONS

  • Personality prompts manipulate agreeableness or extraversion while holding other traits neutral, pairing trait levels with behavior-specific instructions relevant to debate.Instructions for agreeableness range from accommodating others’ reasoning to challenging flaws and resisting updates without strong counter-evidence.Source: §F.2 · p. 18, §F.3 · p. 19

F.5 DEBATE UPDATE PROMPT

  • After the first answer, later speakers receive previous responses as context; prompts combine available debate history with fixed task, answer-format, and update components.The first speaker answers independently, the second sees the first response, and the third sees the first two.Source: §F.5 · pp. 19–20, §F.7 · p. 20

G ALTERNATIVE PERSONALITY PROMPTS

  • Under an alternative personality prompt, directional results broadly match the main experiments; n_n_al again has the highest accuracy among the eight prompted configurations, but the check uses one model combination.Lower agreeableness shifts influence toward the manipulated side, while extraversion again lacks a systematic high-versus-low influence pattern.Source: G ALTERNATIVE PERSONALITY PROMPTS · p. 20, Table 9 · p. 21

H PILOT ANALYSIS ON THE NUMBER OF DEBATE ROUNDS

A pilot across 50 GPQA-Diamond questions found that agreement and accuracy gains largely plateau by Round 4, motivating the use of four rounds in the main experiments.

H PILOT ANALYSIS ON THE NUMBER OF DEBATE ROUNDS

  • Agreement and accuracy gains largely plateau by Round 4, so the main experiments use four rounds to balance revision opportunity against computational cost.Unanimous agreement rises sharply in the first rounds; accuracy gains are concentrated between Rounds 1 and 4.Source: H PILOT ANALYSIS ON THE NUMBER OF… · p. 21
  • The pilot averages majority-answer accuracy and unanimous agreement across debate rounds for 50 randomly sampled GPQA-Diamond questions and model combinations.Unanimous agreement is the share of questions where all three agents give the same answer in a round; accuracy is the share where the majority answer is correct.Source: H PILOT ANALYSIS ON THE NUMBER OF… · p. 21, Figure 4 · p. 21
  • Figure 4 plots majority-answer accuracy on the left and inter-agent unanimous agreement rate on the right across debate rounds.Source: Figure 4 · p. 21

I REPRESENTATIVE DEBATE TRACE

A representative Weak-Weak-Strong trace contrasts immediate accommodation under the no-personality baseline with preservation of the strong agent’s correct answer under low agreeableness.

I REPRESENTATIVE DEBATE TRACE

  • In Round 1, the strong agent adopts the weak agents’ answer H under the baseline, but retains and challenges with its independently generated correct answer B under low agreeableness.Table 10 presents this representative first-round trace for SuperGPQA Question 9.Source: I REPRESENTATIVE DEBATE TRACE · p. 22, Table 10 · p. 22
  • Under the baseline, all three agents stay with H, whereas under low agreeableness both weak agents switch to B in Round 2 and retain it thereafter.B is the correct answer; the weak agents identify shaft diameter as the primary criterion for selecting key dimensions.Source: I REPRESENTATIVE DEBATE TRACE · p. 22
  • The trace illustrates how low agreeableness can keep the strong agent’s answer in the discussion long enough for weak agents to adopt it, but it is only one example under W-W-S.Source: I REPRESENTATIVE DEBATE TRACE · p. 22

J PER-DATASET ACCURACY AND INFLUENCE RESULTS

The appendix reports per-dataset final accuracy and strong-agent influence for the speaking-order comparison and, separately, personality interventions restricted to W-W-S.

J PER-DATASET ACCURACY AND INFLUENCE RESULTS

Related papers

Loading 2609.38964v1…