When Identity Skews Debate: Anonymization for Bias-Reduced Multi-Agent Reasoning

1 Introduction

• The paper identifies a gap in systematic investigation of sycophancy and self-bias in multi-agent debate, despite their prior study in single-agent interactions.These behaviors can distort debate dynamics and undermine collective reasoning's intended benefits.Source: §1 · p. 1
• The paper introduces Conformity and Obstinacy to measure alignment with a peer's prior answer versus an agent's own prior answer under disagreement.It also introduces the Identity Bias Coefficient (IBC) and extends analysis to heterogeneous agents and multiple-peer settings.Source: §1 · p. 2
• Response Anonymization masks response-source labels while preserving argument exchange and evaluation, requiring no retraining, auxiliary loss, or architectural changes.The method is described as directly applicable across model families and debate settings.Source: §1 · p. 2
• 0.608 to 0.024: on MMLU, Qwen-32B's Conformity–Obstinacy gap fell under anonymization, while experiments across models and benchmarks found identity bias pervasive and anonymization effective.The reported values are for the vanilla and anonymized settings, respectively.Source: §1 · p. 2
• The authors formalize identity-weighted belief updating and unify sycophancy and self-bias as identity bias in multi-agent debate.Their framework attributes skew to asymmetric identity weights, separating it from genuine differences in agents' beliefs.Source: §1 · p. 2

2 Preliminaries

• MAD is a collaborative framework in which multiple LLM agents iteratively exchange opinions and responses on a task; a common simultaneous-talk protocol has agents generate and exchange opinions each round.The cited preliminaries identify simultaneous talk as a common design choice.Source: §2 · p. 2
• At each debate round, an agent observes assigned peers' previous responses and may also condition on its own previous output.The protocol begins with agents sampling initial answers from their respective stochastic functions.Source: §2 · p. 2
• The described Figure 1 comparison uses a five-agent debate with one peer per agent and includes four specified model versions.The listed models are Qwen2.5-7b-instruct, Llama3.1-8b-instruct, Mistral-7b-instruct-v0.3, and Qwen2.5-32b-instruct.Source: Figure 1 · p. 3
• After T rounds, the system aggregates agents' final responses by majority voting.The supplied protocol passage identifies majority vote as the debate outcome rule.Source: §2 · p. 3

3 How Does Agent Identity Affect Multi-Agent Debate?

• The section frames identity-conditioned updating as a reliability concern and motivates formal modeling and an intervention to address it.The motivating setup treats debate as a way to amplify correct reasoning and enable mutual error correction.Source: §3 · p. 3, §3.2 · p. 4

3.1 Motivating Analysis

• Conformity measures alignment with a peer’s prior answer during disagreement, while Obstinacy measures repeating one’s own prior answer.The analysis begins with homogeneous agents in a single-peer setting to avoid confounding effects from group dynamics.Source: §3.1 · p. 3

3.2 Empirical Evidence of Identity Bias in Multi-Agent Debate

• Conformity generally exceeds Obstinacy across four LLMs and three benchmarks, though Mistral-7B on GSM8K shows the reverse pattern.The reported metrics aggregate observations from five agents across multiple dataset samples.Source: §3.2 · p. 3

4.1 Formalizing Debate Under Identity Bias

• The formalization distinguishes sycophancy, or alignment with peers, from self-bias, or persistence on an agent’s own prior output.It addresses a modeling gap in prior work that treated peer influence and self-reliance uniformly.Source: §4.1 · p. 4
• The framework represents agents’ answer generation with a Dirichlet-Compound-Multinomial model and updates beliefs using identity-weighted responses.The Dirichlet prior represents internal belief, while the Multinomial represents stochastic answer generation.Source: §4.1 · pp. 4–5
• Theorem 1 expresses Conformity and Obstinacy under identity-driven updates and decomposes their difference into belief differences and identity-weight asymmetry.The belief component reflects prior confidence in peer versus self answers, while the identity component reflects asymmetric weighting.Source: §4.1 · p. 5
• The DCM model’s estimated parameters and identity weights closely match values computed from the underlying data in anonymized and non-anonymized conditions.The authors present this comparison as validation of the model’s approximation of observed debate behavior.Source: §4.1 · p. 5

4.2 Response Anonymization

• Response anonymization removes identity labels, enforcing equal self and peer weights so relative conformity depends on belief state rather than identity.Under symmetric weights, the expected Conformity–Obstinacy difference is zero in the homogeneous-agent setting.Source: §4.2 · p. 6
• The Identity Bias Coefficient (IBC) isolates the identity-attributable portion of an agent’s peer-versus-self orientation after accounting for belief differences.Positive IBC indicates stronger peer-identity weighting; negative IBC indicates stronger self-identity weighting.Source: §4.2 · p. 6

5.1 Setup

• The evaluation covers five model families and four benchmarks spanning knowledge, professional medicine, commonsense, and grade-school math tasks.The models include Qwen, Llama, Mistral, and GPT-OSS families.Source: §5.1 · p. 6

5.2 Experimental Results

• Across 20 model–dataset cases, 18 IBC values are positive and 2 negative, showing identity bias is widespread and strongly skewed toward sycophancy.The IBC sign indicates sycophantic behavior when positive and self-biased behavior when negative.Source: §5.2 · p. 6
• On MMLU, Qwen-32B’s Δ falls from 0.608 in vanilla MAD to 0.024 with Response Anonymization, matching the predicted collapse of identity-driven distortion.Table 1 reports first-round measurements.Source: §5.2 · pp. 6–7, Table 1 · p. 7
• Anonymization generally reduces Subversion more than Correction; for Qwen-32B on MMLU Professional Medicine, the decreases are 64.3% and 14.9%, respectively.Subversion measures becoming wrong after previously being right when the peer was wrong; Correction measures becoming right after previously being wrong when the peer was right.Source: §5.2 · p. 7, Figure 3 · p. 8
• Illustrative debates show a vanilla agent abandoning a correct answer to align with a disagreeing peer, whereas anonymized agents assess arguments without speaker identity.The examples are presented as qualitative evidence that anonymization shifts attention from identity to argument content.Source: §5.2 · pp. 7–8
• Additional analyses find similar identity effects with heterogeneous agents, compounding bias with more peers, and accumulating distortions across longer debates.The paper also examines anonymization with a domain-expert agent.Source: §5.2 · p. 8

Multi-Agent Debate

• MAD research spans task applications, protocol and theoretical enhancements, agent diversity strategies, and learned methods for optimizing debate dynamics.The section cites examples across these strands, including task applications [Du et al., 2024; Chan et al., 2024].Source: §6 · p. 8
• Prior evaluations report failure modes, inconsistent gains over single agents, incorrect answers, majority-driven convergence, and performance degradation over multiple rounds [Cemri et al., 2025; Zhang et al., 2025a].The cited work also examines other shortcomings, including incorrect answers and convergence dynamics.Source: §6 · p. 8

Sycophancy and Self-Bias

• Identity-bias research mainly concerns single-agent interactions, and few MAD studies address sycophancy; the authors present their work as unifying sycophancy and self-bias for MAD [Agarwal and Khanna, 2025].They also propose response anonymization as a method for eliminating identity bias from multi-agent systems.Source: §6 · pp. 8–9

• Agents’ deference to peers or adherence to their own prior answers can undermine effective error correction in multi-agent debate.Source: §7 · p. 9
• The paper unifies these behaviors as identity bias and proposes response anonymization to remove identity cues and encourage content-driven reasoning.Source: §7 · p. 9
• Experiments across models and benchmarks find pervasive identity bias and report that anonymization mitigates it while improving debate trustworthiness.Source: §7 · p. 9

• The framework treats identity bias as the primary source of heterogeneous weights, not the only possible influence on debate.Source: Limitations · p. 9
• Context length and response quality are proposed as additional factors that could affect how influence is distributed.The paper suggests that more peers may dilute attention and that well-reasoned answers could receive greater influence regardless of speaker identity.Source: Limitations · p. 9
• The authors identify quality-based weighting and contextual scaling as directions for a richer account of influence allocation and more reliable system-design strategies.Source: Limitations · p. 9

• The work aims to improve the reliability of multi-agent debate systems.Source: Ethical Considerations · p. 9
• The authors report that their anonymization intervention does not manipulate sensitive data or individuals and does not negatively affect privacy or welfare.Source: Ethical Considerations · p. 9
• Experiments use publicly available benchmarks, with no human subjects or conflicts of interest reported.Source: Ethical Considerations · p. 9

• LLM tools assisted in polishing portions of the writing.Source: Disclosure of LLM Usage · p. 9
• LLM tools also assisted with literature searches for relevant related work.Source: Disclosure of LLM Usage · p. 9
• The searches were intended to check for relevant related work the authors might have missed.Source: Disclosure of LLM Usage · p. 9

A Qualitative Examples

• With identity cues, agents sometimes deferred to peers despite correct conclusions or adopted peer opinions without substantive reevaluation; anonymized agents evaluated argument soundness instead.The comparison uses Vanilla MAD and Anonymized MAD, which differ only in whether identity cues are included in the debate input.Source: A Qualitative Examples · p. 14
• In a tongue-cancer example, the debate output revisited the agents’ opinions and changed its assessment to evaluate submandibular nodes first.The output says these nodes are the primary drainage site for the tongue apex, after initially identifying submental nodes as the first choice.Source: A Qualitative Examples · p. 17
• For a tumor at the tongue apex, the debate output recommends evaluating the submental lymph nodes first for metastasis.Its reasoning identifies the submental nodes as a significant drainage pathway for the apex and a first filter for cancer cells.Source: A Qualitative Examples · p. 18
• One pneumonia debate selects amoxicillin after opinions differ between amoxicillin for community-acquired pneumonia and erythromycin for possible Mycoplasma infection.The final output recommends amoxicillin for a young, noncritically ill patient with mild symptoms and patchy infiltrates.Source: A Qualitative Examples · p. 19
• In a later exchange about the same case, the output favors erythromycin, echoing an opinion that cites possible Mycoplasma pneumoniae despite a competing amoxicillin recommendation.The competing opinion identifies amoxicillin as the preferred initial treatment for community-acquired pneumonia in adults who are not critically ill.Source: A Qualitative Examples · p. 20

B.1 Dataset Details

• Evaluation used 198 GPQA Diamond items, 300 GSM8K samples, all 272 MMLU Professional Medicine test items, and 300 HellaSwag samples.Decoding used temperature 1.0 and nucleus sampling with p = 0.9.Source: §B.1 · p. 21, §B.2 · p. 21

B.3 Evaluation Details; D DCM Parameter Estimation

• Conformity and Obstinacy are averaged across dataset instances and agents, then used to derive Δ and IBC; the estimates are justified as maximum-likelihood estimators under homogeneity and i.i.d. samples.The DCM parameter estimates closely match ground truth in the reported comparisons.Source: §B.3 · p. 21, D DCM Parameter Estimation · p. 22

C.1 Standard Debate Prompt; C.2 Anonymized Debate Prompt; C.3 Persona Prompts

• The standard prompt labels the agent’s prior answer and peer opinions, while the anonymized prompt removes attribution and randomly orders the opinions.Clinical-knowledge personas include Assistant, Doctor, Psychologist, Mathematician, and Programmer.Source: §C.1 · p. 21, §C.2 · p. 21, §C.3 · p. 22

D DCM Parameter Estimation

• Fitting the DCM model yields estimates that closely match ground-truth parameters and identity weights in both anonymized and non-anonymized conditions.The reported comparisons cover Qwen-7B on GPQA and MMLU Professional Medicine, and Llama-8B on MMLU Professional Medicine.Source: D DCM Parameter Estimation · p. 22, Table 2 · p. 22, Table 3 · p. 22

E Extension to Heterogeneous Agents

• For Qwen-7B, heterogeneous agents’ raw Δ falls from 0.457 to 0.083 with anonymization, while IBC falls from 0.408 to 0.374 versus homogeneous agents.Similar anonymization trends hold across other models; the IBC comparison suggests persona diversity reduces identity asymmetries.Source: E Extension to Heterogeneous Agents · p. 23, Table 5 · p. 23

F Identity Bias Across Debate Rounds

• IBC consistently increases in debate round two across five models and four benchmarks, indicating that identity bias persists and amplifies as debate progresses.Anonymization may prevent identity-driven tendencies from accumulating across rounds.Source: F Identity Bias Across Debate Rounds · p. 23, Figure 4 · p. 23

G Extension to Multiple Peers

• Across benchmarks, four disagreeing peers reduce IBC versus a single peer on Qwen-7B, with the size of the reduction varying by task.The framework defines conformity as aligning with a disagreeing peer and obstinacy as retaining one’s prior answer when peers disagree.Source: G Extension to Multiple Peers · pp. 24–25, Figure 5 · p. 25

H Anonymization when Domain-expert Personas are Present

• Anonymization reduces conformity toward Doctor and Assistant personas by the same relative amount, 33.3%; baseline conformity is higher toward Assistant.Both personas achieve approximately 80% accuracy on the benchmark, and the authors report no strong expert-preferring behavior in this setup.Source: H Anonymization when Domain-expert… · p. 25, Table 6 · p. 25

J Effect of Anonymization on Task Performance

• Across four benchmarks, accuracy remains largely unchanged after anonymization for Qwen-7B, Qwen-32B, and GPT-OSS-20B.The authors frame anonymization as removing identity-driven distortions, not amplifying persuasive or error-correcting effects.Source: J Effect of Anonymization on Task… · p. 27, Figure 6 · p. 28
• In four of five examined GPQA cases, anonymized agents weighted peer responses uniformly and converged to an incorrect conclusion.In the remaining case, the agent reasoned at length but did not state a final answer; the authors attribute such behaviors to base-model limitations rather than a systematic anonymization effect.Source: J Effect of Anonymization on Task… · p. 28

J.1 Proof of Martingale Property

• Under the weighted update, predictive probabilities remain a martingale when all agents draw from the same predictive distribution.The proof supports the claim that anonymization does not introduce new evidence or asymmetries expected to improve performance.Source: §J.1 · p. 28