PaperlayerView exact revision on arXiv

Source-linked AI summary

The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization

Luoxi Tang, Yuqiao Meng, Joseph Costa, Yingxue Zhang, Muchao Ye, Zhaohan Xi

arXiv:2602.07186v1cs.MAcs.LG

TL;DR

Multi-agent debate can suffer debate collapse, while existing approaches lack principled mechanisms to detect or prevent it. The paper introduces hierarchical behavioral uncertainty measures and uncertainty-driven policy optimization; its reported analyses distinguish failed from successful reasoning, and mitigation improves accuracy while reducing behavioral volatility.

  • Problem

    Debate collapse remains a vulnerability of multi-agent debate, and existing methods lack well-established indicators of collapse and mitigation that inherently improves MAD robustness.

    Source: §1 · p. 1

  • Method

    The paper proposes hierarchical uncertainty measures and UDPO, which uses an asymmetric objective to penalize uncertainty while incorporating a shared task reward.

    Source: §1 · pp. 1–2, §7 · pp. 8–9

  • Results

    Across reasoning benchmarks, failures show higher uncertainty than successes, while UDPO substantially improves accuracy and reduces behavioral volatility.

    Source: §3.2 · p. 3, §7 · pp. 8–9

  • Takeaways & Limitations

    Behavioral uncertainty signals can diagnose debate failures, and UDPO is reported to yield more reliable, better-calibrated multi-agent systems.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Multi-agent debate (MAD) systems improve LLM reasoning through iterative deliberation, but remain vulnerable to debate collapse, a failure type where final agent decisions are compromised on erroneous reasoning. Existing methods lack principled mechanisms to detect or prevent such failures. To address this gap, we first propose a hierarchical metric that quantifies behavioral uncertainty at three levels: intra-agent (individual reasoning uncertainty), inter-agent (interactive uncertainty), and system-level (output uncertainty). Empirical analysis across several benchmarks reveals that our proposed uncertainty quantification reliably indicates system failures, which demonstrates the validity of using them as diagnostic metrics to indicate the system failure. Subsequently, we propose a mitigation strategy by formulating an uncertainty-driven policy optimization to penalize self-contradiction, peer conflict, and low-confidence outputs in a dynamic debating environment. Experiments demonstrate that our proposed uncertainty-driven mitigation reliably calibrates the multi-agent system by consistently improving decision accuracy while reducing system disagreement.

1. Introduction

Multi-agent debate (MAD) uses iterative deliberation to refine LLM reasoning, but debate collapse and limited ways to detect or prevent it leave important gaps. The paper frames its work as addressing these gaps at two research levels.

1. Introduction

  • MAD systems iteratively refine reasoning through collaborative perspectives and have shown effectiveness across reasoning, question-answering, and generation tasks.The introduction contrasts this approach with single-agent generation, which is sensitive to instance-specific errors.Source: §1 · p. 1
  • Debate collapse can involve influence from a confident but compromised minority or endless loops of conflicting arguments.The introduction says existing approaches often treat collapse as an outcome-level error and lack granular metrics for detecting derailment.Source: §1 · p. 1
  • The paper identifies gaps in frameworks for indicating debate collapse and in mitigation that improves MAD robustness without auxiliary components such as judge agents.The work is introduced as research addressing these two gaps.Source: §1 · pp. 1–2

I. Uncertainty-driven indicators of debate collapse

The paper treats behavioral volatility as a potential signal of multi-agent reliability and measures uncertainty within agents, between agents, and at the system level. Across reasoning benchmarks, incorrect predictions show higher uncertainty on all three measures than correct predictions.

I. Uncertainty-driven indicators of debate collapse

  • Frequent stance revisions, divergence from peers, and low-confidence outputs are described as behaviors that may disrupt system robustness.The paper frames quantifying these signals as a way to diagnose debate robustness before an incorrect final decision.Source: §I · p. 2
  • The framework measures stance changes within an agent, disagreement between agents, and entropy of the final consensus.The paper motivates these measures as behavioral indicators of reliability, including instability and inconsistency when confidence is misaligned with correctness.Source: §I · p. 2
  • Incorrect predictions consistently show higher intra-agent flip rates, inter-agent disagreement, and system-level output entropy than correct predictions across reasoning benchmarks.The paper presents these behavioral patterns as a distinct footprint of debate collapse and validates the measures as diagnostic indicators.Source: §I · p. 2

II. Uncertainty-driven collapse mitigation

The paper proposes UDPO, which uses uncertainty-derived indicators to penalize unstable reasoning, peer conflict, and low-confidence outputs during debate. Experiments report higher accuracy than baseline multi-agent fine-tuning and reduced collapse, including under attacks.

II. Uncertainty-driven collapse mitigation

  • UDPO penalizes self-contradiction, peer conflict, and low-confidence outputs through an uncertainty-driven reward in dynamic debate.Its asymmetric objective assigns agent-specific penalties while incorporating a shared task reward for problem-solving correctness.Source: §II · p. 2
  • MAD fine-tuned with UDPO recovers from collapse scenarios with more than 25% higher accuracy than baseline multi-agent fine-tuning methods.Under explicit attacks involving compromised agents, the method reduces hallucination-induced uncertainty and improves MAD accuracy.Source: §II · p. 2
  • The paper reports that UDPO reduces debate collapse and remains robust to attacks during multi-agent consensus.The conclusion also describes improved accuracy and reduced behavioral volatility.Source: §II · p. 2

2. Uncertainty Quantification in MAD

The section defines question-level uncertainty measures for multi-agent debate at three complementary levels: individual stance changes, peer disagreement, and collective output confidence. It expects correct predictions to have lower uncertainty, an intuition evaluated across reasoning tasks.

2. Uncertainty Quantification in MAD

  • The framework quantifies uncertainty per question at intra-agent, inter-agent, and system levels, then aggregates measures over datasets.In the described MAD setting, agents exchange arguments, critique responses, and refine reasoning over multiple deliberation rounds.Source: §2 · p. 2
  • Intra-agent uncertainty combines stance-flip and belief-revision measures to capture how frequently agents revise answers.The section describes large intra-agent uncertainty as indicating unstable reasoning on the question.Source: §2 · pp. 2–3
  • Inter-agent uncertainty measures how much agents disagree through pairwise conflict during each debate round.The section also associates high inter-agent uncertainty with fluctuations and discrepancy in decision-making.Source: §2 · p. 3
  • System-level uncertainty aggregates normalized output entropy, final disagreement, and leave-one-out instability.The section identifies these as complementary signals of output diversity, lack of consensus at termination, and sensitivity to individual agents.Source: §2 · p. 3
  • The section expects correct predictions to show low uncertainty, with stable agents, agreement, and confident convergence.It says the intuition is then empirically validated across diverse reasoning tasks.Source: §2 · p. 3

3. Evaluations of Uncertainty Quantification

This evaluation tests whether behavioral uncertainty at three levels can indicate failures in multi-agent debate. Across the reported analyses, uncertainty distinguishes failed from successful reasoning and provides signals for mitigation design.

3.1. Evaluation Setting

  • The evaluation uses five agents over five debate rounds on GSM8K, TruthfulQA, and CommonsenseQA, with iterative critique and refinement followed by majority vote.It considers both natural collapse and a compromised-agent condition involving stealthy errors.Source: §3.1 · p. 3

3.2. Evaluation Results

  • Across all uncertainty types, failures show higher scores than successes; mean system-level uncertainty (Usys) is 0.67 for failures versus 0.09 for successes.The distributions are statistically separated, with p < 0.001 and Cohen’s d > 0.8.Source: §3.2 · p. 3, Figure 2 · p. 4
  • r < 0 (p < 0.001): all three uncertainty measures correlate negatively with accuracy, while their mutual correlations approach 1.Their distributions nevertheless differ significantly, consistent with the measures capturing complementary aspects of uncertainty.Source: §3.2 · p. 3, Figure 3 · p. 4
  • Low-uncertainty subsets reach up to 100% accuracy at k = 50%, while accuracy degrades sharply as higher-uncertainty traces are added.The analysis ranks traces by each uncertainty measure and retains the lowest-uncertainty top k%; the pattern also holds across datasets.Source: Figure 4 · p. 4, §3.2 · p. 4
  • The authors interpret differing uncertainty distributions as reflecting distinct failure modes, including isolated mistakes later corrected or amplified and confident agents shifted by peers.They report that the indicators remain effective under adversarial perturbations.Source: §3.2 · p. 4

4. Uncertainty-Driven Mitigation

The mitigation framework uses uncertainty metrics to shape debate dynamics, then optimizes agents asymmetrically according to their uncertainty profiles. Its rewards target reasoning stability, inter-agent agreement, and system-level confidence and robustness.

4. Uncertainty-Driven Mitigation

  • Each agent’s debate trajectory provides the basis for rewards that steer the interactive process using uncertainty metrics.The trajectory comprises N agents across T rounds.Source: §4.2 · p. 5

4.2. Uncertainty-Driven Reward Design

  • The reward design targets intra-agent stability, inter-agent agreement, and system-level confidence and robustness.The agreement reward favors early convergence over late-stage consensus; the system-level reward incorporates entropy, disagreement, and leave-one-out instability.Source: §4.2 · p. 5

4.3. Asymmetric Objective for Uncertainty-Driven Policy Optimization (UDPO)

  • UDPO assigns different training intensities to agents based on their uncertainty profiles, rather than treating all agents as equally prone to debate collapse.Agents with higher uncertainty receive stronger uncertainty-driven rewards, while already-stable agents receive relatively higher task rewards, avoiding over-regularization of well-behaved agents.Source: §4.3 · p. 5
  • The agent-specific objective combines intra-agent, inter-agent, and system-level rewards with task supervision tied to majority-vote correctness.Coefficients are tailored using empirical uncertainty profiles measured during a warm-up phase.Source: §4.3 · pp. 5–6
  • A clipped relative-update objective constrains policy changes and anchors them to a reference policy, with stronger anchoring for instability-prone agents.The reference may be a lagged snapshot or the base model.Source: §4.3 · p. 5

5. Evaluations of Mitigation

UDPO improves debate performance while reducing uncertainty, and its gains persist under attacks and increase with question uncertainty. Ablations show that all three targeted loss components contribute, with system-level stability most important for accuracy.

5. Evaluations of Mitigation; 5.1. Setup

  • The evaluation asks whether UDPO improves overall performance, how its loss components contribute, whether it withstands attacks, and when it helps most.It uses the same datasets and LLMs as Section 3 and compares against Standard MAD, MAPPO, and RMAAC.Source: §5 · p. 6, §5.1 · p. 6

5.2. Main Results (RQ1)

  • UDPO consistently achieves the best overall performance, gaining up to 25 percentage points over Standard MAD and up to 20 points over MAPPO on GSM8K (N=5).On GSM8K, UDPO also reduces system-level uncertainty by roughly 80% relative to Standard MAD.Source: §5.2 · p. 6
  • UDPO penalizes flip-flopping, unresolved peer disagreement, and low-confidence outcomes, directly targeting three failure modes rather than relying on centralized credit assignment or worst-case perturbation robustness.The authors describe these objectives as driving debates toward stable, verifiable agreement, including under attacks.Source: §5.2 · pp. 6–7

5.3. Ablation Study (RQ2)

  • Removing any loss component harms accuracy and uncertainty; removing Lsys causes the largest accuracy drop, followed by Lintra and Linter across all three models.Each removal most affects its targeted uncertainty measure, and the full model significantly outperforms every ablation.Source: §5.3 · p. 7

5.4. Robustness to Attack (RQ3)

  • Under attacks, UDPO degrades more gracefully than baselines and retains a larger accuracy margin as the number of compromised agents rises.The attack evaluation injects stealthy semantic errors into one, two, or three agents in an N=5 debate; baselines show increasing disagreement and system-level indecision.Source: §5.4 · p. 7, Table 3 · p. 7

5.5. Analysis: When does UDPO help most? (RQ4)

  • UDPO’s accuracy gains rise from +2.1% at Usys < 0.2 to +18.3% at Usys ≥0.8, while MAPPO and RMAAC gains peak at medium uncertainty and decline on hard questions.Figure 5 averages accuracy improvement over Standard MAD across datasets and models.Source: §5.5 · p. 8, Figure 5 · p. 8

6. Related Work

Prior work explores debate architectures and single-agent uncertainty estimation, but does not directly address behavioral dynamics and reliability in multi-agent debate. This paper positions its hierarchical uncertainty metrics and training objective as complementary contributions.

Multi-agent debate

  • MAD research has explored structured argumentation, role-based discussion, and judge-mediated deliberation, but lacks principled mechanisms to detect or prevent debate collapse.The cited work reports improved accuracy over single-agent baselines while focusing mainly on architectural innovations.Source: §6 · p. 8

Uncertainty quantification in LLMs

  • Existing uncertainty methods include token entropy, semantic-equivalence clustering, self-consistency, confidence calibration, and verbalized uncertainty, primarily for single-agent settings.These approaches do not capture the behavioral dynamics of multi-agent systems.Source: §6 · p. 8

Uncertainty quantification in LLMs; Multi-agent debate

  • The paper extends uncertainty quantification to MAD through intra-agent, inter-agent, and system-level components that capture distinct failure signatures.It presents these diagnostics and a training objective for mitigating failure modes as complementary to existing debate research.Source: §6 · p. 8

7. Conclusion

The paper introduces hierarchical uncertainty diagnostics for multi-agent debate and builds UDPO to encourage stable reasoning, agreement, and confident outputs. It reports improved accuracy and reduced behavioral volatility across multiple reasoning benchmarks.

7. Conclusion

  • The framework quantifies behavioral instability at intra-agent, inter-agent, and system levels.The paper reports that these signals distinguish successful debates from failed ones across multiple reasoning benchmarks.Source: §7 · p. 8
  • UDPO uses an asymmetric training objective to encourage stable reasoning, agent agreement, and confident outputs.The optimization framework is built on the paper’s uncertainty analysis.Source: §7 · pp. 8–9
  • The authors report that UDPO substantially improves accuracy while significantly reducing behavioral volatility, yielding more reliable and better-calibrated multi-agent systems.Source: §7 · p. 9

Impact Statement

The paper frames its methods as research support for reliable, interpretable collaborative reasoning, not direct real-world decision-making deployment. Its appendices describe training, agent-specific calibration, uncertainty-prioritized replay, and supplementary diagnostic results.

Impact Statement

  • The work is intended to support research on reliable, interpretable collaborative reasoning rather than direct deployment in real-world decision-making.The authors call for further study of responsible deployment and human oversight in practical applications.Source: Impact Statement · p. 9

A. Training Details

  • Models are fine-tuned for 10 epochs with AdamW at learning rate 1 × 10−5, using eight NVIDIA A100 GPUs and batches of 32 trajectories per update.Training uses 500 warmup steps, cosine decay, and gradient clipping at a maximum norm of 1.0.Source: §A.1 · p. 11

A.1. Experimental Settings

  • The experimental settings use eight NVIDIA A100 GPUs, batches of 32 trajectories per update, and a 500-step warmup followed by cosine decay.The reported fine-tuning uses AdamW, learning rate 1 × 10−5, and gradient clipping at 1.0.Source: §A.1 · p. 11

A.2. Asymmetric Training and Agent-Tailored Hyper-parameter Selection

  • Agent-tailored coefficients scale training intensity to each agent’s uncertainty profile, avoiding over-regularization of stable agents and focusing effort on agents contributing most to collapse.Higher-uncertainty agents receive stronger equilibrium rewards and anchoring, and weaker task rewards; κ defaults to 1.5.Source: §A.2 · p. 11

A.3. Uncertainty-Driven Replay

  • Uncertainty-driven replay samples high-uncertainty trajectories more often instead of discarding them at inference time.The procedure scores buffered trajectories using intra-, inter-, and system-level uncertainty, then prioritizes sampling by that score.Source: §A.3 · p. 11

A.4. UDPO Algorithm

  • The UDPO algorithm runs multi-agent debates, computes uncertainty-based and task rewards, forms advantages, and updates policy parameters against a reference policy.The algorithm periodically refreshes the reference policy and returns optimized parameters.Source: §A.4 · p. 12

B. Additional Results on Uncertainty Quantification

  • Supplementary uncertainty analyses validate the metrics as diagnostic tools on the natural, unmitigated MAD system.The appendix presents additional results complementing the main uncertainty findings.Source: B. Additional Results on Uncertainty… · p. 12

B.1. Uncertainty Distributions

  • On TruthfulQA and CommonsenseQA, incorrect predictions exhibit significantly higher uncertainty across all three metrics, with substantial effect sizes.The authors report that the pattern is consistent with GSM8K results.Source: §B.1 · p. 12

B.2. Correlation Analysis

Across supplementary analyses, uncertainty remains associated with incorrect predictions and lower accuracy, including under adversarial attack. UDPO retains accuracy advantages across homogeneous ensembles, debate-round settings, and attack conditions.

B.2. Correlation Analysis

  • Across additional datasets, uncertainty is higher for incorrect predictions, and all uncertainty types negatively correlate with accuracy while correlating positively with one another.These results supplement the analyses across GSM8K, CommonsenseQA, and TruthfulQA.Source: §B.2 · p. 12, Figure 7 · p. 13, Figure 6 · p. 13

B.3. Selective Prediction

  • Retaining only low-uncertainty predictions substantially improves accuracy on GSM8K and CommonsenseQA, consistent with the TruthfulQA results.The appendix figure reports uncertainty-guided abstention across additional datasets.Source: §B.3 · p. 13, Figure 8 · p. 14

C.1. Homogeneous Model Ensembles

  • Homogeneous ensembles gain 18–24 percentage points in accuracy with UDPO over Standard MAD; Qwen-3 performs best across datasets, while Gemma-3n improves most relatively.The evaluation uses five agents of the same base model; homogeneous ensembles have lower baseline accuracy than the heterogeneous ensemble.Source: §C.1 · p. 14, Table 4 · p. 14

C.2. Effect of Debate Rounds

  • UDPO maintains a 16–20 percentage-point advantage over Standard MAD across T = 3, 5, and 10 debate rounds.More rounds improve all methods with diminishing baseline returns; computational cost scales linearly with T, and the authors identify T = 5 as a practical default.Source: §C.2 · p. 14, Table 5 · p. 15

C.3.1. ATTACK SETUP

C.3.2. UNCERTAINTY STILL SEPARATES CORRECT FROM INCORRECT

  • Under attack, incorrect predictions show higher uncertainty across all three metrics, with Cohen’s d = 2.24–3.53 on GSM8K and similar patterns on TruthfulQA and CommonsenseQA.The strongest separation is for system-level uncertainty (d = 3.53).Source: §C.3.2 · p. 15, Figure 9 · p. 15

C.3.3. CORRELATIONS REMAIN STRONG

  • Under attack on GSM8K, uncertainty correlates negatively with accuracy (r = −0.77 to −0.89) and positively across uncertainty metrics (r = 0.57–0.83).The authors report slightly stronger attack-setting correlations than in natural settings, including Usys at r = −0.89 versus −0.82 on GSM8K.Source: §C.3.3 · pp. 15–16, Table 6 · p. 16, Figure 10 · p. 16

C.3.4. UDPO IS MOST ROBUST

  • Under attack, UDPO achieves 77.2% accuracy on GSM8K and 66.4% on TruthfulQA, with drops of −5.2pp and −14.8pp, respectively.Standard MAD loses over 34 percentage points, while RMAAC drops 23.8pp.Source: §C.3.4 · p. 16, Table 7 · p. 16

C.4. Analysis by Question Difficulty

The difficulty-stratified analysis reports increasing UDPO gains at higher uncertainty, while the appendix defines system-level uncertainty as a combination of distinct indicators and uses its inverse as a reward.

C.4. Analysis by Question Difficulty

  • UDPO gains rise from +2.3% to +17.1% by Uintra and from +2.5% to +16.8% by Uinter, while both baselines plateau or decline.Across all three uncertainty stratification metrics, UDPO gains are larger on harder questions; MAPPO and RMAAC show an inverted-U pattern.Source: §C.4 · pp. 16–17, Figure 11 · p. 17

D. Uncertainty Quantification Details

D.1. Component Definitions

  • System-level uncertainty combines answer entropy, minority disagreement, and leave-one-out instability, which respectively capture answer spread, dissent, and precarious majorities.The disagreement indicator is one if any agent differs from the majority and zero only for unanimous answers; leave-one-out instability measures sensitivity to removing one agent.Source: §D.1 · pp. 17–18, §D.2 · p. 18

D.2. System-Level Uncertainty and Reward

  • The system-level reward directly inverts uncertainty and is maximized when entropy is low, disagreement is absent, and leave-one-out instability is zero.These conditions correspond to confident prediction, unanimous agreement, and robust consensus.Source: §D.2 · p. 18

Related papers

Loading 2602.07186v1…