PaperlayerView exact revision on arXiv

Source-linked AI summary

Remember and Reweight: Enhancing Multi-Agent Debate with Experience Memory and Confidence Estimation

Xuanfa Jin, Zhijian Ma, Yongcheng Zeng, Xinyu Cui, Haifeng Zhang, Jun Wang

arXiv:2609.03619v1cs.CLcs.AI

TL;DR

MAD can amplify shared misconceptions when incorrect majorities persuade correct agents, while existing methods leave biased concept priors insufficiently addressed. R2-MAD uses debate-state-aware experience retrieval to correct priors and memory-derived confidence weights to modulate peer influence. Across benchmarks and models, it consistently improves over single-agent and MAD baselines, though it adds overhead and depends on informative, task-relevant memory.

  • Problem

    Shared misconception causes MAD to amplify incorrect majority beliefs, while existing approaches mainly address peer skew rather than agents’ biased concept priors.

  • Method

    R2-MAD retrieves debate-state-relevant historical experiences to calibrate concept priors and estimate per-agent reliability for confidence-weighted peer influence.

  • Results

    R2-MAD consistently improves over single-agent methods and existing MAD baselines across various benchmarks and models.

  • Takeaways & Limitations

    The results support jointly intervening on concept prior and peer skew to mitigate shared misconceptions in multi-agent debate.

  • Takeaways & Limitations

    R2-MAD requires informative memory and an outcome signal, may degrade under deployment-task shift, and adds overhead that makes it less suitable for latency-critical deployments.

Abstract

from arXiv · show

Multi-agent debate (MAD) improves the reasoning capabilities of large language models by having multiple agents iteratively refine their responses through discussion. However, MAD suffers from a critical vulnerability known as shared misconception: when a majority of agents initially converge on an incorrect answer, the debate process tends to amplify rather than correct the error. Existing methods primarily address peer skew but leave the agents' inherently biased concept priors unaddressed. To mitigate this systematic weakness, we propose R$^2$-MAD (Remember and Reweight for Multi-Agent Debate), a framework that equips agents with an experience memory accumulated from past debates. R$^2$-MAD intervenes on both failure modes through two complementary mechanisms: A debate-state-aware retrieval policy dynamically calibrates the concept prior by retrieving relevant historical evidence based on the current consensus level. Then these retrieved experiences provide a basis for estimating per-agent reliability, yielding confidence weights to modulate peer influence. Experiments on various benchmarks show that R$^2$-MAD achieves consistent improvements over existing single-agent and MAD baselines.

1 Introduction

MAD can improve LLM reasoning through iterative multi-agent refinement, but shared misconception causes incorrect majorities to persuade correct agents and amplify errors. R2-MAD addresses both biased concept priors and peer skew using debate-state-aware memory retrieval and confidence weighting.

  • Motivation: MAD improves reasoning by having multiple LLM agents iteratively discuss and refine responses.The paper describes gains over single-agent baselines across reasoning, evaluation, and problem-solving tasks.
  • Motivation: Shared misconception occurs when an incorrect majority persuades correct agents, producing and amplifying a wrong consensus.The paper characterizes this as a systematic MAD weakness rather than an occasional anomaly.
  • Motivation: Latent-concept analysis attributes the failure to a biased concept prior compounded by peer skew from agreeing agents.Existing approaches primarily manipulate peer responses, leaving the agents’ intrinsic priors insufficiently addressed.
  • R2-MAD: R2-MAD retrieves historical experiences according to consensus level to calibrate priors, then estimates agent reliability to weight peer influence.High consensus favors diverse, contrastive experiences, while divided debates favor experiences associated with positive outcomes.
  • Results: Experiments report consistent improvements over single-agent and existing MAD baselines, with ablations supporting both memory-based prior correction and confidence weighting.The framework is also presented as intervening on both concept prior and peer skew.

2 Related Work

Related work places R2-MAD at the intersection of multi-agent debate and memory-augmented LLM agents. Prior research studies debate interaction, memory storage, retrieval, and methods for limiting error propagation.

  • Multi-Agent Debate: MAD research explores divergent debate, LLM-as-judge evaluation, efficient communication structures, and other collaborative applications.These approaches refine how multiple LLM agents interact to improve reasoning and related tasks.
  • Positioning: The paper situates R2-MAD among prior MAD and memory-based methods while targeting shared misconception through experience memory.Its stated contribution is distinctively framed around combining debate-state-aware retrieval with confidence weighting.
  • Memory-Based LLM Agents: Memory-based LLM agents store observations, reflections, feedback, reusable skills, and long-term interaction histories.Memory supports behavior, iterative self-improvement, interaction reuse, and persistent histories.
  • Memory-Based LLM Agents: Recent methods improve memory retrieval, compression, and organization, while MAD-specific approaches mask unreliable memories or retrieve structured debate transcripts.The cited methods address memory selection and error propagation in different ways.

3 Preliminary

The preliminary MAD formulation models agents that independently generate initial responses and iteratively update them using the previous round’s joint responses. A latent-concept decomposition separates intrinsic generation from peer-induced coordination bias.

  • Multi-Agent Debate Framework: MAD uses n LLM agents to resolve task x over up to T iterative debate rounds.Each agent has parameters ϕ_i representing factors such as model weights, training data, or prior contexts.
  • Multi-Agent Debate Framework: At round zero, each agent independently generates a response; later rounds condition on the previous round’s joint responses.Debate stops at the maximum horizon or when consensus is established, after which answers are extracted.
  • Latent Concept Decomposition: The latent-concept framework assumes each task-answer pair arises from a true latent concept θ⋆.A conditional-independence assumption supports analyzing agent responses through this latent concept space.
  • Latent Concept Decomposition: The generation process factorizes into an agent’s intrinsic capability and coordination bias introduced by peer interactions.This decomposition distinguishes what an agent would generate independently from the influence accumulated during debate.

4 R2-MAD Framework

R2-MAD augments debate agents with experience memories that correct concept priors and reweight peer influence. Its retrieval policy adapts to consensus, while historical outcomes estimate confidence and theoretically reduce erroneous majority dominance.

  • Framework Overview: R2-MAD models debate generation with a memory-corrected concept prior and confidence-weighted peer contributions.The two modifications target the distinct prior and peer-skew components of the latent decomposition.
  • Framework Overview: At each round, agents summarize debate state, retrieve experiences, estimate peer confidence, and generate updated responses.The overview describes retrieval from a memory bank, prior calibration, peer weighting, and response generation as the recurring pipeline.
  • Experience Memory: Each memory case records debate responses, the task answer, response correctness, and an outcome reward requiring a signal for whether the debate reached a good answer.The reward can come from labels, a verifier model, or execution feedback on verifiable tasks.
  • Debate-State-Aware Memory Retrieval: Retrieval first selects top-3K state-similar cases, then uses MMR to choose K cases while balancing relevance, outcome quality, and redundancy.The consensus-dependent coefficient favors relevant positive experiences at low consensus and diverse contrastive experiences at high consensus.
  • Debate-State-Aware Memory Retrieval: Retrieved experiences act as a likelihood-ratio correction to the bare concept prior, counteracting erroneous-concept bias when they provide stronger evidence for the true concept.This mechanism is formalized as Proposition 4.1.
  • Theoretical Analysis: When misconception-sharing agents receive weight α, the convergence rate toward the erroneous concept slows from O(m) to O(αm).The paper also states that memory and confidence lifts are additive in log-odds space under the stated conditions.

5 Experiments

Experiments evaluate R2-MAD across reasoning and knowledge-intensive benchmarks, models, ablations, retrieval policies, and the shared misconception regime. R2-MAD consistently improves accuracy, with evidence that both memory retrieval and confidence weighting contribute to its gains.

  • Experiment Setups: R2-MAD is evaluated on MATH500, Economics and Engineering from MMLU-Pro, and TruthfulQA using three primary open-source models plus larger-model tests.Debate methods use three agents over three rounds; Llama-3.3-70B-Instruct and GPT-4o-mini provide additional evaluation.
  • Overall Results: R2-MAD achieves the highest average accuracy on all three primary models, outperforming single-agent and debate baselines across the evaluated benchmarks.It also achieves the best average accuracy against CoT and MAD on Llama-3.3-70B-Instruct and GPT-4o-mini.
  • Component Ablation: On Qwen2.5-7B-Instruct, average accuracy falls from 0.607 with full R2-MAD to 0.585 without confidence weighting and 0.576 without memory retrieval.The ablation results support complementary contributions from both components.
  • Retrieval-Policy Ablation: The debate-state-aware retrieval policy reaches 0.663 average accuracy, ahead of fixed-λ retrieval at 0.657, similarity-based retrieval at 0.647, and random retrieval at 0.635.The policy is best or tied-best in three of four individual settings.
  • Shared Misconception: On the shared misconception subset, R2-MAD substantially outperforms MAD and MAD-M2 on Qwen2.5-7B-Instruct, including Economics at 29.41% versus 17.65% for MAD and Engineering at 27.10% versus 17.42%.Relative improvements are larger on this subset than in the overall results.
  • Stance Transitions: Confidence weighting reduces harmful correct-to-wrong transitions on three benchmarks and increases wrong-to-correct recoveries on three of four benchmarks.On Engineering, C →W falls from 0.410 to 0.255; on Economics, W →C rises from 0.075 to 0.172.

6 Conclusion

R2-MAD addresses shared misconceptions by correcting biased concept priors and modulating peer influence with experience-derived confidence weights. It consistently improves over single-agent and MAD baselines, but adds computation and depends on informative, task-relevant memories.

  • Conclusion: R2-MAD uses debate-state-aware retrieval to calibrate agents’ prior beliefs and memory-derived confidence weights to modulate peer influence.The framework targets the two compounding factors identified in shared misconceptions: biased concept priors and amplifying peer skew.
  • Conclusion: Experiments across benchmarks and models show consistent improvements over single-agent methods and MAD baselines, especially under shared misconceptions.Ablations indicate that both components provide distinct benefits and that their combination performs best overall.
  • Limitations: R2-MAD adds a bounded constant-factor computation cost for debate-state summarization, experience retrieval, and per-peer confidence estimation.This overhead makes the framework less suitable than plain debate for latency-critical deployments.
  • Limitations: The method depends on informative task-relevant memory, outcome signals for past debates, and offline memory that may degrade when deployment tasks diverge substantially.Systematically mistaken memory banks may reinforce erroneous consensus, motivating auditing before consequential use.

A.1 Benchmark Details

The evaluation covers reasoning, domain knowledge, and factual judgment benchmarks, using single-agent, self-consistency, MAD, MAD-M2, and retrieval-augmented baselines. Baseline configurations are designed to compare methods under matched or explicitly described conditions.

  • Benchmarks: The benchmark suite includes MATH500 for mathematical reasoning, Economics and Engineering from MMLU-Pro for domain knowledge, and TruthfulQA for factual judgment.The four benchmarks cover both reasoning and knowledge-intensive tasks.
  • Benchmarks: MATH500 evaluation uses 134 Level 5 problems, with the remaining problems used for memory construction.The selected test problems represent the dataset’s highest difficulty level.
  • Benchmarks: Economics and Engineering use random 3:1 train-test splits, yielding 633/211 and 727/242 questions respectively.Economics contains 844 questions, while Engineering contains 969 questions, both with 10-option multiple-choice formats.
  • Benchmarks: TruthfulQA evaluates answers to adversarial questions designed to elicit misconceptions and falsehoods using a multiple-choice format.Questions with four to nine candidate options yield 498 training and 166 test examples.
  • Baselines: Baselines include CoT, Self-Consistency, MAD, MAD-M2, and ICL-CoT, with Self-Consistency matched to nine reasoning paths against three agents over three rounds.ICL-CoT retrieves from the same memory bank as R2-MAD but uses task similarity because a single agent has no debate state.

A.3 Implementation Details

Experiments use fixed multi-agent configurations, offline memory construction, and debate-state retrieval at test time. Additional analyses compare self-consistency budgets and MAD-M2 masking strategies across models and benchmarks.

  • Memory Construction: Memory is constructed offline from full three-agent, three-round debates on training data, excluding round-0 responses and storing later-round cases per agent.Agents continue through the maximum rounds to diversify experiences and avoid premature consensus.
  • Debate and Retrieval Procedure: At each test-time round after the first, retrieval embeds the debate state and ranks the agent’s memory bank by cosine similarity to retain the top-3K cases.The initial filtering reduces the computational cost of subsequent retrieval and processing.
  • Debate and Retrieval Procedure: Confidence annotations are added according to consensus-ratio thresholds, while early stopping occurs at unanimous consensus and final answers use majority voting.The consensus ratio is computed from extracted answers and the most frequent answer.
  • Additional Analyses: MAD-M2 comparisons cover subjective and objective masking across four benchmarks and three models, while self-consistency is evaluated with different path counts.The main comparison uses nine paths to match the generation-call budget of three agents over three rounds.

B.3 Additional Shared Misconception Results

On Qwen3-8B, R2-MAD remains effective in the shared misconception regime and leads on three benchmarks. Economics is especially difficult when a strong model initially converges on misconceptions.

  • Additional Shared Misconception Results: Figure 4 reports final accuracy for different debate methods with Qwen3-8B on instances where standard MAD had a majority incorrect at round 0.This subset directly represents the shared misconception setting.
  • Additional Shared Misconception Results: R2-MAD leads on three benchmarks for Qwen3-8B under the shared misconception regime.This extends the shared-misconception analysis previously reported for Qwen2.5-7B-Instruct and Gemma-3-4B-IT.
  • Additional Shared Misconception Results: All three debate methods achieve below 10% accuracy on Economics for Qwen3-8B.The passage characterizes these errors as particularly resistant to correction when the stronger model converges on misconceptions at the prior stage.

B.4 Additional Flip and Recovery Analysis

Confidence weighting improves stance stability under shared misconception, reducing correct-to-wrong flips and increasing recovery across models and benchmarks. R2-MAD also outperforms a single-agent baseline using the same retrieved memory, indicating that debate-state retrieval and peer reweighting add value beyond static exemplars.

  • Flip and recovery rates: Confidence weighting lowers correct-to-wrong flip rates on three of four benchmarks for both Qwen3-8B and Gemma-3-4B-IT.On Economics, the flip rate falls from 0.333 to 0.077 for Qwen3-8B and from 0.385 to 0.196 for Gemma-3-4B-IT.
  • Flip and recovery rates: Across three models, flip and recovery rates improve in nine of twelve model-benchmark pairs under confidence weighting.The largest table differences consistently favor the full model.
  • Comparison with ICL-CoT: R2-MAD matches or exceeds ICL-CoT in all four settings and is strictly better in three.Both methods use the same memory bank and number of retrieved cases, but R2-MAD retrieves by debate state and reweights peer influence.
  • Comparison with ICL-CoT: ICL-CoT does not reliably improve over naive CoT, falling below it on Economics with Qwen2.5-7B-Instruct and TruthfulQA with Gemma-3-4B-IT.Thus, access to related question-answer pairs alone is not sufficient.

B.6 Results on Larger Models

R2-MAD retains an advantage on stronger models, achieving the best average accuracy on both Llama-3.3-70B-Instruct and GPT-4o-mini. However, gains are concentrated on Economics and Engineering, with mixed results on TruthfulQA.

  • Larger-model evaluation: R2-MAD achieves the best average accuracy on both Llama-3.3-70B-Instruct and GPT-4o-mini.The evaluation covers both open-source and commercial stronger models.
  • Larger-model evaluation: 0.702 to 0.717 is the improvement over MAD for Llama-3.3-70B-Instruct.The corresponding GPT-4o-mini improvement is from 0.648 to 0.661.
  • Benchmark-level pattern: R2-MAD gains concentrate on Economics and Engineering, while TruthfulQA is matched on Llama-3.3-70B-Instruct and slightly lower than MAD on GPT-4o-mini.The per-benchmark pattern is therefore more mixed than in the smaller-model results.
  • Benchmark-level pattern: The authors suggest stronger LLMs may already resolve much of what memory supplies on knowledge-saturated tasks, leaving less headroom for prior correction.This is presented as a plausible reading of the benchmark-level differences.

C.2 Proof for Proposition 4.2

The proof analyzes how memory correction and confidence weighting alter debate log-odds under a shared misconception. It shows that confidence weighting slows majority dominance, while the two mechanisms contribute additively.

  • Proof setup: Under a shared misconception, the proof assumes m agents share the most-likely erroneous concept θ′ and analyzes their majority contribution.The analysis follows the shared-misconception setup in which the dominant concept is repeated across agents.
  • Majority dominance: As m grows, non-θ′ terms vanish and only the θ = θ′ summand survives in the normalized posterior expression.This establishes convergence toward θ′-dominance under the majority-response assumption.
  • Confidence weighting: Under confidence weighting with α < 1, competing-term decay changes from r^m to r^(αm), slowing convergence to θ′-dominance from O(m) to O(αm).The effective majority size becomes m_eff = m/α > m for the same degree of dominance.
  • Joint decomposition: The joint log-odds decomposition separates a memory term depending on retrieved experiences from a confidence term depending on weights and peer responses.The proof states that neither mechanism interferes with the other.
  • Joint improvement: Theorem C.3 models expected accuracy improvement through a nonnegative memory lift δ_mem and a confidence lift proportional to (1 − α)mκ.The confidence contribution grows linearly with the number of misconceived agents.
Loading 2609.03619v1…