Source-linked AI summary
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Andries Smit, Paul Duckworth, Nathan Grinsztajn, Thomas D. Barrett, Arnu Pretorius
TL;DR
Truthful question answering requires a broad set of skills, and this paper benchmarks prompting strategies, including multi-agent debate, while testing agreement modulation. The original MAD implementations do not outperform other ensembling methods, but the authors report that tuning agreement intensity can yield significant gains.
Problem
Truthful question-answering assistants require a broad range of advanced skills.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
MAD may be more sensitive to hyperparameter settings rather than inherently worse than single-agent methods, making agreement intensity a tuning consideration.
Source: §4 · p. 9API-based benchmarking faces variable inference times and model updates, while its financial and time costs limit experiment scale and hinder replication and extension.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Recent advancements in large language models (LLMs) underscore their potential for responding to inquiries in various domains. However, ensuring that generative agents provide accurate and reliable answers remains an ongoing challenge. In this context, multi-agent debate (MAD) has emerged as a promising strategy for enhancing the truthfulness of LLMs. We benchmark a range of debating and prompting strategies to explore the trade-offs between cost, time, and accuracy. Importantly, we find that multi-agent debating systems, in their current form, do not reliably outperform other proposed prompting strategies, such as self-consistency and ensembling using multiple reasoning paths. However, when performing hyperparameter tuning, several MAD systems, such as Multi-Persona, perform better. This suggests that MAD protocols might not be inherently worse than other approaches, but that they are more sensitive to different hyperparameter settings and difficult to optimize. We build on these results to offer insights into improving debating strategies, such as adjusting agent agreement levels, which can significantly enhance performance and even surpass all other non-debate protocols we evaluated. We provide an open-source repository to the community with several state-of-the-art protocols together with evaluation scripts to benchmark across popular research datasets.
1. Introduction
Reliable question answering requires advanced comprehension and expert-knowledge skills, yet there is no consensus on which multi-agent strategy to select. The paper benchmarks prompting strategies and introduces agreement modulation to improve Q&A performance.
1. Introduction
- The authors identify the lack of comparative work and consensus on how to use multiple agents for effective interactive reasoning.They describe this as a prescient research question amid growing use of multi-agent approaches.Source: §1 · p. 1
- The paper benchmarks prompting strategies, including multi-agent debate, for multiple-choice questions across domains and examines accuracy, time, and cost trade-offs.It also provides an open-source suite of single-agent and MAD implementations with a unified API for building and evaluating MAD systems.Source: §1 · p. 1
- A proposed prompting strategy modulates agent agreement during debate and improves Q&A performance for a given model class.The paper presents this as a novel strategy alongside its broader benchmark and open-source implementations.Source: §1 · p. 1
2. Multi-Agent Debate
The section introduces multi-agent debate strategies and contrasts their interaction designs with independent reasoning-path methods. It also describes the agent-level prompts and implementation choices used to compare these approaches.
Debate strategies
- MAD systems differ in how agents exchange answers and histories, including direct collaboration, opposing personas with a judge, and sequential or simultaneous discussion.Society of Minds may summarize answers before later rounds; Multi-Persona uses affirmative and negative agents, with a judge able to end debate early; ChatEval includes three interaction modes.Source: §2 · p. 2
Non-debate strategies
- Independent alternatives include self-consistency, which selects the most frequent answer across sampled reasoning paths, and Ensemble Refinement, which aggregates sampled paths over multiple rounds.Both use multiple API calls; neither is a debate protocol.Source: §2 · p. 2
Prompting choices
- The study implements only question randomization and few-shot chain-of-thought ensembling from Medprompt, omitting its k-nearest-neighbor step because it requires a train/test split.The authors consider that requirement an unfair advantage when other protocols use no training set.Source: §2 · p. 2
- Agent-level prompts range from zero-shot Q&A and chain-of-thought to few-shot examples, Solo Performance Prompting, and few-shot chain-of-thought.The few-shot chain-of-thought prompt combines step-by-step reasoning with five Q&A examples and explanations.Source: §2 · p. 2
3. Experiments
The experiments benchmark prompting strategies across medical and reasoning datasets, then test whether agreement modulation can improve debate performance. MAD’s standard configurations do not consistently outperform non-debate strategies, but tuned agreement can produce strong gains, including a reported GPT-3 state-of-the-art result on MedQA.
Results
- SoM reaches 61% on MedQA, but non-debate Medprompt performs best overall at lower cost, and MAD does not consistently outperform other strategies.The paper also reports that performance depends on hyperparameters and dataset/system configurations.Source: §3 · p. 4
The utility of debate
- Across seven datasets, no protocol dominates; Medprompt performs best overall, while MAD protocols do not outperform other protocols on reasoning datasets.Boxplots show performance variability; Medprompt is relatively consistent except on GPQA, while MAD appears more hyperparameter-sensitive.Source: §3 · pp. 4–5, Figure 2 · p. 5
Is MAD simply sensitive to hyperparameters?
- After category-specific hyperparameter selection, every evaluated system except Multi-Persona outperforms single-agent methods on medical datasets, while single agents lead on non-medical datasets except against Multi-Persona and Self-Consistency.The authors conclude that optimal settings tend to be dataset-specific and that gains over a standard single-agent approach are not assured on new datasets.Source: §3 · p. 5, Figure 3 · p. 6
Measuring the performance boost from debating
- Debate changes accuracy unevenly: ChatEval improves modestly, SoM substantially, and Multi-Persona declines relative to the first agent’s initial answer.The authors attribute Multi-Persona’s decline to the devil agent’s designed tendency to contradict, even when the initial answer is correct, and identify initial agreement as an important hyperparameter.Source: §3 · pp. 5–6, Figure 4 · p. 6
Improving MAD via agreement modulation
- Prompted agreement affects Multi-Persona accuracy and first-round agreement, but the preferred direction varies by dataset: higher agreement benefits MedQA and PubMedQA, while CIAR shows the reverse pattern.The method adjusts agreement at debate outset through an instruction specifying how often agents should agree.Source: §3 · p. 6, Figure 5 · p. 7
- On a 376-question USMLE subset, agreement modulation improves Multi-Persona by approximately 15% and SoM by approximately 5%, while ChatEval is hardly affected.The modulation uses prompted agreement intensity; the paper applies 90% intensity to Multi-Persona on full MedQA afterward.Source: §3 · p. 6
- The paper reports a new state-of-the-art result for GPT-3, highlighted in Figure 1 by the red cross symbol; the cited passage does not specify the experimental setting.Source: §3 · p. 7
Analysis of debating behaviour
- SoM has the largest area in the debate-behavior plots, but the authors emphasize balanced performance across dimensions rather than excellence on only one metric.The analysis covers consensus accuracy, independent correct answers, answer changes, and diversity of initial answers.Source: §3 · p. 7, Figure 7 · p. 8
4. Conclusions
With their original implementations, MAD approaches do not outperform methods such as Medprompt and self-consistency, but the authors attribute this in part to greater hyperparameter sensitivity. Tuning agreement intensity substantially improved Multi-Persona, while API-based benchmarking carries cost and reproducibility constraints.
4. Conclusions
- Original MAD implementations do not outperform other ensembling methods such as Medprompt and self-consistency.The authors suggest this may reflect MAD protocols' greater sensitivity to hyperparameters rather than an inherent disadvantage of debate.Source: §4 · p. 9
- Tuning agreement intensity moved Multi-Persona from the worst-performing protocol to the best-performing, ahead of Medprompt and self-consistency.The authors describe this as a significant performance gain from prompt-based manipulation of agent agreement.Source: §4 · p. 9
- MAD typically requires more API calls and tokens, increasing total running cost.Source: §4 · p. 9
- Public-LLM API use exposes experiments to variable inference times and unforeseen model updates, while benchmarking costs constrain scale and replication.The authors suggest future work could use open-source models and in-house infrastructure.Source: §4 · p. 9
Impact Statement
The impact statement describes an evaluation of MAD and other prompting methods on question-answering datasets, alongside an open-source toolkit. It also cautions that medical QA systems may give incorrect, overly confident answers with potential consequences for patient care.
Impact Statement
- The study evaluates MAD strategies and other prompting methods on question-answering datasets and provides an open-source toolkit.Source: Impact Statement · p. 9
- Medical question-answering systems can produce unforeseen incorrect predictions and undue confidence in erroneous responses.Source: Impact Statement · p. 9
- Such inaccuracies could contribute to misinformation or misdiagnosis and potentially affect patient care and outcomes.Source: Impact Statement · p. 9
A.1. Extended results on additional datasets
The extended results cover seven datasets and present system performance through accuracy comparisons with time, token use, and cost, alongside box plots summarizing performance.
A.1. Extended results on additional datasets
- The extended results cover MedQA, PubMedQA, MMLU, CIAR, GPQA, CosmosQA, and Chess.Source: §A.1 · p. 12
- For each scenario, the plots compare accuracy with average answering time, average tokens per question, and total USD cost.Source: §A.1 · pp. 12, 15
- Box plots summarize the performance of each strategy across the reported scenarios.Source: §A.1 · p. 12
A.2. Table of Experiments
Table 3 records experiment configurations across six datasets, including debate and agent prompt names. The complete configurations are listed there, with prompt descriptions in Appendix A.6.
A.2. Table of Experiments
- Table 3 includes experiment configurations and identifies the debate and agent prompts used.Source: §A.2 · p. 16, Table 3 · p. 18
- The experiment table lists MedQA, PubMedQA, MMLU, CosmosQA, CIAR, and GPQA as datasets, with Score and Cost columns.Source: §A.2 · p. 17, Table 3 · p. 18
- Table 3 is identified as the complete table of experiment configurations, including debate and agent prompt names.Full prompt descriptions are in Appendix A.6.Source: §A.2 · p. 16, Table 3 · p. 18
A.3. Additional Debate Metrics
The additional debate metrics track agreement, correctness, answer changes, and debate length, along with the diversity of initial answers.
A.4. Additional Debate Metrics
- Final-round consensus measures agent agreement, while correctly parsed consensus excludes agents with incorrectly parsed answers.Source: §A.3 · p. 19
- Debate correctness measures include whether any agent answered correctly at least once and the number of agents changing answers, including a correctly parsed variant.Source: §A.3 · p. 19
- The metrics also track average debate rounds and the average number of unique first answers, including a correctly parsed variant.Source: §A.3 · p. 19
A.4. Additional Agent Metrics
The additional agent metrics describe answer accuracy and behavior, parsing, prompt and response usage, and time and cost.
A.5. Additional Agent Metrics
- Agent outcome metrics include the percentage answered correctly, answer changes, reliance on another agent, and being bullied into changing an answer.Source: §A.4 · p. 19
- Parsing and answer-history metrics track incorrectly parsed answers, correct rounds, the first correct round, and unique answers throughout a debate.Source: §A.4 · p. 19
- Prompt and response measures include token totals and averages, response length, and messages removed when the prompt limit is reached.Source: §A.4 · p. 19
- Resource measures include average round cost, cost per question, and time per question; the metrics also record each agent's engine and name.Source: §A.4 · p. 19
A.5. Debate Prompts
The appendix lists debate-level prompts for the strategies, including instructions for debaters, summarizers, aggregation, and moderation. Their wording specifies how agents use prior answers and conclude their responses.
A.5. Debate Prompts
- CE MAD assigns debaters to seek the correct answer without requiring full agreement and has a summarizer relay the debate's main points to the next round.Source: §A.5 · p. 20
- ER MAD CoT prompts step-by-step medical question reasoning, while its aggregation prompt asks agents to select useful correct arguments and avoid wrong reasoning.Source: §A.5 · p. 20
- ER MAD also has a multiple-choice prompt and an aggregation prompt that presents prior student reasonings as potentially correct or incorrect.Source: §A.5 · p. 20
- SoM MAD uses separators to introduce other agents' solutions and the main points discussed so far; its suffix asks for an updated answer ending with a capital option letter.Source: §A.5 · p. 20
- MP MAD prompts a moderator to evaluate two debaters' answers each round, while Medprompt uses a general medical multiple-choice answering instruction.Source: §A.5 · p. 20
A.6. Agent Prompts
The appendix lists agent-level prompts used by the strategies, with prompt names linked to Table 3. Its examples show role instructions and debate formats.
A.6. Agent Prompts
- Multi-Persona prompts an affirmative “angel” and negative “devil” to answer before a judge chooses a final solution.The judge may end the debate early when satisfied with the answers.Source: §A.6 · p. 21
- The Universal Mode prompt asks a moderator to assess both sides, decide whether one answer is preferred, and continue debating if not.It specifies a JSON response containing the preference, supported side, reason, and answer.Source: §A.6 · p. 21
- SPP prompts begin by identifying participants and their profiles, then direct them to collaborate over multiple rounds with critical comments and suggestions.The Expert variant limits participants to an AI assistant or expert and asks the expert to provide detailed suggestions when needed.Source: §A.6 · pp. 21–22
- The poem example assigns poet, computer-scientist, and child roles to guide form, subject matter, and age-appropriate wording.The example shows the child requesting simpler vocabulary and the assistant revising the poem.Source: §A.6 · pp. 21–22
- The arithmetic example presents a multi-round exchange in which an expert critiques proposed solutions to the 24 game.The task asks the solver to use each of 6, 12, 1, and 1 once.Source: §A.6 · pp. 21–22