Source-linked AI summary
Voting or Consensus? Decision-Making in Multi-Agent Debate
Lars Benedikt Kaesberg, Jonas Becker, Jan Philip Wahle, Terry Ruas, Bela Gipp
TL;DR
Multi-agent debate research lacks controlled evidence about how decision protocols affect different tasks. This paper isolates the decision mechanism while comparing seven protocols across knowledge and reasoning tasks, finding task-specific advantages for consensus and voting and proposing diversity-enhancing methods.
Problem
Prior work often changes decision protocols alongside other discussion parameters, leaving their task-specific effects on multi-agent collaboration insufficiently established.
Method
The study systematically evaluates three consensus and four voting protocols across knowledge and reasoning tasks while holding the broader setup consistent, and proposes AAD and CI to increase answer diversity.
Results
Voting protocols improve reasoning-task performance by 13.2%, while consensus protocols improve knowledge-task performance by 2.8%; AAD and CI improve performance by up to 3.3% and 7.4%, respectively.
Takeaways & Limitations
The findings support choosing consensus for knowledge tasks and voting for reasoning tasks, while using AAD or CI to improve answer diversity.
Takeaways & Limitations
The study uses sampled dataset subsets because multi-agent debates are computationally expensive, which can introduce variance despite controls across three independent runs.
Abstract
from arXiv · showhide
Much of the success of multi-agent debates depends on carefully choosing the right parameters. The decision-making protocol stands out as it can highly impact final model answers, depending on how decisions are reached. Systematic comparison of decision protocols is difficult because many studies alter multiple discussion parameters beyond the protocol. So far, it has been largely unknown how decision-making influences different tasks. This work systematically evaluates the impact of seven decision protocols (e.g., majority voting, unanimity consensus). We change only one variable at a time - the decision protocol - to analyze how different methods affect the collaboration between agents and measure differences in knowledge and reasoning tasks. Our results show that voting protocols improve performance by 13.2% in reasoning tasks and consensus protocols by 2.8% in knowledge tasks compared to other decision protocols. Increasing the number of agents improves performance, while more discussion rounds before voting reduce it. To improve decision-making by increasing answer diversity, we propose two new methods, All-Agents Drafting (AAD) and Collective Improvement (CI). Our methods improve task performance by up to 3.3% with AAD and up to 7.4% with CI. This work demonstrates the importance of decision-making in multi-agent debates beyond scaling.
1 Introduction
This study examines how decision protocols shape multi-agent collaboration across knowledge and reasoning tasks. It systematically compares voting and consensus approaches while introducing methods to increase answer diversity.
- Prior studies rarely isolate how decision-making affects LLM collaboration and problem-solving.
- The study systematically compares three consensus and four voting protocols across three knowledge and three reasoning tasks.
- 13.2% higher reasoning-task performance is obtained with voting protocols, while consensus protocols improve knowledge-task performance by 2.8%.
- The paper identifies decision-protocol comparison and scaling analysis as central contributions beyond simply increasing agents or communication rounds.
2 Related Work
Prior multi-agent debate research has used consensus or voting protocols without systematically comparing them across tasks. This paper addresses that gap while controlling other experimental parameters.
- Multi-agent debate research has implemented human decision protocols, including consensus and voting, but has often studied only one protocol class.
- Exchange-of-Thought uses consensus for iterative answer refinement but does not compare consensus with other decision protocols.
- Prior studies often change multiple parameters alongside decision protocols, making protocol-specific effects difficult to quantify.
- This work evaluates seven voting and consensus approaches on both knowledge and reasoning datasets to identify task-based advantages.
3 Methodology
The methodology builds multi-agent debates from discussion, decision, and response-generation components, then evaluates seven protocols across six task datasets. It also defines consensus and voting mechanisms and uses response generators to shape agent interaction.
- Experimental setup: Each debate combines a discussion paradigm, a decision protocol, and an agent response generator, using three automatically generated expert personas.
- Experimental setup: The experiments use Llama 3 8B and 70B, with all agents within a discussion sharing the same base model.
- Agent prompts: The response generators are Simple, Critical, and Reasoning, respectively prompting neutral responses, critique of prior answers, or sharing of reasoning paths.
- Decision protocols: Consensus protocols select a shared answer after reaching majority, supermajority, or unanimity agreement.
- Decision protocols: Voting protocols present multiple solutions in parallel and have all agents vote on the final solution.
- Evaluation: The study evaluates six datasets: three knowledge tasks and three reasoning tasks, using subsets because of computational constraints.
4 Experiments
The experiments compare seven decision protocols across knowledge and reasoning tasks while varying only the decision mechanism. Results show task-dependent protocol effects, trade-offs between agents and discussion rounds, and benefits from methods that increase answer diversity.
- Experimental design: The study compares seven decision protocols across three knowledge and three reasoning tasks while changing only the decision protocol.The protocols comprise four voting methods and three consensus methods, evaluated against baseline results.
- Decision protocols: Consensus protocols outperform voting protocols on knowledge tasks, whereas voting protocols perform better on reasoning tasks.The reported average improvements over the other protocol family are 2.3% on MMLU, 4.9% on MMLU-Pro, and 1.3% on GPQA for consensus; voting leads by 13.1% on SQuAD 2.0, 0.2% on StrategyQA, and 26.4% on MuSR.
- Decision protocols: Voting achieves a higher overall SQuAD 2.0 F1-score than consensus, while consensus is more effective for unanswerable samples.Overall F1 is 56.7% for voting and 43.6% for consensus; stricter consensus methods perform better on unanswerable samples.
- Practical trade-offs: Multi-agent debate requires substantially more computation than the CoT baseline, with consensus using approximately five times and voting ten times the resources.Consensus often reaches decisions faster, while voting can require more rounds and compute.
- Agents and discussion rounds: More agents generally improve accuracy, while additional discussion rounds slightly reduce it.The authors therefore recommend scaling the number of agents rather than turns to create a larger knowledge base.
- Agents and discussion rounds: Challenge-based extra discussion rarely improves answers and can produce worse solutions on reasoning tasks.Providing discussion history reduces challenge rates by 10%, and challenged answers become worse in 3% of reasoning-task cases.
- Answer diversity: AAD and CI improve performance by increasing answer diversity, whereas reasoning or critical response generators do not reliably improve results.AAD improves performance by 3.3% on average and CI by 7.4%; the reasoning generator decreases answer diversity and task performance.
5 Conclusion
The study shows that decision protocols materially shape multi-agent debate performance, with task-dependent advantages and benefits from increasing answer diversity. It recommends matching protocols to task type, scaling agents rather than rounds, and using AAD or CI to diversify answers.
- Answer diversity: AAD requires every agent to contribute draft ideas initially, whereas CI limits communication until agents exchange proposed solutions after each turn.Both methods are designed to increase answer diversity through more independent reasoning.
- Protocol effects: Voting outperforms consensus by up to 13.2% on reasoning tasks, while consensus outperforms voting by up to 2.8% on knowledge tasks.The study evaluated three knowledge and three reasoning tasks.
- Answer diversity: AAD improves performance by up to 3.3%, and CI improves it by up to 7.4% over multi-agent debate baselines.Relative to a single-model CoT baseline, the gains are 6.1% and 10.2%, respectively.
- Recommendations: The study recommends voting for reasoning tasks, consensus for knowledge tasks, scaling agents instead of discussion rounds, and using AAD or CI to increase answer diversity.These recommendations summarize the reported task-specific protocol effects and scaling findings.
Limitations
The study’s computational demands required sampled dataset subsets, and its protocol-focused design left prompt and persona effects incompletely explored. Agents also often converged on similar responses despite efforts to increase diversity.
- Computational constraints: The study uses sampled dataset subsets because multi-agent debates can require hundreds of forward passes per model.Sampling used a 95% confidence level and three independent runs to estimate variance.
- Answer diversity: Agents often converged on similar responses despite efforts to improve answer diversity, motivating more advanced techniques for independent solutions.This limits how fully the proposed diversity methods separate agent reasoning paths.
- Unexamined parameters: The study acknowledges that prompt design and persona selection are relevant parameters, but omits a dedicated persona ablation because of computational cost.Initial explorations with more diverse prompts did not yield consistent improvements over the simpler design.
A.1 Task Performance with Llama 3 70B
With Llama 3 70B, most results are only slightly above baseline, and multi-agent discussions surpass the CoT baseline in only a few cases. SQuAD 2.0 and StrategyQA show the largest gains.
- Overall performance: Llama 3 70B performs much better overall than Llama 3 8B, but multi-agent discussions exceed the CoT baseline in only a few cases.The larger model does not show the large performance gain observed with the smaller model.
- Task-level results: SQuAD 2.0 and StrategyQA show the largest performance gains and can outperform the CoT baseline.The passage reports this pattern for the Llama 3 70B results.
- Decision rounds: Most voting protocols reach a final answer in the first round in which voting is allowed, while ranked and cumulative voting sometimes require another round.Simple voting has the highest agreement rate among the described voting protocols.
- Decision rounds: Approval voting reaches a decision in about 27% of cases, with about 14% requiring another round and the remainder canceled after the fifth round.The passage attributes this difficulty to ties caused by models voting for many answers.
B Additional Details on Datasets
The experiments cover diverse knowledge and reasoning datasets, use statistically motivated subsets and repeated runs, and report protocol performance with standard deviations. The sampling strategy applies a finite-population correction when appropriate.
- Dataset selection: The evaluation separates three knowledge datasets—MMLU, MMLU-Pro, and GPQA—from three reasoning datasets—StrategyQA, MuSR, and SQuAD 2.0.The knowledge tasks retain domain knowledge demands, while the reasoning tasks emphasize multistep reasoning or textual comprehension.
- Sampling strategy: Dataset subsets are selected to provide a 95% confidence level with a 5% margin of error because multi-agent discussions are expensive.The sampling approach follows prior multi-agent debate studies.
- Sampling strategy: For finite datasets, the sample-size calculation applies a finite population correction using N, the total number of samples.The specific resulting sample sizes are listed in Table 3.
- Evaluation procedure: Each dataset is evaluated three times to obtain standard deviations and provide performance estimates across repeated evaluations.The reported task-performance figure also uses standard deviation over three runs.
- Evaluation procedure: The experiments use the Multi-Agent Large Language Models framework for the multi-agent evaluation.The framework is identified as MALLM.
C.1 Architecture Overview
The architecture organizes multi-agent discussions around discussion paradigms, decision protocols, and agent response generation. These components determine how agents communicate, refine answers, and select the final solution.
- Architecture: MALLM combines discussion paradigms, decision protocols, and agent response generation into each multi-agent discussion.Discussions use automatically generated expert personas and multiple turns to incorporate diverse viewpoints.
- Discussion Paradigms: Discussion paradigms control agent communication order and which answers are visible to each agent.Built-in paradigms are static, with predefined turn orders.
- Discussion Paradigms: Memory exposes all previous answers, whereas Relay exposes only the previous agent’s answer.Both paradigms use sequential agent turns.
- Discussion Paradigms: Report and Debate use a moderator, but Debate permits communication for predefined rounds before forwarding reasoning to the moderator.The moderator is the only agent with access to all messages in Report.
- Discussion Paradigms: Collective Refinement has agents generate answers independently and then refine them using all other agents’ responses each round.There is no turn order, and all agents have the same information in each round.
- Decision Protocols: Consensus protocols require sequential agreement, while voting protocols discuss for a fixed number of turns before selecting among final solutions.Majority, supermajority, and unanimity consensus require 50%, 66%, or all agents to agree, respectively.
- Decision Protocols: Simple, Ranked, Cumulative, and Approval Voting select answers using votes, rank sums, allocated points, or approval counts.Cumulative Voting allows each agent to distribute up to 25 points across possible answers.
D Solution Counting Ablation
The solution-counting ablation compares frequency-based answer selection with prompted consensus and voting protocols across four datasets. Solution counting is feasible but generally performs worse, reinforcing the value of explicit decision protocols.
- Ablation Setup: Solution counting selects the answer agents provide most frequently as the final evaluation answer.The ablation uses MMLU, MMLU-Pro, GPQA, and StrategyQA, where direct counting of multiple-choice answers is straightforward.
- Comparison: Solution counting is compared with prompted decision protocols across four datasets using Accuracy ± std.Bold values indicate the highest result per dataset.
- Findings: Solution counting generally underperforms consensus- and voting-based decision protocols.Consensus methods outperform it on MMLU-Pro and GPQA, while voting improves performance on StrategyQA.
- Findings: The ablation reinforces the main finding that explicit decision protocols add value beyond simply counting agents’ final answers.The comparison covers both knowledge-based and reasoning-task examples.
G.1 Successfull Voting Discussion
The successful voting discussion shows agents producing mostly convergent answers and voting for a common candidate. The MALLM solution matches the real answer: Southampton Philharmonic Choir.
- Outcome: The MALLM solution is Southampton Philharmonic Choir, matching the real solution.The answer is supported by the context’s mention of the choir among organizations hosting classical concerts at the Guildhall.
- Discussion: Agents’ final answers include Southampton Philharmonic Choir, Southampton Choral Society, or both.The Music Connoisseur and Music Journalist independently provide Southampton Philharmonic Choir.
- Simple Voting: All three agents vote for the Music Journalist’s answer in the simple-voting discussion.The Music Connoisseur, Choir Conductor, and Music Journalist each select that answer.
- Approval Voting: Approval Voting records approvals for answers from the Information Architect and Music Critic, with the Information Architect’s answer approved by multiple agents.The Local Music Event Coordinator approves the Information Architect’s answer.
- Ranked Voting: Ranked Voting produces identical rankings from all three voters, placing the Music Journalist first.Each voter ranks the Music Journalist above the Local Music Expert and Concert Promoter.
G.2 Agents Tricked Discussion
The agents converge on a biodiversity-based answer and elaborate a Public-Private Partnership framework, but the discussion is explicitly described as being tricked by contextual information. The recorded MALLM solution does not match the real solution included in the passage.
- Caveat: The discussion is explicitly characterized as having tricked the agents through information provided in the context.This statement qualifies the apparent agreement and solution quality.
- Outcome: The MALLM solution states that biodiversity provides critical support for drug discovery and economic resources, while the real solution is not included in context.The passage explicitly contrasts the generated solution with the real solution.
- Refinement: Agents improve the proposed Public-Private Partnership framework by adding community engagement, benefit-sharing, traditional-knowledge protection, and capacity building.The Research Biologist explicitly endorses these additions.
- Refinement: The Economist recommends measurable KPIs and monitoring to assess the partnership’s effects on the pharmaceutical sector, communities, and the environment.The proposal is framed as an accountability and evaluation process for responsible bioprospecting.
- Discussion: The agents’ final answers converge on biodiversity or a longer biodiversity-based statement.The Economist, Ethnobotanist, and Research Biologist all select biodiversity-related answers.
G.3 Bad Voting Discussion
The discussion contrasts an initial attempt to apply Czech enclitic placement rules to English with a revised answer recognizing that English placement varies by sentence structure and context. Despite some agents voting for the revised answer, the final multi-agent output included both the revised answer and “[UNKNOWN]” responses, while another discussion produced an unsupported choir answer.
- Initial proposal: The initial proposal transferred Czech’s fixed second-slot enclitic rule to English, despite English having different grammatical characteristics.The Czech rule places enclitics after the first stressed unit, whereas English word order and enclitic placement are more flexible.
- Revised solution: The revised answer places English enclitics near the main verb, usually before adverbs or modifiers, while allowing variation by sentence structure and context.This revision is presented as a more nuanced account than directly applying the Czech rule.
- Voting outcome: Three agents voted for the Linguist’s answer, which stated the revised English enclitic rule.The English Language Teacher and Grammar Analyst otherwise returned “[UNKNOWN]” as their final answers.
- Bad voting example: The multi-agent solution for the choir question named Southampton Choral Society, whereas the real solution was Southampton Philharmonic Choir.A choir-member persona nevertheless endorsed Southampton Choral Society as a choir that had sung at the Guildhall.