Source-linked AI summary
Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, Yangqiu Song
TL;DR
The paper asks whether multi-agent discussions are genuinely superior to strong single-agent prompting for reasoning. It addresses this through systematic benchmark experiments and introduces CMD, a group-discussion framework. The results show comparable performance with strong prompts, discussion gains without demonstrations, and improvement of weaker agents by stronger ones.
Problem
The paper reevaluates the claim that multi-agent discussions outperform single agents on reasoning tasks, where prior discussion frameworks commonly rely on demonstrations.
Method
The paper conducts systematic experiments across standard reasoning benchmarks and introduces CMD, a multi-agent framework inspired by human group discussion.
Results
A strong single agent matches multi-agent discussion performance with robust prompts, while discussion frameworks outperform single agents on most tasks without demonstrations.
Takeaways & Limitations
Multi-agent discussion is most useful when demonstrations are unavailable, and stronger LLM agents can improve weaker agents during interaction.
Takeaways & Limitations
The study simplifies each AI agent to an LLM session and mainly examines a limited set of LLMs and tasks.
Abstract
from arXiv · showhide
Recent progress in LLMs discussion suggests that multi-agent discussion improves the reasoning abilities of LLMs. In this work, we reevaluate this claim through systematic experiments, where we propose a novel group discussion framework to enrich the set of discussion mechanisms. Interestingly, our results show that a single-agent LLM with strong prompts can achieve almost the same performance as the best existing discussion approach on a wide range of reasoning tasks and backbone LLMs. We observe that the multi-agent discussion performs better than a single agent only when there is no demonstration in the prompt. Further study reveals the common interaction mechanisms of LLMs during the discussion.
1 Introduction
This paper reexamines whether multi-agent discussion reliably outperforms strong single-agent prompting for reasoning. It introduces CMD and finds that discussion advantages depend on prompt demonstrations and model strength.
- Motivation: LLMs remain vulnerable to hallucination and reasoning perspective mistakes despite strong language understanding and generation abilities.
- Motivation: Existing discussion frameworks use multiple LLM agents and commonly rely on task-specific demonstrations to improve reasoning.
- Evidence: On FOLIO-wiki, a single agent with a demonstration reaches a performance upper bound similar to all compared discussion frameworks.
- Approach: The paper proposes CMD, a group-discussion framework, and evaluates prompt components across ECQA, GSM8k, and FOLIO-wiki with varied pipeline configurations.CMD is inspired by human group discussion and is designed to broaden comparisons between single-agent and discussion settings.
- Findings: Multi-agent discussion generally outperforms a single agent without demonstrations, but strong prompts can make single agents comparable to discussion systems.
- Findings: Stronger LLMs can improve weaker agents during multi-LLM discussions, while discussion errors include judge mistakes and wrong-answer propagation.
2 Preliminary
Multi-agent discussion is an autonomous dialogue among LLM-powered agents guided by prompts and discussion rules. The paper motivates CMD as a group-discussion framework that organizes communication and final decisions.
- What is Multi-Agent Discussion?: Multi-agent discussion is an interactive setup in which multiple LLM-powered agents autonomously engage in dialogue about a given topic.Each agent receives a prompt specifying background knowledge and behavioral guidance.
- Discussion Pipeline: The discussion pipeline comprises a Kick-start prompt, agents, an algorithm, and a result.The algorithm specifies rounds, decision-making procedures, and discussion structure.
- Existing Frameworks: Existing frameworks include Debate, MAD, and ReConcile, while Debate uses turn-based exchanges whose responses are added to later inputs.
- CMD Motivation: Prior frameworks had not addressed group discussion, motivating the proposed CMD framework.
3 CMD: Conquer-and-Merge Discussion
CMD is a new multi-agent framework designed around group discussion, synchronized message passing, voting, and final decisions. It organizes agents into groups that share information differently within and across groups before voting.
- CMD addresses the lack of group-discussion-based frameworks by simulating a human group discussion process.
- Each CMD agent uses an LLM to generate a viewpoint and explanation over multiple discussion rounds while maintaining response history.
- CMD uses a message-passing algorithm to synchronize which agents receive each message across discussion architectures and LLM protocols.
- CMD proceeds through group discussion, voting, and a final decision stage that is typically needed only to resolve ties.
- Within groups, agents access peers’ answers and explanations, whereas across groups they see only answers before the voting stage.
4 Experimental Setups
The experiments compare single-agent reasoning with four multi-agent frameworks across three reasoning benchmarks and several LLM backbones. Accuracy is evaluated under controlled discussion-round and prompting conditions.
- The study contrasts a single-agent setup with Debate, MAD, ReConcile, and CMD under varied prompt conditions.
- Experiments primarily use ChatGPT-3.5, Gemini Pro, and Bard as the underlying LLMs.
- All multi-agent frameworks use a maximum of 3 discussion rounds, and accuracy measures performance across tasks.
- Evaluations use ECQA for commonsense reasoning, GSM8K for mathematical word problems, and FOLIO-wiki for symbolic and natural-language deductive reasoning.
5 Experiments on Single LLM
Experiments with ChatGPT-3.5 examine prompting, cross-task performance, and discussion errors. Strong demonstrations make single-agent performance comparable to discussion frameworks, while discussions generally help more without demonstrations.
- The experiments first analyze FOLIO-wiki, then extend evaluation to GSM8K and ECQA before studying discussion mistakes.
- Single-agent prompts vary detailed question descriptions, answer-format instructions, and task-specific demonstrations.
- Demonstrations significantly improve single-agent performance, bringing a well-supported agent on par with discussion frameworks on FOLIO-wiki.
- With demonstrations, discussion frameworks and single-agent settings have comparable average performance across the tested tasks.
- Without demonstrations, CMD generally surpasses single-agent settings on average and on most individual tasks.
- MAD is the least effective tested framework, while divergent disagreement can introduce irrelevant information that complicates decision-making.
- Multi-agent discussions can reach incorrect conclusions when a single agent answers correctly, through judge mistakes or wrong-answer propagation.
- A judge may select an incorrect verdict during disagreement, while wrong-answer propagation can shift initially correct agents toward an incorrect consensus.
6 Experiments on Multiple LLMs
Across multiple LLMs, the experiments compare single-agent configurations with ReConcile and CMD under prompts with and without demonstrations. Discussion frameworks match a strong demonstrated single agent, while discussion without demonstrations performs better and stronger models support weaker agents over rounds.
- Experimental setup: Three single-agent configurations are compared with ReConcile and CMD across all tasks, using prompts with and without demonstrations.CMD uses six agents arranged into two groups of three, with ChatGPT-3.5, Gemini Pro, and Bard represented in each group.
- Stronger-model support: The results are presented as round-level performance for each LLM in multi-agent discussions on the FOLIO-wiki dataset.
- Results with demonstrations: With demonstrations, discussion frameworks perform comparably to a single agent powered by Gemini Pro.This supports the conclusion that a strong LLM and well-designed prompt can rival multi-agent discussion.
- Results without demonstrations: Without demonstrations, multi-agent discussions outperform single-agent setups across the multi-LLM experiments.CMD and ReConcile show similar performance when both lack demonstrations and use the same LLMs.
- Stronger-model support: Bard is least effective as a single agent, yet weaker agents such as Bard and ChatGPT-3.5 improve over rounds with support from Gemini Pro.Gemini Pro helps bridge knowledge and reasoning gaps, although its own performance briefly drops in the second round before recovering.
7 Related Work
Related work spans prompt-engineering strategies for improving single-LLM reasoning and interactive multi-agent discussion frameworks. This paper focuses on how prompting contents affect discussion through its CMD group-discussion framework.
- Background: Recent LLM development has produced models with differing reasoning capabilities through varied training sources and parameter sizes.Examples include OPT, LLaMA, BLOOM, and PaLM, while Gemini Pro extends LLM capabilities to multimodality.
- 7.1 Prompting LLM for Reasoning: Prompt engineering has produced numerous strategies for improving LLM reasoning, including CoT, Auto-CoT, Active-Prompt, ToT, GoT, and re-reading prompts.These methods structure, diversify, automate, or revisit reasoning processes in different ways.
- 7.2 Multi-agent Discussion for Reasoning with LLMs: Multi-agent discussion uses multiple LLM-powered agents to collectively discuss and reason about problems interactively.The area remains under exploration compared with the extensive work on improving single-LLM reasoning.
- 7.2 Multi-agent Discussion for Reasoning with LLMs: Existing discussion frameworks include symmetric Debate and asymmetric mechanisms such as MAD and ReConcile.MAD assigns debater and judge roles asymmetrically, while ReConcile involves different LLMs.
- 7.2 Multi-agent Discussion for Reasoning with LLMs: CMD differs from prior work by examining the effects of prompting contents within a defined multi-agent group-discussion framework.
8 Conclusion
The conclusion reexamines whether multi-agent discussion is superior to a single agent through systematic benchmark experiments. It reports comparable performance for strong single agents, advantages for discussion without demonstrations, and improvement of weaker agents through stronger models.
- 8 Conclusion: The paper reexamines the claim that multi-agent discussions are superior to single agents in reasoning tasks through systematic experiments.
- 8 Conclusion: A single agent with a strong prompt and strong LLM achieves comparable performance with multi-LLM multi-agent discussions.
- 8 Conclusion: Without demonstrations, multi-agent discussion frameworks outperform single agents on most tasks.
- 8 Conclusion: As discussion progresses, stronger LLMs can enhance the performance of agents powered by weaker LLMs.
9 Ethical Considerations
The study reports using public datasets and official APIs under stated ethical-use practices. It says no personal data was processed and API usage complied with applicable policies.
- 9 Ethical Considerations: The study uses publicly available datasets designed for academic research and processes no personal data.
- 9 Ethical Considerations: LLM access uses official APIs in compliance with API usage policies, fair-use standards, and intellectual-property requirements.
- 9 Ethical Considerations: The authors state that they provide detailed experimentation transparency.
10 Limitations
The study identifies limitations in agent complexity, task diversity, and the range of language models evaluated, leaving several directions for future work.
- Enhancing Agent Complexity: The frameworks treat each LLM session as an AI agent, simplifying the more sophisticated agent concept used in the literature.Future systems could integrate Tree-of-Thought, Cumulative Reasoning, external tools, or knowledge bases.
- Expanding Task Diversity: The evaluation mainly focuses on reasoning tasks, while future work could examine strategic planning and interactive gaming.
- Experimenting with Additional LLMs: Computational constraints limited experiments with additional LLMs, so broader model coverage could clarify generalizability and scalability.
A Discussion Engineering and Agent Symmetry
The paper formalizes discussion engineering as the joint design of LLM inferences, prompt decorators, and a mechanism pipeline, then analyzes agent symmetry through computational graphs and permutations.
- Discussion Engineering: Discussion engineering jointly designs each LLM inference, its prompt decorator, and the mechanism organizing those inferences.The mechanism is represented as M = (M, {(Li, pi)}^n_i=1).
- Computational Graph: A computational graph encodes inference nodes, input and output nodes, and directed dependencies between inference operations.The graph contains the information needed to determine the mechanism.
- Agent Assignment: A multi-agent discussion is defined as D = (M, A, α), combining the mechanism, agent set, and inference-to-agent assignment.The assignment map α links each inference node to an agent, with matrix P equivalent to α.
- Agent Symmetry: Mechanism invariance tests whether permuting agents preserves an isomorphism between the resulting colored computational graphs.Model invariance additionally requires permuted agents to use identical underlying LLMs.
- Agent Symmetry: The largest possible symmetry group for m agents is S_m, but mechanism and model asymmetries can reduce the realized symmetry group.Asymmetry may arise from the computational graph, mechanism pipeline, or prompt decorators.
B.1 Motivation
The motivation develops CMD as a group-discussion alternative to existing multi-agent frameworks, addressing discussion overhead while organizing agents into staged, multi-round deliberation and voting.
- Motivation: Increasing agents or discussion rounds may improve performance or consensus, but additional agents sharply increase input-token overhead and context-limit pressure.
- Related Frameworks: Existing comparison frameworks include Debate’s turn-based discussion, MAD, and ReConcile’s consensus-oriented interaction.
- Motivation: CMD introduces a group-discussion framework inspired by real-life discussions to provide a broader comparison of single-agent and multi-agent mechanisms.
- Discussion Representation: CMD represents each agent’s answer as a viewpoint and explanation, while discussion history records information exchanged across rounds.
- Information Sharing: Agents receive all group information but only viewpoints from agents outside their group, while local group members provide explanations.
- CMD Procedure: CMD proceeds through initialization, grouped multi-round discussion, and final voting, with ties resolved by a secretary or higher-level representatives.