ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs

Justin Chih-Yao Chen; Swarnadeep Saha; Mohit Bansal

Source-linked AI summary

ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs

Justin Chih-Yao Chen, Swarnadeep Saha, Mohit Bansal

arXiv:2309.13007v3Sep 22, 2023cs.CLcs.AIcs.LG

TL;DR

LLMs struggle with natural-language reasoning, especially when single-model reflection or debate provides insufficiently diverse insights. ReCONCILE coordinates diverse LLM agents through iterative discussion, corrective explanations, and confidence-weighted voting, improving reasoning across benchmarks and sometimes surpassing GPT-4.

Problem
Single-agent reflection and same-model multi-agent reasoning provide limited novelty or diversity for challenging reasoning tasks.
Method
RECONCILE uses distinct LLM agents that exchange answers and explanations across multiple rounds, learn to convince or correct one another, and use confidence-weighted voting.
Results
Across seven benchmarks, RECONCILE consistently outperforms prior single-agent and multi-agent baselines and improves upon GPT-4 on some benchmarks.
Takeaways & Limitations
Diversity originating from different models is critical to RECONCILE’s superior performance, while discussion improves both team and individual agent performance.
Takeaways & Limitations
API-based agents have unknown training data and parameter scales, limited behavioral control, and post-hoc confidence estimates.

Abstract

from arXiv · show

Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds (Minsky, 1988), we propose ReConcile, a multi-model multi-agent framework designed as a round table conference among diverse LLM agents. ReConcile enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidence-weighted voting mechanism that leads to a better consensus. In each round, ReConcile initiates discussion between agents via a 'discussion prompt' that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answer-rectifying human explanations, used for convincing other agents. Experiments on seven benchmarks demonstrate that ReConcile significantly improves LLMs' reasoning -- both individually and as a team -- surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets. ReConcile also flexibly incorporates different combinations of agents, including API-based, open-source, and domain-specific models, leading to an 8% improvement on MATH. Finally, we analyze the individual components of ReConcile, demonstrating that the diversity originating from different models is critical to its superior performance. Code: https://github.com/dinobby/ReConcile

1 Introduction

RECONCILE addresses limited reasoning diversity in single-model reflection and debate by coordinating distinct LLM families in iterative, confidence-aware discussion. Agents exchange explanations, attempt to convince one another, and reach consensus through weighted voting.

Self-reflection can suffer from Degeneration-of-Thought when a model is overly confident and fails to generate novel thoughts.
Prior multi-agent debate increases reasoning diversity but typically uses multiple instances of the same underlying model.
RECONCILE uses diverse LLM families in multiple discussion rounds to rectify incorrect answers or increase confidence in correct ones.
Each agent conditions on grouped prior answers, explanations, and uncertainty while learning from convincing corrective explanations.
Experiments span seven benchmarks and show gains across individual agents and teams, with diversity, confidence estimation, and convincingness analyzed as framework components.
11.4% improvement over the leading multi-agent debate baseline appears on date understanding, while StrategyQA improves over GPT-4 by 3.4% and MATH over GPT-4 and DeepSeekMath by 8%.

2 Related Work

Related work develops reasoning through prompting, self-reflection, collaboration, and ensembles, but single-agent approaches lack diverse insights and same-model multi-agent systems may lack sufficient response diversity. RECONCILE builds on this landscape by combining diverse models in collaborative discussion.

Reasoning methods include Chain-of-Thought, Tree-of-Thought, self-consistency, scratchpads, verifiers, self-collaboration, self-reflection, and mathematical fine-tuning.
Eliciting reasoning from a single agent is fundamentally limited by a lack of diverse insights.
Student-teacher multi-agent work distills reasoning capabilities from stronger teachers to weaker students, unlike RECONCILE’s peer discussion among diverse agents.
Figure 2 presents RECONCILE as a three-phase process combining initial responses, multi-round discussion, and confidence-weighted team voting.
Different training data and architectures produce distinct capabilities, motivating ensembles and mixture-of-experts approaches for robustness and accuracy.

3 Problem Setup

RECONCILE models a round table with distinct LLM agents that generate answers and Chain-of-Thought explanations, learn from corrective demonstrations, and discuss repeatedly to improve team consensus.

The setup includes n distinct LLM agents that may differ in pre-training data and model architecture.
For each agent, a small number of demonstrations pair questions, answers, and human explanations that rectify initially incorrect answers.
The objective is to improve team performance through multiple discussion rounds, uncertainty quantification, and convincing agents toward better consensus.

4 RECONCILE: A Collaborative Discussion Framework

RECONCILE uses diverse LLM agents in multiple discussion rounds to revise answers and explanations, then generates a team answer through calibrated confidence-weighted voting.

RECONCILE operates in three phases: initial response generation, multi-round discussion, and team answer generation.
Phase 1: Initial Response Generation: Each agent initially generates an answer, Chain-of-Thought explanation, and confidence in the range [0, 1].
Phase 2: Multi-round Discussion: During each discussion round, agents condition on grouped previous answers, explanations, confidence scores, and convincing samples from other agents.The discussion prompt is designed to help agents revise their answers and generate explanations that convince other agents.
Phase 2: Multi-round Discussion: Convincing samples pair questions with gold answers and corrective human explanations that previously rectified an agent’s incorrect answer.Experiments use four such samples per agent when human explanations are available; the agent’s own samples are excluded from its prompt.
Phase 3: Team Answer Generation: After consensus or at most R rounds, RECONCILE recalibrates agent confidences and uses them as weights in the team-answer vote.The calibrated weighted vote is used because it performs slightly better and works across all seven evaluated tasks.

5 Experimental Setup

The experiments use diverse LLM agents and seven reasoning benchmarks, with API-call constraints leading to evaluation on subsets of 100 samples and comparisons against single- and same-backbone multi-agent baselines.

Agents in RECONCILE: RECONCILE primarily uses ChatGPT, Bard, and Claude2 for up to three discussion rounds.The framework is also tested with GPT-4, LLaMA-2-70B, and DeepSeekMath as alternative API-based, open-source, and domain-specific agents.
Datasets: The evaluation covers seven benchmarks: two commonsense, three math, one logical reasoning, and one NLI task.
Baselines: Comparisons include single-agent systems and multi-agent debate methods using multiple ChatGPT instances as agents.The same-backbone baselines include multi-agent debate and debate with judge methods.
Implementation Details: Because of API costs and call limits, experiments use subsets of 100 samples, with full-test-set checks reported for StrategyQA and Date understanding.The full-test-set experiments find similar trends.

6 Results

RECONCILE consistently improves reasoning across benchmarks and agent combinations, with gains attributed especially to diverse models. Its discussion process also improves individual agents, accelerates consensus, and links greater consensus with higher accuracy.

Main Results: Across all five reasoning benchmarks, RECONCILE outperforms single-agent and multi-agent baselines built on the same models, also surpassing GPT-4 on StrategyQA and CSQA.Results use ChatGPT, Bard, and Claude2 as agents; GPT-4 is not used as an agent in this comparison.
Main Results: 11.4% (75.3% →86.7%) improvement occurs on date understanding, while StrategyQA improves 7.7% (71.3% →79.0%) over multi-agent debate with Claude2.These comparisons use the strongest baseline, multi-agent debate with Claude2.
Agent Generalization: 12.1% (75.6% →87.7%) improvement over zero-shot GPT-4 occurs on StrategyQA when GPT-4 participates alongside comparatively weaker agents.This result demonstrates that a stronger agent can use external feedback from weaker agents.
Agent Generalization: 7.8% and 9.6% gains over zero-shot DeepSeekMath and GPT4-based Debate, respectively, occur when RECONCILE combines GPT-4, Claude2, and DeepSeekMath on MATH.RECONCILE therefore incorporates a domain-specific mathematical model while improving its performance.
Additional Benchmark: 9.4% improvement over Debate on ANLI shows that RECONCILE also benefits a challenging Natural Language Inference benchmark.The result extends the reported evaluation beyond commonsense, mathematical, and logical reasoning tasks.
Ablations and Analysis: Ablations show positive contributions from every component, including a 6.8% gain from multiple models and a 4.5% gain from convincing samples.The evaluated components also include grouping and confidence estimation.
Ablations and Analysis: Different models produce the highest response diversity and accuracy, with similarity 0.8739 and accuracy 79.0%, compared with 0.9102 for the single-model variant.Lower similarity indicates greater diversity; the analysis connects alternate solutions and external feedback with better discussion and reasoning accuracy.
Ablations and Analysis: RECONCILE improves each agent’s accuracy alongside overall team accuracy, while reaching consensus for all samples by round 3.Accuracy peaks at 79.0% versus 71.3% for debate baselines, and greater consensus is positively correlated with accuracy.

7 Conclusion

RECONCILE is a multi-agent reasoning framework in which diverse LLM agents discuss, estimate confidence, and generate explanations intended to correct one another. It achieves strong benchmark performance, consistently surpassing prior single-agent and multi-agent baselines and improving upon GPT-4 on some benchmarks.

RECONCILE engages diverse LLM agents in multiple discussion rounds using confidence estimation and corrective explanations.
RECONCILE consistently outperforms prior single-agent and multi-agent baselines across multiple reasoning benchmarks.
RECONCILE improves upon GPT-4 on some benchmarks.

Limitations

The API-based models used in RECONCILE have unknown training data exposure and parameter scales, and their behavior cannot be fully controlled. Their confidence estimates are therefore prompted post hoc.

API-based models have unknown exposed data and parameter scales, while API access limits control over their behavior.
RECONCILE relies on prompted confidence estimates for API-based models, making those estimates post hoc.
These limitations could potentially be mitigated by future open-source models that robustly follow long instructions.

A.1 Implementation Details

RECONCILE uses an initial prompt and a discussion prompt, with implementation settings covering model decoding, convincing demonstrations, and confidence instructions. Its evaluations include full-test-set comparisons and analyses of convincing samples.

Implementation settings: ChatGPT and Bard use temperature 0.7, Claude2 uses its default setting, and each agent receives four convincing demonstrations.
Prompt design: RECONCILE uses an initial prompt and a discussion prompt.
Prompt design: The initial prompt includes convincing samples, the test question, step-by-step reasoning, and a confidence level from 0.0 to 1.0.
Evaluation: On StrategyQA, RECONCILE without convincing samples outperforms multi-agent debate, while convincing samples provide further gains.
Prompt design: The discussion prompt extends the initial prompt by asking agents to review other solutions and uses grouped answer categories with plausible explanations.
Evaluation: Full-test-set results for StrategyQA and Date Understanding confirm the same trends as the reported sample-based evaluation.

B.2 Convincing Samples Improve Both RECONCILE and Multi-agent Debate

RECONCILE uses convincing samples, confidence recalibration, and weighted voting to improve discussion outcomes. Convincing samples help both RECONCILE and debate, while agents otherwise tend to retain incorrect initial opinions.

Framework: RECONCILE generates answers, explanations, and confidences, then uses discussion prompts and weighted voting to produce team answers.
Convincing samples: Absolute 7.8 points separate RECONCILE from the debate baseline when no human explanations are available, while selected answer-rectifying samples improve results substantially.
Comparisons: RECONCILE outperforms 9-way Self-Consistency by a large margin on most datasets despite using nine calls only on 12% of GSM8K samples and 9% of Date Understanding samples.
Confidence and voting: The main recalibration weights w∗ = [1.0, 0.8, 0.5, 0.3, 0.1] outperform alternative settings across datasets and majority vote across all seven datasets.
Framework: RECONCILE continues for up to R rounds or until consensus, using recalibrated confidence scores as voting weights.
Confidence and voting: Weighted vote performs best against maximum-confidence and unweighted majority voting on StrategyQA.
Convincing samples: Without convincing samples, agents maintain their initial opinions and fail to converge to the correct answer in the illustrated case.

C.4 RECONCILE with Convincing Samples

Convincing human explanations help agents revise incorrect answers during multi-round discussion, whereas their absence can prevent convergence. Diverse-model feedback is important: same-model debate may preserve an initial error.

Bard and Claude2 revise their initial incorrect predictions and ultimately reach the correct answer when convincing samples are present.The Bruce Lee example supplies the discussion context and the reported outcome.
Without convincing samples, agents largely retain their original opinions and fail to converge to the correct answer.The illustrative example shows little opinion change across discussion rounds.
With convincing samples, agents change their opinions and explanations during multiple discussion rounds.The examples describe answer-rectifying human explanations that help correct an agent’s initial answer.
In debate among multiple instances of ChatGPT, all agents persist with the same incorrect response when external feedback from diverse models is lacking.The example contrasts same-model debate with the diverse feedback used by ReConcile.

D Benchmark Licenses

The supplied material records dataset-license documentation and illustrates component analyses of ReConcile. These analyses show persistent errors without diverse feedback, improved calibration after confidence rescaling, and more confident reasoning with convincing samples.

Dataset licenses for the experiments are documented in Table 14.
Without external feedback from diverse models, multiple ChatGPT agents persist with an initially incorrect response throughout debate.
Confidence rescaling produces a significant drop in Expected Calibration Error in ReConcile.Figure 9 compares ECE before and after rescaling.
Conditioning on convincing samples makes agents more confident and less uncertain in their reasoning.For Bard, the reported confidence score rises from 0.7 to 0.95.