PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration

Zhenran Xu, Senbao Shi, Baotian Hu, Jindi Yu, Dongfang Li, Min Zhang, Yuxiang Wu

arXiv:2311.08152v2cs.CL

TL;DR

Single-model reasoning can be difficult to improve through self-correction alone when external feedback is unavailable. This paper uses peer-review-style collaboration among agents, and reports superior accuracy across ten datasets, with analyses supporting confidence-bearing reviews and feedback exchange.

  • Problem

    Self-correction may not enable an LLM to generate novel thoughts or correct an initial answer without external feedback.

    Source: §1 · p. 2

  • Method

    Agents independently solve a question, review peers’ solutions with confidence scores, and revise their initial submissions using the reviews.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across all ten datasets, the method outperforms prior baselines; on GSM8K, it exceeds the prior state-of-the-art by +1.40%.

    Source: §1 · p. 2, §4.4 · p. 5

  • Takeaways & Limitations

    The analyses support confidence integration and feedback exchange over mere solution sharing as useful elements of multi-agent collaboration.

    Source: §1 · p. 2, §5 · p. 9

    Multi-agent methods have token-consumption costs, and collaboration may also depend on agent roles and tool specification.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable capabilities in general natural language processing tasks but often fall short in complex reasoning tasks. Recent studies have explored human-like problem-solving strategies, such as self-correct, to push further the boundary of single-model reasoning ability. In this work, we let a single model "step outside the box" by engaging multiple models to correct each other. We introduce a multi-agent collaboration strategy that emulates the academic peer review process. Each agent independently constructs its own solution, provides reviews on the solutions of others, and assigns confidence levels to its reviews. Upon receiving peer reviews, agents revise their initial solutions. Extensive experiments on three different types of reasoning tasks show that our collaboration approach delivers superior accuracy across all ten datasets compared to existing methods. Further study underscores the effectiveness of integrating confidence in reviews, demonstrates the superiority of feedback exchange over mere solution sharing, and highlights the role of capability and diversity in fostering successful collaboration.

1 Introduction

The introduction argues that single-model self-correction can fail without external feedback, and proposes peer-review-style collaboration in which agents create, review, and revise solutions. The method outperforms prior baselines across ten datasets, with further analyses highlighting confidence and feedback exchange.

1 Introduction

  • Single-model self-correction can fail because an LLM may remain convinced its incorrect initial answer is correct without external feedback.The introduction also cites Liang et al. (2023), who suggest that self-reflection may not generate novel thoughts after an initial solution.Source: §1 · p. 2
  • The proposed method has agents independently solve a question, review peers’ reasoning with confidence scores, and revise their initial submissions.The process is framed as an iterative cycle of creation, review, and revision.Source: §1 · p. 2
  • The method outperforms prior single-agent and multi-agent baselines across all ten datasets covering mathematical, commonsense, and symbolic reasoning.The authors also report that confidence in reviews and feedback exchange are effective components of the approach.Source: §1 · p. 2
  • The analyses find confidence integration effective and feedback exchange superior to sharing solutions alone.The authors also identify capability and diversity as relevant to successful collaboration.Source: §1 · p. 2

2 Related Work

Related work surveys prompting and self-correction strategies for reasoning, then describes multi-agent collaboration approaches. The paper positions its peer discussions and mutual feedback within this emerging area.

2 Related Work

  • Chain-of-thought, self-consistency, plan-and-solve, and least-to-most prompting use rationales, voting, or decomposition to support multi-step reasoning.Plan-and-solve and least-to-most prompting break complex problems into simpler subproblems solved in sequence.Source: §2 · p. 2
  • Prior findings suggest self-correction may fail to produce novel thoughts or correct an answer without external feedback.The section contrasts these findings with earlier work exploring self-correction.Source: §2 · p. 2
  • Prior multi-agent work includes role-playing, debate, and other collaboration paradigms, while this paper proposes peer discussions and mutual feedback.The section also relates multi-agent collaboration to Society of Mind theory.Source: §2 · pp. 2–3

3 Methodology

The methodology organizes collaboration into creation, review, and revision: agents first solve independently, review peers’ reasoning, and then revise using received feedback. The final prediction is selected by majority vote.

3 Methodology

  • Agents independently generate reasoning chains and answers for a question during the creation stage.The overview defines creation, review, and revision as the method’s three stages.Source: §3 · p. 3
  • During review, each agent examines peers’ reasoning step by step and assigns confidence from 1 to 10 to indicate feedback reliability.Agents review peers’ solutions one at a time.Source: §3 · p. 3
  • During revision, agents receive peer reviews together and refine their initial solutions using those reviews and peer solutions as additional information.In the Figure 2 example, two of three agents incorporate feedback and converge on the correct solution.Source: §3 · pp. 3–4
  • Table 1 is identified as detailing ten evaluation datasets, with Math, CS, and Sym. denoting mathematical, commonsense, and symbolic reasoning.The supplied caption does not give the dataset-level statistics.Source: Table 1 · p. 4
  • The method does not explicitly require consensus; instead, it determines the final prediction by majority vote among the agents.The authors describe this choice as ensuring impartiality and balance.Source: §3 · p. 4

4 Experiment

The experiments evaluate peer review collaboration across ten datasets spanning mathematical, commonsense, and symbolic reasoning, then examine its performance and collaboration factors.

4.1 Benchmarks

  • The ten evaluation datasets cover mathematical, commonsense, and symbolic reasoning.The mathematical set includes GSM8K, SVAMP, AQuA, MultiArith, AddSub, and SingleEq; commonsense examples include ARC-c and StrategyQA, while symbolic examples include Colored Objects and Penguins.Source: §4.1 · p. 4

4.4 Results

  • Across mathematical, commonsense, and symbolic benchmarks, collaboration surpasses prior baselines; mathematical gains over prior state-of-the-art are +1.40% on GSM8K, +3.80% on SVAMP, and +2.75% on AQuA.Table 2 reports mathematical accuracy; Tables 3 and 4 report commonsense and symbolic results.Source: Table 2 · p. 5, Table 3 · p. 5, Table 4 · p. 5

4.2 Baselines / 4.3 Implementation Details

  • Comparators included zero-shot CoT, self-correction, majority voting, and debate; main experiments used gpt-3.5-turbo-0613.Further collaboration analyses also used gpt-3.5-turbo-16k-0613, claude-instant-1.2, and claude-2.1.Source: §4.2 · p. 5, §4.3 · p. 5

4.4 Results

  • Confidence let agents prioritize more reliable feedback on mathematical tasks, but could leave or reduce performance on commonsense and symbolic tasks.Without confidence, revision tended to be misled by ineffective feedback; confidence could guide selective aggregation in math.Source: §4.4 · p. 6
  • Removing peer solutions lowered performance on 9 of 10 datasets, while feedback-only exchange beat solution-only debate on 8 of 10; Figure 3 tracks GSM8K answer changes.For both GSM8K and Penguins, collaboration had a larger Incorrect→Correct share than Correct→Incorrect.Source: §4.4 · p. 6, §4.5 · p. 6, Figure 3 · p. 6

4.5 Analysis

  • Accuracy peaked at four agents then declined at five; extra review rounds showed no significant improvement, while Table 5 favors smaller capability gaps and higher diversity; role prompts mattered.The role-prompt experiments are reported in Appendix D.Source: §4.5 · pp. 7–8, Table 5 · p. 7
  • Most verbalized confidence scores fell between 80% and 100%, yet feedback was overconfident; Penguins feedback was less calibrated than GSM8K feedback.The reported reliability diagrams show bin accuracy substantially below corresponding confidence.Source: §4.5 · p. 7, Figure 5 · p. 8

4.6 Case study

  • In the GSM8K case, three agents converged on a correct solution after two wrong starts and peer critique of a misunderstanding and ratio step.Agent B explicitly addressed Agent A’s misunderstanding and ratio-step error.Source: §4.6 · pp. 8–9

5 Conclusion

The paper presents peer-review-style multi-agent collaboration for complex reasoning and reports superior accuracy across ten datasets and three tasks. Its analysis also supports confidence integration and feedback exchange over solution sharing.

5 Conclusion

  • The approach delivers superior accuracy across all ten datasets and three reasoning tasks compared with existing methods.Source: §5 · p. 9
  • The strategy engages multiple models in an iterative cycle of creation, review, and revision.Source: §5 · p. 9
  • Further analysis supports confidence integration and finds feedback exchange superior to solution sharing.Source: §5 · p. 9

Limitations

The paper identifies token consumption as a cost of multi-agent methods and calls for balancing effectiveness with efficiency. It also notes that collaboration may depend on agent roles and tools as well as capability and answer diversity.

Limitations

  • Token consumption is an associated cost of multi-agent methods, making effectiveness-efficiency balance an area for exploration.Source: Limitations · p. 9
  • Factors underlying collaboration include agent role and tool specification, in addition to capability and answer diversity.Source: Limitations · p. 9
  • Further research may identify which types of agents collaborate more effectively to inform practical multi-agent system optimization.Source: Limitations · p. 9

A Implementation Details

The experiments use GPT-3.5 Turbo with default parameters and a three-agent, one-review-cycle setup, alongside specified explorations of larger agent groups and additional rounds. Evaluation reports dataset accuracy on randomly selected samples, with baseline implementations described.

A Implementation Details

  • 3 agents and 1 review cycle form the default setup, using gpt-3.5-turbo-0613 with default parameters.Source: A Implementation Details · p. 11
  • The gpt-3.5-turbo-16k-0613 version is used to test five agents and increased review cycles, with accuracy calculated after each of four rounds.The first-round Figure 4 result may differ from Table 2 because the latter reports a different result.Source: A Implementation Details · p. 11
  • The self-correction baseline follows Huang et al. (2023), while multi-agent debate uses the official repository with three agents and two debate rounds.Source: A Implementation Details · p. 11
  • Accuracy is the evaluation metric, and each dataset contributes at most 500 randomly selected samples because of rate limits and budget constraints.Source: A Implementation Details · p. 11

B Example of Confidence Ablation

The example contrasts confidence-free review with confidence-guided review: ineffective feedback leaves an initially wrong answer unchanged without confidence, while confidence helps guide revision to the correct answer.

B Example of Confidence Ablation

C Answer Change Comparison

This section compares revised answers with initial answers and summarizes changes after self-correction and the collaboration method on Penguins.

D Impact of Diverse Role Prompts

The experiments examine role diversity in multi-agent collaboration on GSM8K and SVAMP. The method with diverse role prompts outperforms the same-role setting.

D Impact of Diverse Role Prompts

E Case Study

The case study follows collaboration through creation, review, and revision. Its examples show agents identifying errors in peer solutions and arriving at a correct answer after revision.

E Case Study

  • The case study presents creation, review, and revision as three stages, with Tables 8–11 illustrating those stages.Source: E Case Study · p. 12
  • In the creation example, agents independently produce solutions; Agent A and Agent C are wrong, while Agent B is correct.The example includes a GSM8K question and its gold answer.Source: Table 8 · p. 13
  • During review, Agent A identifies the primary error in Agent C’s solution, while Agent B points out Agent A’s error but misses Agent C’s.Source: Table 9 · p. 14
  • Figure 6 categorizes Penguins answer changes as unchanged, correct-to-incorrect, incorrect-to-correct, or incorrect-to-incorrect.It compares changes after self-correction and the collaboration method.Source: Figure 6 · p. 15
  • Agent C’s review accurately assesses others’ answers and identifies the primary error in Agent A’s solution.Source: Table 10 · p. 15
  • After peer reviews, all agents arrive at the correct answer in the revision example.Table 11 presents the third stage, revision.Source: Table 11 · p. 16

Related papers

Loading 2311.08152v2…