PaperlayerView exact revision on arXiv

Source-linked AI summary

More Agents Is All You Need

Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, Deheng Ye

arXiv:2402.05120v2cs.CLcs.AIcs.LG

TL;DR

The paper investigates whether increasing the number of raw LLM agents generally improves performance, addressing limited prior evidence on this scaling phenomenon. It introduces Agent Forest, a sampling-and-voting method, and finds broad performance improvements, compatibility with existing methods, and task-difficulty-dependent gains.

  • Problem

    Prior studies observed benefits from multiple agents or reasoning chains but lacked a dedicated, comprehensive study of whether raw-agent performance scaling generally exists.

  • Method

    Agent Forest repeatedly samples outputs from an LLM or integrated method and uses majority voting to select the final answer.

  • Results

    Across diverse LLMs and reasoning and generation datasets, performance generally improves with ensemble size, existing methods can improve further when combined, and gains vary with task difficulty.

  • Takeaways & Limitations

    Adding more instantiated agents can provide comparable performance without additional handcrafted prompts or complex collaboration frameworks, within the evaluated scope.

  • Takeaways & Limitations

    Increasing ensemble size raises token usage proportionally, and optimizing the resulting cost remains future work.

Abstract

from arXiv · show

We find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method, termed as Agent Forest, is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence. Our code is publicly available at: https://github.com/MoreAgentsIsAllYouNeed/AgentForest

1 Introduction

The paper asks whether increasing the number of instantiated LLM agents generally improves performance, and studies this through a simple sampling-and-voting method across tasks and models.

  • Motivation: Prior ensemble and multi-agent studies suggest that using multiple agents or reasoning chains can improve performance, but did not systematically study raw-agent scaling.The paper identifies limited scenarios and experiments in prior work as leaving the generality of the phenomenon unresolved.
  • Method: Agent Forest repeatedly queries a single LLM or an existing agent framework, then uses majority voting to select the final answer.It is designed as a simple plug-in rather than requiring complex chain-of-thought paths.
  • Findings: Across diverse reasoning and generation datasets and LLMs of different sizes, performance generally improves as ensemble size increases.The experiments cover a wide range of tasks and models.
  • Findings: A brute-force ensemble of smaller LLMs can achieve comparable or superior performance to larger LLMs.Figure 1 illustrates this pattern on GSM8K for Llama2 and GPT-3.5-Turbo models.
  • Findings: Agent Forest can enhance existing methods, while its gains are greater on difficult tasks and with weaker models.The paper analyzes inherent difficulty, reasoning-step length, and the prior probability of a correct answer.

2 Related Work

Related work spans homogeneous self-ensembles, supervised heterogeneous ensembles, and interactive multi-agent frameworks; Agent Forest instead studies scaling raw agents through unsupervised sampling and voting.

  • LLM Self-Ensemble: LLM self-ensemble methods generate multiple outputs or thought chains from homogeneous models and select a final answer through voting or self-consistency.These methods mainly focus on reasoning tasks and CoT compatibility.
  • Positioning: Unlike prior categories, Agent Forest evaluates scaling the number of LLMs and is compatible with prompt engineering and multi-agent collaboration.The paper reports applicability to both reasoning and generation tasks.
  • Heterogeneous LLM Ensemble: Heterogeneous LLM ensemble methods combine or route different models, primarily using supervised learning and task-specific annotated data.The paper contrasts their limited generalizability with its unsupervised approach.
  • Multiple LLM Agents Collaboration: Multiple-agent collaboration methods improve task solving through structured interactions such as debate, reflection, or multi-round communication.Their emphasis is on interaction architectures rather than the scaling trend of adding raw agents.

3 Method

Agent Forest uses a two-phase sampling-and-voting procedure: it generates multiple outputs for a query and selects the candidate most similar to the others.

  • Integration: The method can feed either the task query alone or the query combined with prompt engineering methods into the agents.An LLM agent may be a single LLM or a multiple-LLM-agent collaboration framework.
  • Sampling: Agent Forest first samples N outputs by repeatedly querying an LLM or an LLM integrated with another method.The resulting sample set is S = {s1, s2, ..., sN}.
  • Voting: The voting phase scores each sample by its cumulative similarity to the other samples and returns the highest-scoring candidate.The procedure is represented by A = arg maxsi∈S V(si).
  • Voting: Similarity is measured with BLEU for open-ended generation and occurrence frequency for close-ended tasks such as multiple-choice questions.The selected answer is the sample with the highest cumulative similarity.

4 Experimental Setup

The evaluation tests Agent Forest across multiple tasks, model scales, and integrations with existing methods, using repeated runs and controlled ensemble sizes.

  • Evaluation design: The experimental design compares coverage of tasks, backbone models, and combined methods to examine comprehensiveness, applicability, and compatibility.Table 1 summarizes these comparisons with related work.
  • Tasks: The study covers arithmetic reasoning, general reasoning, and code generation through GSM8K, MATH, MMLU, Chess, and HumanEval.HumanEval outputs are aggregated using pairwise BLEU similarity.
  • Language models: The models include Llama2-Chat at 13B and 70B parameters, GPT-3.5-Turbo, and GPT-4.This provides model scales from the Llama2 and GPT series.
  • Methods enhanced: The experiments combine Agent Forest with prompt methods including CoT, Zero-Shot CoT, and Solo Performance Prompting.The number of queries is increased and majority voting determines the most consistent answer.
  • Methods enhanced: The study also evaluates integration with LLM-Debate and self-reflection by repeatedly operating these methods and applying majority voting.Debate uses a smaller ensemble-size limit because its communication architecture adds significant computational overhead.

5 Experimental Results

Across tasks and LLMs, Agent Forest improves performance as ensemble size increases, remains effective across hyperparameter settings, and can complement other methods. Its main cost is proportional growth in token usage with the number of agents.

  • Generalizability: 12% to 24% accuracy gains on GSM8K, 6% to 10% on MATH, 1% to 4% on Chess, 5% to 11% on MMLU, and 4% to 9% on HumanEval occur as ensemble size increases.These gains are reported across arithmetic reasoning, general reasoning, and code-generation tasks.
  • Generalizability: 59% accuracy on GSM8K lets enhanced Llama2-13B outperform Llama2-70B, which scores 54%.The smaller model achieves this result simply by scaling up the ensemble size.
  • Compatibility: 10% to 21% gains on GSM8K and 1% to 15% on MATH show that Agent Forest further improves integrated methods.Additional gains are also reported on Chess, MMLU, and code generation, although debate integration fails for Llama2-13B and Llama2-70B.
  • Effectiveness: The method achieves the highest average ranking across LLMs and tasks without additional prompts or complex collaboration frameworks.The rankings are derived from Table 3 across five tasks for each LLM.
  • Hyperparameter robustness: Scaling ensemble size consistently improves performance across tasks despite changes in temperature T and nucleus probability p.The ablation uses GPT-3.5-Turbo and averages results over 20 runs.
  • Token usage: Token usage increases proportionally with the number of agents or when Agent Forest is combined with other methods.The method therefore allows a trade-off between a higher token budget and improved performance on specific tasks.

6 Understanding the Performance Gains

Controlled experiments identify three task-difficulty dimensions shaping Agent Forest gains: inherent difficulty, reasoning-step count, and correct-answer prior probability. Gains generally strengthen with difficulty or longer reasoning, but excessive inherent difficulty produces diminishing returns.

  • Overall difficulty: Relative gains are larger on harder tasks: Llama2-13B gains 28%-200% versus 8%-16% for GPT-3.5-Turbo, while MATH gains 34%-200% versus 16%-69% on GSM8K.The relative gain is defined as the difference between method performance and single-query performance, normalized by single-query performance.
  • Inherent difficulty: Agent Forest gains initially increase with inherent difficulty but taper at extreme difficulty, with the largest gains observed at I = 100 and I = 200.Experiments vary I from 10 to 400 while holding S and K constant across groups.
  • Number of steps: Performance gains increase as the number of reasoning steps rises, reaching 16%-48% for {I = 100, K = 4} versus 4%-18% for {I = 10, K = 2}.S is varied from 1 to 8 while I and K remain fixed within experimental groups.
  • Number of steps: Agent Forest improves accuracy at each reasoning step, although accumulated errors from earlier steps reduce single-query accuracy as step count increases.Step-wise Agent Forest applies the ensemble separately to each explicitly prompted step.
  • Derived methods: Step-wise Agent Forest yields 15%-42% gains at S = 8 and K = 4, while hierarchical decomposition improves homogeneous performance from 21% to 31% and heterogeneous performance from 35% to 47%.Hierarchical Agent Forest uses intermediate answers to focus later solving, potentially assigning different models to different levels.
  • Prior probability: Performance increases with prior probability: varying K from 4 to 32 changes the correct-answer prior from 1/4 to 1/32, and higher prior probability yields better performance.The experiments isolate prior probability while controlling the other task dimensions.

7 Conclusions and Future Work

The paper concludes that increasing the number of sampled LLM agents generally improves performance and remains compatible with other enhancement methods. It also characterizes how gains vary with task difficulty, while identifying sampling cost as an unresolved practical issue.

  • Conclusions: Agent Forest generally improves LLM performance by increasing ensemble size and can be combined with existing methods for further improvements.The conclusion presents the method as orthogonal to existing approaches.
  • Conclusions: Performance gains increase then decrease with inherent difficulty, increase with reasoning-step count, and improve with correct-answer prior probability.These three properties are reported from isolated analyses of task difficulty.
  • Future work: Increasing the number of agents leaves the input unchanged, so the sampling phase may be optimized to reduce cost.The paper leaves this cost optimization for future work and notes that escalating costs commonly affect methods requiring multiple LLM calls.

A Detailed Experiment Settings

The experiments use specified model versions, with GPT-3.5-Turbo and GPT-4 identifiers referring to different dated releases across experimental sections.

  • Model versions: Section 4 uses gpt-3.5-turbo-0613 and gpt-4-0613, while Section 6 uses gpt-3.5-turbo-1106 and gpt4-1106-Preview with JSON mode enabled.The stated model-version conventions apply to the corresponding GPT-3.5-Turbo and GPT-4 experiments.

A.2 Experiments on Arithmetic Reasoning Tasks

Arithmetic-reasoning experiments extract candidate answers from sampled responses and select the final answer through task-specific similarity-based voting.

  • Arithmetic reasoning: For GSM8K and MATH, numerical or mathematical-expression answers are extracted by matching “boxed{{X}}”.The extraction occurs during Agent Forest’s initial sampling phase.
  • Arithmetic reasoning: The arithmetic voting phase uses mathematical-equivalence comparisons and selects the most probable sample as the final answer.The selected answer is compared with ground truth using mathematical equivalence.
  • General reasoning: For general reasoning tasks, samples are extracted from option or chess-position patterns, and the most frequently occurring option is selected.This passage covers MMLU and Chess rather than GSM8K or MATH.
  • Code generation: For HumanEval code generation, Python snippets are sampled and the candidate with the highest cumulative BLEU score is selected.BLEU similarity is computed with sacreBLEU during voting.

B.1 Additional Accuracy Curves

Accuracy curves across multiple datasets and LLMs show that increasing ensemble size generally improves standalone performance and enhances other methods.

  • Generalizability: Increasing ensemble size generally enhances standalone performance across datasets and LLMs.The curves cover Llama2-13B, Llama2-70B, and GPT-3.5-Turbo settings.
  • Compatibility: Increasing ensemble size generally improves the performance of other methods as well.This compatibility property is evaluated across the reported accuracy and debate-accuracy curves.

B.2 Statistical Test Results

Statistical tests find significant accuracy differences among ensemble sizes, while the token-budget plot frames accuracy as a budget tradeoff.

  • Statistical significance: All one-way ANOVA p-values are less than 0.05 across ensemble sizes 1, 10, 20, 30, and 40.The analysis uses 10 independent runs for each comparison.
  • Statistical significance: The ANOVA evaluates accuracy differences across ensemble sizes for different LLMs and datasets.Table 7 reports the corresponding p-values from these tests.
  • Token-accuracy tradeoff: Figure 14 plots token budget on the x-axis against accuracy on the y-axis.The figure presents the relationship as a tradeoff between token usage and accuracy.
Loading 2402.05120v2…