Source-linked AI summary
Mixture-of-Agents Enhances Large Language Model Capabilities
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou
TL;DR
The paper asks how to harness the collective expertise of multiple LLMs despite their individual limitations and diverse specializations. It introduces a layered Mixture-of-Agents architecture that iteratively refines responses using outputs from prior layers, achieving state-of-the-art results across several benchmarks. The method also incurs high time to first token because aggregation is iterative.
Problem
LLMs have scaling constraints, while different models possess complementary strengths, motivating how to harness their collective expertise.
Method
Mixture-of-Agents uses multiple LLM agents in successive layers, with each layer using preceding agents’ outputs as auxiliary information for iterative refinement.
Results
The method achieves state-of-the-art performance on AlpacaEval 2.0, MT-Bench, and FLASK, including a 65.8% AlpacaEval 2.0 win rate versus 57.5% for GPT-4 Omni.
Takeaways & Limitations
Providing models with other models’ outputs can improve response quality, including when the auxiliary responses are lower quality than independent outputs.
Takeaways & Limitations
Iterative response aggregation delays the first token until the final MoA layer, potentially increasing time to first token and harming user experience.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) demonstrate substantial capabilities in natural language understanding and generation tasks. With the growing number of LLMs, how to harness the collective expertise of multiple LLMs is an exciting open direction. Toward this goal, we propose a new approach that leverages the collective strengths of multiple LLMs through a Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered MoA architecture wherein each layer comprises multiple LLM agents. Each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response. MoA models achieves state-of-art performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.
1 Introduction
LLMs have diverse strengths but face costly scaling constraints, motivating whether their collective expertise can improve capability. The paper identifies collaborativeness among LLMs and introduces layered MoA refinement, reporting state-of-the-art benchmark results.
- Motivation: LLMs offer strong language capabilities but remain constrained by model size and training-data limits, making further scaling exceptionally costly.Scaling can require extensive retraining on several trillion tokens.
- Motivation: Different LLMs specialize in distinct task aspects, motivating the question of whether their collective expertise can produce a more capable and robust model.Examples include instruction following and code generation.
- Collaborativeness: LLMs tend to generate better responses when given outputs from other models, even when those auxiliary responses are lower quality than their independent outputs.The improvement is reported across six popular LLMs on AlpacaEval 2.0.
- Mixture-of-Agents: MoA uses multiple agents in successive layers, with each layer refining responses generated by the preceding layer.Agents may reuse models across layers, and selection considers both performance and output diversity.
- Results: 65.8% AlpacaEval 2.0 win rate versus 57.5% for GPT-4 Omni establishes a new state-of-the-art result for the proposed method.The paper evaluates response quality on AlpacaEval 2.0, MT-Bench, and FLASK.
- Contributions: The work contributes a framework for leveraging multiple LLMs, evidence of inherent collaborativeness, and state-of-the-art performance across multiple benchmarks.The contribution summary names AlpacaEval 2.0, MT-Bench, and FLASK.
2 Mixture-of-Agents Methodology
The methodology frames LLM collaboration as a layered process in which proposer and aggregator roles iteratively combine and refine model outputs. MoA operates through prompting, uses multiple full LLMs across layers, and avoids fine-tuning.
- Collaborative roles: LLMs can act as proposers that provide diverse reference responses or aggregators that synthesize inputs into higher-quality outputs.Proposers need not score highly independently, while aggregators should maintain or improve quality when integrating weaker responses.
- Collaborative roles: GPT-4o, Qwen1.5, and LLaMA-3 were effective in both assisting and aggregating, whereas WizardLM excelled as a proposer but struggled as an aggregator.
- Layered architecture: MoA iteratively refines responses by passing outputs from one layer to multiple agents in the next, with additional aggregators enabling repeated synthesis.The architecture can reuse LLMs within or across layers, and repeated aggregation produces progressively refined responses.
- Layered architecture: Each MoA layer contains n LLMs, and a single-proposer configuration arises when many layer models are identical and produce stochastic alternative outputs.
- Prompt-based design: MoA uses only prompting and generation interfaces, so it requires no fine-tuning and can apply to off-the-shelf LLMs across sizes and architectures.The method operates at the model level with multiple full LLMs rather than modifying internal activations or weights.
- Prompt-based design: MoA extends the Mixture-of-Experts idea from activation-level specialized subnetworks to multiple full LLMs coordinated through prompts.In MoE, expert networks specialize in different skill sets; MoA instead uses full models across layers.
3 Evaluation
The evaluation finds that MoA improves performance across AlpacaEval 2.0, MT-Bench, and FLASK, including with open-source models, while budget analyses identify cost- and compute-efficient configurations. Mechanism studies attribute gains to aggregation of multiple diverse proposals rather than simple reranking.
- Benchmark Results: MoA improves results on AlpacaEval 2.0, MT-Bench, and FLASK, with open-source configurations outperforming GPT-4o on AlpacaEval 2.0 and FLASK.
- Benchmark Results: 65.1% LC win rate on AlpacaEval 2.0 is achieved by open-source MoA, compared with 57.5% for GPT-4o.MoA-Lite reaches 59.3%, exceeding GPT-4o by 1.8 percentage points.
- Benchmark Results: MT-Bench improvements are relatively incremental, yet MoA secures the top leaderboard position.The passage attributes the modest gains to already high single-model performance on this benchmark.
- Benchmark Results: On FLASK, MoA improves robustness, correctness, efficiency, factuality, commonsense, insightfulness, completeness, and metacognition, but is marginally less concise.
- What Makes Mixture-of-Agents Work Well?: MoA significantly outperforms an LLM-ranker baseline, indicating that aggregation generates a new response rather than merely selecting one proposer answer.
- What Makes Mixture-of-Agents Work Well?: Scores increase monotonically with the number of proposers, and diverse multiple-proposer settings consistently outperform single-proposer settings.The authors identify larger numbers of diverse agents as beneficial and propose further width scaling as future work.
- Budget and Token Analysis: MoA configurations form a cost-performance Pareto frontier, while MoA-Lite matches GPT-4o’s cost at higher quality and is approximately 4% better than GPT-4 Turbo at over twice its cost-effectiveness.The budget analysis also evaluates tera floating operations as a latency proxy and identifies a corresponding Pareto frontier.
4 Related Work
Prior work improves LLM outputs through structured prompting, output reranking or routing, and interactive multi-agent collaboration. These approaches include linear, branching, graph-based, and role-asymmetric strategies for reasoning or model selection.
- Prompt-engineering research includes Chain-of-Thought, Auto-CoT, task decomposition, Tree-of-Thought, graph-based reasoning, and Natural Program prompting.
- Reranking methods compare candidate outputs, while routing methods predict the best-performing model from a fixed set to reduce multi-LLM inference costs.
- Multi-agent collaboration studies use symmetric discussion or asymmetric roles such as debater and judge.
5 Conclusion
The paper presents Mixture-of-Agents as successive-stage collaboration among multiple LLMs, reporting improved response quality across three benchmarks. It also identifies high time to first token as a limitation and suggests reducing layers or using chunk-wise aggregation.
- Mixture-of-Agents leverages multiple LLMs through successive stages of iterative collaboration.The approach uses collective strengths from agents across stages to improve output quality.
- MoA substantially improves response quality on AlpacaEval 2.0, MT-Bench, and FLASK, achieving an LC win rate up to 65%.The evaluations support improvements across all three named benchmarks.
- Iterative response aggregation delays the first token until the final MoA layer, potentially increasing Time to First Token and harming user experience.The paper suggests limiting layers or using chunk-wise aggregation to mitigate this issue.
- MoA may enhance interpretability because its intermediate outputs are expressed in natural language.The paper connects these intermediate outputs with improved alignment to human reasoning.
A Spearman Correlation using Different Similarity Functions
The section measures how TF-IDF and Levenshtein similarity relate to evaluator preference scores using Spearman correlation. It reports a positive correlation between win rate and both similarity measures.
- Spearman correlation compares n similarity scores with n GPT-4-based evaluator preference scores within each sample.The similarity measures are TF-IDF-based similarity and Levenshtein similarity.
- Positive correlations appear between win rate and both TF-IDF similarity and Levenshtein similarity.The result is presented in Figure 6.
- The evaluation task asks the model to identify the best output by returning only its model identifier.
B LLM Ranker
The LLM-Ranker evaluates and ranks outputs from multiple LLMs to select the best answer. The section presents its evaluation prompt setup and references a case involving high-quality model answers.
- The LLM-Ranker is designed to evaluate and rank outputs generated by multiple LLMs.
- The ranker selects the best answer, after which the AlpacaEval evaluator evaluates that ranked answer.
- Table 5 presents the template used to prompt the model during these evaluations.
- Table 6 presents a case in which some models produce high-quality answers.
C Case Study
The case studies examine how the aggregator combines proposed responses. The examples show incorporation of useful content from multiple responses, including when no proposal receives a high preference score.
- The case study shows selected response fragments and underlines similar expressions between proposals and the aggregated response.Content mentioned by all proposed responses is omitted for brevity.
- 0.99 is both GPT-4’s preference score for WizardLM 8x22B’s response and the final aggregated answer’s preference score.The Qwen1.5-110B-Chat aggregator favors its own content while incorporating key points from Llama-3-70B-Instruct and WizardLM 8x22B.
- 0.33 is the preference score achieved by the aggregator when none of the proposed responses has a high GPT-4 preference score.The aggregator identifies and incorporates strong points from the proposed responses.
D MATH Task
The approach applies to MATH reasoning tasks, where it consistently improves accuracy by a significant margin and complements Chain of Thought and Self-consistency. Table 8 evaluates different aggregators with six models serving as proposers in each MoA layer.
- The method consistently enhances accuracy on the MATH reasoning task by a significant margin.
- The approach is complementary to Chain of Thought and Self-consistency reasoning techniques.
- Table 8 evaluates different aggregators with all six models serving as proposers in each MoA layer.