PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?

Wenzhe Li, Yong Lin, Mengzhou Xia, Chi Jin

arXiv:2502.00674v1cs.CLcs.LG

TL;DR

The paper asks whether mixing different LLMs is genuinely beneficial for MoA, given the trade-off between model diversity and proposer quality. It introduces Self-MoA, which aggregates repeated outputs from a single top-performing model, and finds that it often outperforms Mixed-MoA while Self-MoA-Seq extends the approach under context-length constraints.

  • Problem

    The paper examines whether cross-model diversity in MoA is beneficial when mixing different LLMs may introduce lower-quality proposers.

  • Method

    Self-MoA aggregates multiple repeatedly sampled outputs from one top-performing model, while Self-MoA-Seq performs this aggregation sequentially with a sliding window.

  • Results

    Self-MoA outperforms Mixed-MoA across many benchmarks, including a 6.6 point improvement on AlpacaEval 2.0, and its quality-diversity analysis finds MoA highly sensitive to proposer quality.

  • Takeaways & Limitations

    In-model diversity from a single strong model can be more effective than cross-model diversity, while sequential aggregation supports larger inference budgets under context constraints.

  • Takeaways & Limitations

    Self-MoA cannot scale indefinitely by simply increasing computation because its aggregation is constrained by context length, motivating Self-MoA-Seq.

Abstract

from arXiv · show

Ensembling outputs from diverse sources is a straightforward yet effective approach to boost performance. Mixture-of-Agents (MoA) is one such popular ensemble method that aggregates outputs from multiple different Large Language Models (LLMs). This paper raises the question in the context of language models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an ensemble method that aggregates outputs from only the single top-performing LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA achieves $6.6\%$ improvement over MoA on the AlpacaEval 2.0 benchmark, and an average of $3.8\%$ improvement across various benchmarks, including MMLU, CRUX, and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0 directly achieves the new state-of-the-art performance on the leaderboard. To understand the effectiveness of Self-MoA, we systematically investigate the trade-off between diversity and quality of outputs under various MoA settings. We confirm that the MoA performance is rather sensitive to the quality, and mixing different LLMs often lowers the average quality of the models. To complement the study, we identify the scenarios where mixing different LLMs could be helpful. This paper further introduces a sequential version of Self-MoA, that is capable of aggregating a large number of LLM outputs on-the-fly over multiple rounds, and is as effective as aggregating all outputs at once.

1 Introduction

The paper questions whether MoA benefits from mixing different LLMs and proposes Self-MoA, which repeatedly samples and aggregates outputs from one high-performing model. Experiments show that this approach often outperforms mixed-model MoA, while Self-MoA-Seq addresses context-length constraints during inference.

  • MoA queries multiple proposer LLMs and uses an aggregator to synthesize their responses.
  • Self-MoA instead synthesizes multiple outputs repeatedly sampled from the same model, leveraging in-model diversity.
  • 6.6 point improvement over Mixed-MoA was achieved by Self-MoA on AlpacaEval 2.0 using the strongest model.
  • Over 200 experiments found a quality-diversity trade-off, with MoA especially sensitive to proposer quality and often strongest at high quality with relatively low diversity.
  • Self-MoA-Seq uses sequential aggregation to process more outputs with shorter context lengths without compromising final performance.

2 Related Work

Related work situates Self-MoA among LLM ensembles, repeated sampling, and collaborative-agent methods. These approaches combine model outputs, repeated generations, or iterative interactions to improve responses and solve difficult tasks.

  • Ensembles of LLMs: LLM ensembling combines model strengths through prompting, weight averaging, routing, or generative fusion.
  • LLM Inference with Repeated Sampling: Repeated sampling can improve pass@k and task success in mathematics and code tasks with automatic verifiers.
  • Collaborative Agents: Collaborative-agent research uses verification, critique, discussion, debate, and refinement to improve or combine model responses.

3 Is Ensembling Different LLMs Beneficial?

This section introduces Self-MoA, which uses repeated outputs from a single top-performing model, and evaluates it against Mixed-MoA across general-purpose and specialized-model settings. Across the reported experiments, Self-MoA improves over mixed-model ensembling, including a 6.6-point AlpacaEval 2.0 gain and consistent task-level advantages.

  • Method: Self-MoA uses one top-performing model as proposer to generate multiple outputs, which are then aggregated into a final response.This approach targets in-model diversity rather than relying on different proposer models.
  • 3.1 Experiments on AlpacaEval 2.0 with General Purpose Models: 6.6 points: Self-MoA outperforms the Mixed-MoA baseline on AlpacaEval 2.0 when using Qwen1.5-110B-Chat as aggregator.The comparison uses six open-source instruction fine-tuned models for Mixed-MoA and repeated outputs from a strong model for Self-MoA.
  • 3.1 Experiments on AlpacaEval 2.0 with General Purpose Models: 2-3 points: Self-MoA consistently gains over the top-performing AlpacaEval 2.0 models and reaches the leaderboard’s top position.Each top-performing model serves as both proposer and aggregator, with four outputs used because of context-length constraints.
  • 3.2 Experiments on Multiple Datasets with Specialized Models: The specialized-model experiments evaluate MMLU-redux, CRUX, and MATH using Qwen2-7B-Instruct, DeepSeek-Coder-V2-Lite-Instruct, and Qwen2-Math-7B-Instruct.These models are selected for strengths in common-sense question answering, coding, and mathematics, respectively.
  • 3.2 Experiments on Multiple Datasets with Specialized Models: Self-MoA also outperforms Mixed-MoA across all 13 tested proposer mixtures on MMLU, CRUX, and MATH when the general-purpose model is the aggregator.Using a task-specific aggregator adds 1-2 points, while greater model diversity does not consistently improve performance.
  • 3.2 Experiments on Multiple Datasets with Specialized Models: Figure 2 measures proposer quality by average performance and diversity with the Vendi Score, framing the trade-off underlying Mixed-MoA and Self-MoA.The figure compares mixtures of different proposer models with repeated outputs from the same proposer model.

4 The Quality-Diversity Trade-off

The paper studies how MoA performance depends on proposer quality and diversity, finding that quality is generally the more influential factor. Mixed-MoA can help when models have similar quality and complementary task specialization, but Self-MoA remains competitive or superior in many settings.

  • Quality-Diversity Trade-off: MoA performance improves when either proposer diversity or quality increases while the other factor remains fixed.These trends are consistently aligned across MMLU, CRUX, and MATH.
  • Quality-Diversity Trade-off: The achievable Pareto front contains a trade-off between proposer quality and diversity.Seeking cross-model diversity can introduce lower-quality models and reduce overall mixture quality.
  • Quality-Diversity Trade-off: MoA performance is typically best in high-quality, high-diversity mixtures, showing strong sensitivity to proposer quality.The reported best-performing mixtures appear in the bottom right of each subplot.
  • Statistical Analysis: Linear regression finds significant relationships between MoA performance, quality, and diversity, with the model explaining approximately 70% of performance variation across three datasets.Both regression coefficients have p-values below 0.001, while the R2 values are approximately 0.7.
  • Statistical Analysis: Quality has a stronger effect than diversity across MMLU, CRUX, and MATH, especially on CRUX and MATH.The coefficient for quality exceeds the coefficient for diversity in all three datasets.
  • When Mixed-MoA Outperforms Self-MoA?: Mixed-MoA can outperform Self-MoA when models have similar quality and specialize in different tasks, but Self-MoA improves by 3.8% in single-task comparisons.On the combined MMLU, CRUX, and MATH task, the strongest Mixed-MoA results improve over Self-MoA by 0.17% and 0.35%.

5 Scaling Inference Compute with Self-MoA

This section examines whether Self-MoA scales with more repeated samples and introduces Self-MoA-Seq to overcome aggregation and context-length constraints. Self-MoA-Seq performs comparably to or slightly better than Self-MoA while supporting shorter context lengths.

  • Limitations of Scaling: Self-MoA cannot scale indefinitely because sample diversity plateaus, aggregating many samples is difficult, and aggregators have finite context lengths.For example, Gemma 2 has an 8192-token context limit.
  • Self-MoA-Seq: Self-MoA-Seq uses a sliding window to aggregate a fixed number of responses at a time, addressing aggregation and context-length constraints.The sequential design can handle an unlimited number of responses regardless of context length constraints.
  • Evaluation Setup: The evaluation varies sample sizes on MMLU and CRUX using the best-performing model as both proposer and aggregator.The sampling temperature is 0.7, and Self-MoA-Seq uses a window size of six.
  • Scaling Results: Self-MoA and Self-MoA-Seq both significantly improve performance over the individual base model.
  • Scaling Results: Adding more samples can have positive or negative effects, so there is no universal compute-optimal sample count.
  • Scaling Results: Self-MoA-Seq delivers performance comparable to, or slightly better than, Self-MoA.

6 Conclusion

The paper introduces Self-MoA, which uses in-model diversity during inference, and reports that it often outperforms Mixed-MoA, especially when proposer quality varies. It also extends the approach to constrained context lengths.

  • Self-MoA uses in-model diversity to enhance large language model performance during inference.
  • Self-MoA outperforms traditional Mixed-MoA strategies in many popular benchmarks, particularly when proposer model quality varies.
  • Aggregating outputs from a single high-performing model addresses the quality-diversity trade-off identified for mixed models.
  • The paper extends Self-MoA to constrained context-length settings and identifies scenarios where mixing LLMs may be beneficial.

A.1 Multi-Layer MoA

This appendix describes multi-layer MoA, its response-aggregation formulation, and supporting evaluation concepts. It reports that 2-layer Self-MoA outperforms 3-layer Mixed-MoA with half as many forward passes.

  • Multi-Layer MoA: Multi-layer MoA uses n LLMs in each of l layers, with each layer aggregating model outputs into inputs for the next layer.
  • Multi-Layer MoA: Each proposer generates a response for the layer input, and the aggregator concatenates that response with the original query in its prompt.
  • Comparison: Self-MoA outperforms the 3-Layer Mixed-MoA baseline while requiring only half the forward passes.A forward pass counts either a proposer output or an aggregator synthesis.
  • Vendi Score: The Vendi Score measures diversity as the effective number of unique elements in a sample collection using pairwise similarities.
  • Input Normalization: The input-normalization procedure transforms a sequence of inputs into normalized values using their mean and standard deviation.

B.1 MT-Bench Results

The MT-Bench comparison evaluates Mixed-MoA and Self-MoA under matched settings and reports that Self-MoA performs better, while GPT-4o aggregation achieves the best result with fewer forward passes.

  • MT-Bench Comparison: Self-MoA outperforms its Mixed-MoA counterpart on MT-Bench under the matched experimental comparison.
  • MT-Bench Comparison: Using GPT-4o as the aggregator achieves the best performance even with fewer forward passes.

B.2 Comparison to Universal Self-Consistency

The paper compares Universal Self-Consistency (USC) with MoA and Self-MoA on AlpacaEval 2.0. With proposers and aggregators controlled, USC performs worse than its MoA counterpart.

  • The experiments compare Self-Consistency, MoA, and Self-MoA on AlpacaEval 2.0.
  • USC is used because AlpacaEval 2.0 evaluates instruction following without exact answers.
  • USC performs worse than its MoA counterpart when proposers and aggregators are controlled.

B.3 Normalizing Sub-tasks in Table 6

The evaluation normalizes three subtasks by weighting them according to inverse variance, reducing the influence of high-variance tasks. The normalized results are reported in Table 11 for a more balanced comparison of Self-MoA and Mixed-MoA.

  • Normalization method: Inverse-variance weights are assigned so each of the three tasks contributes equally to the overall performance metric.
  • Normalization method: MMLU, CRUX, and MATH have standard deviations of 3.50, 5.70, and 4.27, respectively, across the reported settings.
  • Comparisons: Table 10 compares Self-MoA and Mixed-MoA on MT-Bench using Qwen1.5-110B-Chat and GPT-4o as aggregators.
  • Normalization method: The MMLU weight is computed from the reciprocal of its standard deviation relative to the reciprocals for CRUX and MATH.
  • Results: Table 11 reports normalized results using a weighted composition of the three subtasks.
  • Results: The weighted composition is intended to prevent high-variance CRUX from disproportionately influencing the overall comparison.
  • Comparisons: Table 12 compares Self-MoA, Mixed-MoA, and USC on the AlpacaEval 2.0 leaderboard using Qwen1.5-110B-Chat as aggregator.
Loading 2502.00674v1…