Source-linked AI summary
Harnessing Multiple Large Language Models: A Survey on LLM Ensemble
Zhijun Chen, Xiaodong Lu, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Ming Li, Likang Xiao, Dingqi Yang, Xiao Huang, Yikun Ban, Hailong Sun, Philip S. Yu
TL;DR
LLM Ensemble addresses performance concerns and heterogeneous strengths, weaknesses, and costs across LLMs. The paper provides the first comprehensive survey, taxonomy, and review of methods organized before, during, and after inference. It concludes with methodological analysis and future directions including label-efficient profiling, adaptation to evolving model distributions, and span-level ensemble approaches.
Problem
LLMs raise concerns about accuracy, hallucinations, and alignment, while differing substantially in strengths, responses, and inference costs; comprehensive review evidence was previously missing.
Method
The paper introduces a taxonomy and systematically reviews LLM Ensemble methods across ensemble-before-inference, ensemble-during-inference, and ensemble-after-inference.
Results
The survey synthesizes LLM Ensemble methods, benchmarks, applications, and existing methodological analyses.
Takeaways & Limitations
The paper identifies label-efficient profiling, adaptation to evolving model distributions, and principled span-level ensemble as future research directions.
Takeaways & Limitations
Existing token-level aggregation methods generally require vocabulary alignment techniques or assume that the LLMs share a common vocabulary.
Abstract
from arXiv · showhide
LLM Ensemble -- which involves the comprehensive use of multiple large language models (LLMs), each aimed at handling user queries during downstream inference, to benefit from their individual strengths -- has gained substantial attention recently. The widespread availability of LLMs, coupled with their varying strengths and out-of-the-box usability, has profoundly advanced the field of LLM Ensemble. This paper presents the first systematic review of recent developments in LLM Ensemble. First, we introduce our taxonomy of LLM Ensemble and discuss several related research problems. Then, we provide a more in-depth classification of the methods under the broad categories of "ensemble-before-inference, ensemble-during-inference, ensemble-after-inference'', and review all relevant methods. Finally, we introduce related benchmarks and applications, summarize existing studies, and suggest several future research directions. A curated list of papers on LLM Ensemble is available at https://github.com/junchenzhi/Awesome-LLM-Ensemble.
1 Introduction
The survey motivates LLM Ensemble by the performance concerns and heterogeneous strengths, weaknesses, and costs of available LLMs, then presents a comprehensive review of the field.
- Over 182,000 large language models are accessible through the Hugging Face library.
- Zero-shot and few-shot LLM use still raises concerns about accuracy, hallucinations, and alignment with human intent.
- LLMs differ substantially in responses and inference costs because of differences in architecture, size, tokenization, dictionaries, training data, and methodology.
- Existing LLM Ensemble methods are categorized by when inference and ensemble occur: before, during, or after inference.
- The survey reviews taxonomy, related problems, methods, benchmarks, applications, and future directions as a comprehensive study of LLM Ensemble.
2 LLM Ensemble Taxonomy and Related Problems
The taxonomy organizes LLM Ensemble by whether models are combined before, during, or after inference, while related problems include merging, collaboration, and weak supervision.
- LLM Ensemble Taxonomy: Ensemble-before-inference routes each query to a suitable model using discrete or continuous utility methods.
- LLM Ensemble Taxonomy: Ensemble-during-inference combines model outputs at token, span, or process granularity, making it the most granular ensemble category.
- LLM Ensemble Taxonomy: Ensemble-after-inference integrates complete responses through non-cascade methods or progressively evaluates a cost-aware cascade of models.
- Related Problems: LLM Merging combines parameters from multiple models into a universal model without original training data or extensive computation.
- Related Problems: LLM Collaboration assigns distinct roles to models that exchange response information, unlike ensemble models used with equal status against user queries.
- Related Problems: Weak supervision aggregates weak labels or trains classifiers, but most existing methods target classification rather than open-ended generation.
3 Methodology
The methodology reviews LLM Ensemble methods according to the taxonomy’s three stages: before inference, during inference, and after inference.
- The survey systematically reviews ensemble-before-inference, ensemble-during-inference, and ensemble-after-inference methods in Sections 3.1, 3.2, and 3.3.
3.1 Ensemble Before Inference
Ensemble-before-inference routes queries by predicting candidate-model utility, using discrete or continuous formulations that support different levels of preference modeling and objective optimization.
- Ensemble-before-inference predicts candidate-model utility for a query under preferences such as performance or cost, then routes the query accordingly.
- Discrete Utility Methods: Discrete utility methods use categorical labels and divide into classification-based and comparison-based routing.
- Discrete Utility Methods: Classification-based routing commonly labels a satisfactory response 1 and an unsatisfactory response 0, forming a multi-label binary classification task.
- Discrete Utility Methods: Comparison-based routing predicts relative preferences between two model responses, reducing supervision difficulty compared with estimating absolute utility for every candidate.
- Continuous Utility Methods: Continuous-utility methods represent model utility as real-valued variables, capturing fine-grained behavior and combining objectives such as latency and cost into one scalar.
- Continuous Utility Methods: OmniRouter trains separate MLP predictors to estimate performance and cost for candidate models.
3.2 Ensemble During Inference
Ensemble-during-inference combines multiple LLMs at token, span, or reasoning-process granularity while decoding. Methods either aggregate model outputs or select a single model’s output, with vocabulary alignment remaining a key token-level challenge.
- Scope: Ensemble-during-inference operates at token, span, and process levels, selecting or combining outputs during decoding.Token-level methods operate at the finest decoding granularity, while span-level and process-level methods handle larger fragments or reasoning steps.
- Token-Level Methods: Token-level aggregation averages or weights probability distributions, whereas selection directly adopts a token from one model.Aggregation produces a final token distribution; selection chooses the output token from a selected model.
- Token-Level Methods: Vocabulary discrepancies impede token-level fusion because different LLMs produce probability distributions over different vocabularies.Some methods address this by constructing a union dictionary or projecting distributions into a shared relative or pivot space.
- Token-Level Methods: GaC builds a union dictionary for averaging, while DeePEn and EVA project distributions into a shared space before averaging or weighted aggregation.UniTe reduces overhead by aggregating only the TOP-K distribution portion, and another method weights models using per-step perplexity.
- Token-Level Methods: Fine-tuning methods use labeled data to train multiple LLMs with boosting or MOE, then perform token-level weighted averaging during inference.Copilot, LLMBoost, and UltraFuser apply these ensemble-learning paradigms to downstream supervised fine-tuning.
- Token-Level Methods: Selection methods use classifiers, reinforcement learning, or agreement to choose tokens from specialized, aligned, general-purpose, or unaligned models.CDS, Co-Llm, CITER, and agreement-based ensembling differ in how they determine the selected model or token.
- Span- and Process-Level Methods: Span-level methods generally generate fragments, score all responses with perplexity, and select the fragment with the highest cumulative score.Process-level methods can use trained Monte Carlo Tree Search to select the highest-reward reasoning output at each step.
3.3 Ensemble After Inference
Ensemble-after-inference combines completed responses rather than intervening during generation. Non-cascade methods select or regenerate from candidate responses, while cascade methods use a deferral decision to determine whether to invoke a subsequent model.
- Non-Cascade: Non-cascade methods either select one response from multiple candidates or select a subset before regeneration by a generative model.These two categories are summarized as selection-based and selection-then-regeneration approaches.
- Non-Cascade: Unsupervised selection methods use response similarity and majority voting to choose the response most similar to the others.Agent-Forest and Smoothie apply this principle, including repeated queries to one model as homogeneous ensemble outputs.
- Non-Cascade: MoRE trains a random forest classifier on response similarity features to select among multiple responses.It represents a supervised response-selection method using training data.
- Non-Cascade: LLM-Blender trains PairRanker to select a response subset and GenFuser to synthesize the final response.LLM-TOPLA builds on LLM-Blender by optimizing its approach, though the supplied passage does not specify the optimization.
- Cascade: Cascade methods center on a deferral rule that decides whether to keep the current model’s output or invoke a subsequent, more powerful model.Unsupervised strategies include user judgment, answer consistency, and class uncertainty; supervised methods train cascade-related modules.
4 Benchmarks and Applications
LLM Ensemble evaluation uses benchmarks tailored to ensemble-after-inference and routing before inference, while applications extend to specialized data and domain tasks. The survey also summarizes approaches by strategy, granularity, and ensemble goal.
- Benchmarks: MIXINSTRUCT evaluates ensemble-after-inference methods, whereas RouterEval and RouterBench target ensemble-before-inference routing.The passage distinguishes these benchmark families by the ensemble stage they evaluate.
- Applications: Applications include instruction-tuning data construction, tabular data imputation, and win-rate evaluation.Instruction-tuning data construction uses ROUGE-L similarity and a similarity-based selection strategy.
5 Discussion
The survey contrasts LLM Ensemble methods by strategy, granularity, and ensemble goal, then identifies label efficiency, adaptability, and broader ensemble designs as key directions.
- Methodological analysis: Aggregation methods combine outputs, whereas selection-based methods choose a single output; regeneration additionally requires model-specific training data and preparation.The survey characterizes aggregation as more sophisticated than selection, while regeneration incurs extra training requirements.
- Methodological analysis: Finer-grained methods, especially token-level ensembles, can exploit each model’s distribution more effectively than coarse response-level ensembles.The comparison concerns how much decoding information is retained during ensemble.
- Methodological analysis: Ensemble-during-inference and non-cascaded ensemble-after-inference methods can use flexible strategies and finer-grained approaches when cost constraints are absent.The survey links this flexibility to the absence of cost considerations.
- Future directions: The survey highlights principled span-level ensemble-during-inference and sophisticated unsupervised cascade ensemble-after-inference for open-ended generation as additional research directions.These directions complement profiling and routing research with new ensemble designs.
- Future directions: Future work should develop label-efficient or weakly supervised profiling using uncertainty, self-consistency, or internal representations instead of extensive labeled data.These intrinsic signals are proposed for inferring model strengths and reliability across domains.
- Future directions: Routing methods should adapt rapidly and data-efficiently as available models and their capabilities evolve beyond fixed, static model pools.Existing evaluations typically assume closed settings with an unchanging candidate distribution.
6 Conclusion
The paper presents LLM Ensemble as ensemble learning applied to large language models and surveys its methods, benchmarks, applications, and future directions. It aims to provide researchers with a comprehensive reference for further advances.
- LLM Ensemble applies the principles of ensemble learning in the era of large language models.
- The survey organizes the field through a comprehensive taxonomy and review of existing methods.
- It also covers relevant applications and benchmarks, analyzes existing approaches, and proposes potential research directions.