PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

Dongfu Jiang, Xiang Ren, Bill Yuchen Lin

arXiv:2306.02561v3cs.CLcs.AIcs.LG

TL;DR

Because different open-source LLMs perform best on different examples, fixed-model selection may be suboptimal. LLM-Blender pairwise-ranks multiple candidates with PairRanker and fuses the top-ranked outputs with GenFuser, achieving the strongest reported overall performance on MixInstruct. Its main limitations are quadratic PairRanker inference for optimal performance and the absence of large-scale human evaluation.

  • Problem

    Different open-source LLMs have complementary strengths, and no single model dominates across examples, motivating dynamic candidate selection.

  • Method

    LLM-Blender uses PairRanker for pairwise candidate ranking and GenFuser to fuse the top-ranked outputs into a final response.

  • Results

    LLM-Blender achieves the best reported overall performance, with average rank 3.2 among 12 methods versus 3.90 for the best individual LLM.

  • Takeaways & Limitations

    The framework provides a post-hoc rank-and-fuse approach that improves results across the reported metrics for instruction-following LLM ensembles.

  • Takeaways & Limitations

    Optimal PairRanker performance may require O(N^2) model calls, and large-scale human evaluation was not affordable.

Abstract

from arXiv · show

We present LLM-Blender, an ensembling framework designed to attain consistently superior performance by leveraging the diverse strengths of multiple open-source large language models (LLMs). Our framework consists of two modules: PairRanker and GenFuser, addressing the observation that optimal LLMs for different examples can significantly vary. PairRanker employs a specialized pairwise comparison method to distinguish subtle differences between candidate outputs. It jointly encodes the input text and a pair of candidates, using cross-attention encoders to determine the superior one. Our results demonstrate that PairRanker exhibits the highest correlation with ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates, generating an improved output by capitalizing on their strengths and mitigating their weaknesses. To facilitate large-scale evaluation, we introduce a benchmark dataset, MixInstruct, which is a mixture of multiple instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender significantly outperform individual LLMs and baseline methods across various metrics, establishing a substantial performance gap.

1 Introduction

LLM-Blender addresses the variation in which open-source LLM performs best across examples by ranking candidate outputs pairwise and fusing the strongest candidates. On MixInstruct, it outperforms individual models and achieves the best reported aggregate results.

  • Motivation: Open-source LLMs have complementary strengths and weaknesses, with Vicuna ranking first on only 21.22% of 5,000 instructions.The results indicate that no open-source LLM dominates across examples.
  • PairRanker: PairRanker compares candidate pairs jointly with the input using a cross-attention encoder to identify the better output.It produces pairwise comparison logits that are aggregated into a ranking for each input.
  • GenFuser: GenFuser fuses the top K ranked candidates to generate an improved output while leveraging their strengths and mitigating weaknesses.This extends selection beyond returning an existing candidate.
  • Benchmark: MixInstruct contains 100k training, 5k validation, and 5k test examples generated from 11 open-source LLMs, with oracle comparisons for evaluation.The benchmark combines multiple instruction-following datasets and supports training candidate-ranking modules.
  • Results: 3.2 versus 3.90: LLM-Blender’s average rank among 12 methods beats the best individual LLM, and its outputs rank top-three on 68.59% of examples versus Vicuna’s 52.88%.It also achieves the highest scores on BERTScore, BARTScore, BLUERT, and ChatGPT-based ranking.

2 Preliminaries

The paper formulates LLM ensembling as producing an output for each input from multiple model candidates, then introduces MixInstruct to benchmark ranking and fusion methods. LLM-Blender ranks candidates with PairRanker and generates a final response by fusing the top candidates.

  • Problem Setup: A fixed model can be suboptimal because different LLMs have different strengths and weaknesses across inputs.The optimal candidate for one input may not come from the model that performs best overall.
  • Problem Setup: The ensemble objective is to maximize aggregate candidate quality Q across the test set.The formulation sums Q(ŷ^(i), y^(i); x^(i)) over test examples.
  • Ensembling Methods: Selection-based methods return one of the existing candidates, whereas generation-based methods fuse multiple candidates to create an unseen output.Selection is bounded by the candidate set, while generation can produce a new response.
  • MixInstruct: MixInstruct contains 110K examples split into 100K training, 5K validation, and 5K test examples, using outputs from 11 open-source LLMs.The dataset benchmarks ensemble models for instruction-following tasks.
  • MixInstruct: Oracle candidate rankings are created by asking ChatGPT to judge all 55 pairs produced by 11 candidates, with conventional metrics also used for training and validation.The metrics include BERTScore, BLEURT, and BARTScore.
  • LLM-Blender: LLM-Blender uses PairRanker to rank candidates and selects the top K = 3 for GenFuser, a seq2seq model that generates the final output.This is a rank-and-fuse pipeline with separate ranking and generation modules.

3 PAIRRANKER: Pairwise Ranking

PairRanker replaces pointwise candidate scoring with joint pairwise comparison, aiming to capture subtle quality differences among strong, open-ended instruction-following outputs. It aggregates pairwise scores into rankings, with MaxLogits used by default because it performs best in the reported appendix experiments.

  • Limitations of Pointwise Ranking: Existing rerankers score each candidate independently from the input, which may miss subtle differences between strong instruction-following outputs.The paper highlights short responses that differ by few words yet vary in helpfulness, harmfulness, or fairness.
  • Pairwise Comparison: PairRanker jointly encodes the input and two candidates with a cross-attention Transformer to model which candidate is better.The sequence is formed as [x; yi; yj], and the model learns pair-specific comparison scores.
  • Training: PairRanker averages scores from multiple quality functions, treating pairwise learning as a multi-task classification problem.The paper considers quality functions such as BERTScore and BARTScore.
  • Inference: At inference, PairRanker computes pairwise logits for candidate pairs and aggregates the resulting matrix to rank all candidates.The matrix entries represent confidence that one candidate is better than another.
  • Aggregation: MaxLogits sums comparison confidence, whereas MaxWins counts victories against other candidates.Both use the pairwise comparison matrix to score each candidate.
  • Aggregation: Bubble-sort aggregation reduces inference complexity from O(N^2) to O(N), requiring N − 1 comparisons to select the best candidate.The method shuffles candidates, initializes an index, and updates it iteratively.
  • Aggregation: MaxLogits is used as PairRanker’s default aggregator because the appendix experiments report its best performance among the aggregation methods.This choice concerns the reported ranking performance, while bubble sort targets inference efficiency.

4 GENFUSER: Generative Fusion

GENFUSER addresses the limitations of selecting a single candidate by fusing multiple top-ranked outputs into an enhanced response. It conditions a seq2seq model on the instruction and selected candidates.

  • Motivation: GENFUSER is introduced to overcome PAIRRANKER’s dependence on candidate-selection quality by merging multiple top-ranked candidates.The approach targets complementary strengths and weaknesses among the selected outputs.
  • Method: GENFUSER takes the input instruction and K top-ranked candidates from the candidate pool to produce an improved final response.The paper gives K = 3 as an example.
  • Method: GENFUSER concatenates the input and candidates with separator tokens and fine-tunes a T5-like model to generate the output.The implementation uses Flan-T5-XL with 3b parameters.

5 Evaluation

The evaluation uses MixInstruct, conventional NLG metrics, and ChatGPT-based GPT-Rank to compare individual models, ranking methods, and the complete framework. PAIRRANKER improves candidate selection, while LLM-BLENDER achieves the strongest reported overall results.

  • 5.1 Setup: MixInstruct evaluates 11 open-source LLMs and ranking methods using BERTScore, BARTScore, BLEURT, GPT-Rank, and comparisons against Vicuna and OpenAssistant.GPT-Rank is based on ChatGPT pairwise comparisons, while the other three are conventional automatic NLG metrics.
  • 5.2 Main results: 21.22%: Vicuna ranks first on the collected instructions, showing that the best model varies substantially across examples.The analysis reports that no open-source LLM dominates the competition.
  • 5.2 Main results: 40%: Koala responses are better than or equal to both OpenAssistant and Vicuna despite Koala’s average GPT-Rank of 6.76.This example illustrates why fixed reliance on top overall models can be suboptimal.
  • 5.2 Main results: 18% relative performance gain: PAIRRANKER’s selected responses achieve average GPT-Rank 3.20, outperforming OpenAssistant’s 3.90.PAIRRANKER also reaches BARTScore −3.14, exceeds or matches Vicuna/OpenAssistant on 54.76%/57.79% of examples, and places in the top three on 65.12%.
  • 5.2 Main results: 3.01: LLM-BLENDER’s GPT-Rank surpasses OpenAssistant’s 3.90, while its BERTScore 79.09, BARTScore −3.02, and BLEURT −0.17 exceed the best model.The framework uses top-3 PAIRRANKER selections as GENFUSER inputs.
  • 5.2 Main results: Highest correlation with GPT-Rank: PAIRRANKER outperforms the other evaluated rankers across all reported correlation types.BARTScore has the highest correlation among the automatic metrics and is selected for supervision.

6 Related Work

The related work connects LLM evaluation, pairwise ranking, and ensemble learning. LLM-BLENDER differs by combining pairwise inference with attention-based comparison and by selecting candidates before generative fusion.

  • LLM evaluation: LLM evaluation includes chatbot-arena-style pairwise assessments of responses from randomly selected LLMs.These evaluations provide a basis for comparing model capabilities as open-source LLMs become more competitive.
  • Pairwise ranking: Prior ranking methods such as RankNet, LambdaRank, and RLHF reward models use pairwise training but compute candidate scores individually.The paper distinguishes this from PAIRRANKER’s pairwise inference.
  • Pairwise ranking: PAIRRANKER combines pairwise training with attention-based pairwise inference to compare candidate outputs jointly.This design is presented as the contrast with prior individually scored ranking approaches.
  • Ensemble learning: Prior fusion approaches combined outputs without first selecting candidates, whereas LLM-BLENDER ranks candidates before feeding them to fusion.The paper contrasts this selection-then-fusion pipeline with Fusion-in-Decoder-based work.

7 Conclusion & Future Directions

LLM-Blender dynamically ranks and fuses outputs from multiple open-source LLMs to exploit their complementary strengths. The paper contributes the MixInstruct benchmark, reports improved performance across metrics, and identifies extensions for future work.

  • 7 Conclusion & Future Directions: LLM-Blender is a post-hoc ensemble method that ranks and fuses outputs from multiple LLMs using PAIRRANKER and GENFUSER.PAIRRANKER selects candidates, while GENFUSER fuses the top-ranked outputs.
  • 7 Conclusion & Future Directions: MixInstruct is a benchmark dataset for training and evaluating LLM ensembling methods on instruction-following tasks.
  • 7 Conclusion & Future Directions: The method significantly improves overall results across various metrics, supporting LLM ensembling as a promising direction for researchers and practitioners.
  • 7 Conclusion & Future Directions: The framework is open-sourced to help others leverage the approach for developing AI systems with robustness, generalization, and enhanced accuracy.
  • 7 Conclusion & Future Directions: Future work includes broader model and modality coverage, more sophisticated ranking and fusion, transfer to other domains, lower computational overhead, and active learning.

*Limitations

The paper identifies inference efficiency and evaluation methodology as limitations of LLM-Blender. PairRanker may require many comparisons, while large-scale human evaluation was not affordable.

  • Limitations: PAIRRANKER may require O(n^2) model calls to obtain the full comparison matrix, making the best-performing solution less efficient.Multiple rounds of bubble sort reduce the number of required inferences.
  • Limitations: PAIRRANKER’s numerous independent inferences can be executed in parallel, partially mitigating the computational burden.
  • Limitations: Automatic metrics have limitations, but large-scale human evaluation was unaffordable because of the number of models and generated candidates.The authors use ChatGPT-based evaluation as an alternative.

*Ethical Statement

The authors state that the work complies with the ACL Ethics Policy and report no known ethical issues.

  • Ethical Statement: The paper declares compliance with the ACL Ethics Policy and no ethical issues to the best of the authors’ knowledge.

A Implementation Details

The implementation trains PAIRRANKER with DeBERTa-v3-large and trains GENFUSER using Flan-T5 variants on top-ranked selections. The reported setup favors specific embedding combinations, BCE loss, and the larger Flan-T5 variant tested.

  • Implementation Details: PAIRRANKER is trained for 5 epochs with Adafactor, a maximum learning rate of 1e-5, a 5% warm-up ratio, and batch size 64.Training finishes on a single RTX 8000 GPU in two days.
  • Implementation Details: PAIRRANKER uses DeBERTa-v3-large as its backbone and concatenates the source with each candidate for its best-performing embedding configuration.
  • Implementation Details: Binary cross-entropy was selected after comparing mean-squared-error and ranking losses.
  • Implementation Details: GENFUSER is trained with top-3 BARTScore selections as input and applies PAIRRANKER’s top 3 selections during inference.
  • Implementation Details: Flan-T5-3b performs much better than the large version, while Flan-T5-xxl provides only marginal improvements despite being larger and slower to train.

B Conventional Tasks

This section evaluates reranking on summarization, constrained generation, and translation, showing that default decoding often misses better candidates and that reranking can improve quality across tasks.

  • Candidate selection: Default top-beam selections are substantially worse than oracle selections from a relatively small candidate pool.The analysis covers CNN/DailyMail summarization, WMT18 Chinese-English translation, and CommonGen constrained generation.
  • Candidate selection: 57% Rouge-2 improvement for PEGASUS and nearly 80% BLEU improvement for Opus-MT are obtained by expanding and selecting from candidate pools.These gains are reported relative to top-beam performance.
  • Reranking motivation: Candidate reranking is motivated as a post hoc response to exposure bias, greedy search, and sampling randomness in autoregressive generation.Prior approaches include unsupervised MLM scoring and supervised reranking methods.
  • Experimental setup: PAIRRANKER is evaluated on CNN/DM, CommonGen, and WMT18 (zh-en) using Rouge, BLEU, and CIDEr metrics.The experiments use PEGASUS-large and BART-large for summarization, T5-large for CommonGen, and Opus-MT for translation.
  • Experimental setup: The reranker is trained and tested with candidate-generation procedures designed to prevent candidate leakage between training and evaluation.Inference uses public checkpoints, while candidate generation includes beam search and diverse beam search.

C.4 Main results

The main results show that Max Logits improves candidate quality across summarization, CommonGen, and translation, while the study also examines generalization, comparison efficiency, consistency, and rank-gap effects.

  • Main results: 6.35% Rouge-1, 9.62% Rouge-2, and 6.25% Rouge-L gains are achieved by Max Logits on CNN/DM with fine-tuned PEGASUS-large.These gains exceed the reported SummaReranker baseline.
  • Main results: 2.45% CIDEr and 6.12% BLEU gains are achieved by Max Logits on CommonGen and WMT2018 (zh-en), respectively.The results support task generalization beyond summarization.
  • Main results: SummaReranker decreases CIDEr by 1.23% on CommonGen and improves translation BLEU by only 0.57%, indicating weaker cross-task generalization.The paper attributes the difference partly to shorter candidate and target lengths and higher in-group similarity.
  • Efficiency: Bubble run offers high performance at low comparison cost, while Max Logits surpasses it after sufficiently many comparisons.The paper identifies N − 1 comparisons as a practical bubble-run operating point.
  • Further study: Shuffling candidate order makes the reranker agree with itself more than 90% of the time, despite input-position sensitivity.The model is also more accurate when the absolute pair-rank difference is larger.
Loading 2306.02561v3…