PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowledge Fusion of Large Language Models

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, Shuming Shi

arXiv:2401.10491v2cs.CL

TL;DR

Training LLMs from scratch is costly, and architectural differences make direct weight blending impractical. FUSELLM transfers knowledge from source-model distributions to a target model; across diverse tasks, its target outperformed source models and a baseline in most tasks.

  • Problem

    Training LLMs from scratch is costly, while architectural differences make directly merging existing LLM weights impractical.

    Source: §1 · p. 1

  • Method

    FUSELLM uses source LLMs’ generative distributions to externalize their knowledge and transfer it to a target through lightweight continual training.

    Source: §1 · p. 2

  • Results

    Across 42 tasks spanning reasoning, commonsense, and code, FUSELLM’s target outperformed each source LLM and the baseline in most tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Knowledge fusion is a promising direction for diverse LLMs, with corpus relevance and fusion-function choice important considerations.

    Source: §1 · p. 2

    FUSELLM still trails OpenLLaMA or MPT on code generation; the authors cite Llama-2’s lower performance as target and about 7.59% coding-related training text.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

While training large language models (LLMs) from scratch can generate models with distinct functionalities and strengths, it comes at significant costs and may result in redundant capabilities. Alternatively, a cost-effective and compelling approach is to merge existing pre-trained LLMs into a more potent model. However, due to the varying architectures of these LLMs, directly blending their weights is impractical. In this paper, we introduce the notion of knowledge fusion for LLMs, aimed at combining the capabilities of existing LLMs and transferring them into a single LLM. By leveraging the generative distributions of source LLMs, we externalize their collective knowledge and unique strengths, thereby potentially elevating the capabilities of the target model beyond those of any individual source LLM. We validate our approach using three popular LLMs with different architectures--Llama-2, MPT, and OpenLLaMA--across various benchmarks and tasks. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseLLM}.

1 INTRODUCTION

The introduction proposes FUSELLM as a way to transfer knowledge from existing LLMs into a target model, avoiding direct weight blending across differing architectures. Evaluations report gains across diverse tasks and motivate further work on corpus selection and fusion strategies.

1 INTRODUCTION

  • Knowledge fusion could reduce the costs of training from scratch, while ensemble inference requires multiple models and weight merging often assumes matching architectures.Weight merging may also be suboptimal when models differ substantially in parameter space.Source: §1 · pp. 1–2
  • FUSELLM transfers source-model knowledge to a target by using their generative distributions and lightweight continual training.It aligns tokenizations across models and explores two ways to fuse their probability distributions.Source: §1 · p. 2
  • Across 42 tasks spanning reasoning, commonsense, and code, FUSELLM’s target outperformed each source LLM and the baseline in most tasks.In a simulated setting with structurally identical LLMs, it also showed stronger potential than ensemble and weight merging on perplexity.Source: §1 · p. 2
  • The authors identify training-corpus relevance to downstream tasks and the choice of fusion function as important considerations for combining models with differing capabilities.They characterize LLM fusion as a promising direction given the diverse structures and substantial sizes of LLMs.Source: §1 · p. 2

2 RELATED WORK

Prior model-fusion approaches either combine outputs or merge parameters, while knowledge distillation transfers teacher knowledge to a student. The paper distinguishes FUSELLM by transferring knowledge across diverse LLM architectures without requiring the target to be smaller than its sources.

Model Fusion

  • Ensemble methods combine model outputs, including through weighted averaging, majority voting, or selection and combination of highly ranked LLM outputs.The approach can enhance system performance but does not explicitly merge the models into a new one.Source: §2 · pp. 2–3
  • Weight merging combines model parameters, with prior work applying it to models with identical structures and to models or adapters configured for different tasks.The cited approaches include weighted averaging and linear operations on adapter parameters.Source: §2 · p. 3
  • FUSELLM transfers knowledge and capabilities to a target model, supporting fusion across diverse LLM architectures unlike ensemble or weight merging.Ensembling deploys multiple models in parallel, while weight merging is generally limited to identical architectures.Source: §2 · p. 3

Knowledge Distillation

  • FUSELLM shares a framework with multi-teacher distillation but does not constrain the target to be smaller than its teachers, and aims for it to surpass source models.Conventional text-generation distillation transfers teacher distributions or generated texts, or minimizes divergence between student and teacher distributions.Source: §2 · p. 3

3 KNOWLEDGE FUSION OF LLMS

FUSELLM transfers source LLM capabilities by externalizing their predictive distributions and continually training a target model. It aligns and fuses source distributions, then trains the target against the fused representation.

3 KNOWLEDGE FUSION OF LLMS

  • FUSELLM stimulates source LLMs to predict next tokens, assesses their distributions, and uses the most accurate predictions to continually train a target model on corpus C.The training uses a causal language modeling objective.Source: §3 · p. 3

3.1 PRELIMINARIES

  • The causal language modeling objective minimizes negative log-likelihood, equivalently viewing token predictions as distributions compared with one-hot gold-token labels.The discrepancy form is equivalent to the negative log-likelihood objective when implemented as KL divergence.Source: §3.1 · pp. 3–4

3.2 LLMS FUSION

  • FUSELLM represents source models’ knowledge with predictive distributions, aligns differing vocabularies, fuses the matrices, and trains the target to match the fused representation.The combined continual-training objective weights the causal language modeling and fusion objectives.Source: §3.2 · p. 4

3.3 IMPLEMENTATION OF FUSELLM

  • MinED aligns tokens across different tokenizers, while MinCE selects the lowest-cross-entropy source distribution and AvgCE averages distributions with weights based on cross-entropy.MinED relaxes exact matching to preserve more information in aligned distributions while introducing minor errors.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate FUSELLM by continual training a Llama-2 7B target on fused source-model distributions, using a compact corpus and benchmarks for reasoning, commonsense, and code. The reported results show gains across these capabilities, faster optimization, and stronger comparisons with several alternative methods.

4.1 EXPERIMENTAL SETUP

  • Experiments use a Llama-2 7B target and MiniPile, evaluating reasoning, commonsense, and code against the original source models and Llama-2 CLM.MiniPile contains approximately 1 million documents across 22 domains and 1.8 billion tokens; MinCE is the setup’s fusion function.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 OVERALL RESULTS

  • 5.16% average relative gain over original Llama-2 on BBH and 1.25% on CS accompany FUSELLM’s gains over baselines across both benchmarks.On BBH, the comparison covers 27 tasks; on CS, it covers five tasks.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • 6.36% average performance gain on MultiPL-E, versus 1.37% for Llama-2 CLM, with FUSELLM beating Llama-2 on 9 of 10 tasks.It still trails OpenLLaMA or MPT; coding-related text is estimated at approximately 7.59% of MiniPile.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3 THE FUSED PROBABILISTIC DISTRIBUTIONS

  • 2.5% higher exact-match accuracy than Llama-2 CLM, with FUSELLM reaching CLM’s best performance within 0.52 billion tokens rather than 1.57 billion.The comparison tracks few-shot CoT performance on BBH as training tokens increase.Source: §4.3 · p. 8, Figure 2 · p. 8

4.4 ANALYSIS OF IMPLEMENTATION PROCESS

  • Fusing one to three models improves performance overall, with gains consistent on BBH but more prominent for two models on CS and ME.The reported variation across benchmarks may relate to the size of performance differences among source models on those tasks.Source: §4.4 · p. 8, Table 4 · p. 8
  • MinED consistently outperforms exact-match token alignment, and MinCE outperforms AvgCE across benchmarks.The authors attribute MinED’s gains to retaining useful token information despite minor tokenizer discrepancies, and AvgCE’s losses to distortions from weighted summation.Source: §4.4 · p. 8, Table 5 · p. 8

4.5 FUSELLM VS. KNOWLEDGE DISTILLATION

  • On BBH, Llama-2 KD improves 2.97% over Llama-2, below FUSELLM’s 5.16% gain.The comparison distills Llama-2 13B into Llama-2 7B; FUSELLM integrates three 7B models with diverse architectures.Source: §4.5 · p. 9, Table 6 · p. 9

4.6 FUSELLM VS. ENSEMBLE/MERGING

  • Across three domains, FUSELLM achieves the lowest average perplexity among the compared fusion methods, including ensemble and weight merging.The experiment creates domain-specific Pythia 1B models from three corpora, then trains FUSELLM on 0.1 billion sampled tokens.Source: §4.6 · p. 9

5 CONCLUSION

FUSELLM transfers knowledge from structurally diverse source LLMs into a unified target and outperforms individual sources and established baselines; the conclusion also compares its perplexity with ensemble and weight-merging methods.

5 CONCLUSION

  • Table 7 compares FUSELLM perplexity with ensemble and weight-merging methods, but the supplied passage does not state the comparison’s outcome.The caption identifies the comparison scope only.Source: Table 7 · p. 9
  • FUSELLM uses source models’ generative distributions to externalize their knowledge and continually train a unified target model.The paper frames this as knowledge fusion for combining capabilities and distinctive strengths across structurally diverse LLMs.Source: §5 · p. 9
  • Experiments show FUSELLM surpasses individual source LLMs and established baselines, while a simulated setting with structurally identical models shows competitive effectiveness against ensemble and weight merging.The conclusion presents this as evidence for further exploration of LLM fusion, particularly for diverse structures and large models.Source: §5 · p. 9

A DETAILS OF TOKEN ALIGNMENT

The appendix describes token alignment across source-model distributions and MiniPile’s curation, covering both the alignment procedure and its compact, diverse training corpus.

A DETAILS OF TOKEN ALIGNMENT

  • Token alignment maps two source models’ distribution matrices along both the text-token dimension and the vocabulary-distribution dimension.Figure 3 illustrates different token-alignment methods.Source: A DETAILS OF TOKEN ALIGNMENT · p. 14, Figure 3 · p. 14
  • MinED aligns tokens with minimal vocabulary edit distance, including differing forms such as “gets” and “get,” while the earlier EM method can leave mismatches unaligned and produce one-hot distributions.For distribution values, MinED also maps “immediately” to “immediate”; EM disregards values involving different tokens.Source: A DETAILS OF TOKEN ALIGNMENT · p. 14

B DETAILS OF MINIPILE

  • MiniPile is curated from The Pile by embedding documents with E5-Large, clustering embeddings with K-means, and filtering low-quality clusters.The resulting dataset is described as compact yet diverse.Source: B DETAILS OF MINIPILE · p. 14

C TRAINING DETAILS

The target model’s training setup specifies AdamW optimization, a cosine learning-rate schedule, and packing multiple instances into training sequences.

C TRAINING DETAILS

  • Training uses AdamW with β1 = 0.9, β2 = 0.95, gradient clipping at 1.0, and weight decay of 0.1.These are the reported optimizer and regularization settings.Source: C TRAINING DETAILS · p. 15
  • The cosine learning-rate schedule uses a maximum learning rate of 1e-5 and a warmup ratio of 0.008.The passage reports these schedule settings for model optimization.Source: C TRAINING DETAILS · p. 15
  • Packing groups multiple training instances into one sequence separated by end-of-sequence tokens, allowing more tokens per batch.The paper cites packing as a training-speed measure.Source: C TRAINING DETAILS · p. 15

D ADDITIONAL EVALUATION RESULTS

Additional generative evaluations cover five task types, and the paper reports that FUSELLM surpasses all source LLMs across every task in these evaluations.

D ADDITIONAL EVALUATION RESULTS

  • FUSELLM is reported to outperform all source LLMs across every task in the additional generative benchmarks.Table 8 presents the overall results; the supplied passage gives no individual task values.Source: D ADDITIONAL EVALUATION RESULTS · p. 15, Table 8 · p. 15
  • The additional benchmarks assess knowledge-based question answering, reading comprehension, content analysis, machine translation, and theorem application.The benchmark set includes TriviaQA, DROP, LAMBADA, IWSLT2017, and SciBench.Source: D ADDITIONAL EVALUATION RESULTS · p. 15
  • TriviaQA, LAMBADA, and IWSLT2017 use zero-shot evaluation, while DROP and SciBench use few-shot chain-of-thought prompts; BLEU is reported for translation and EM for the others.These protocols and metrics describe how the additional benchmarks are evaluated.Source: D ADDITIONAL EVALUATION RESULTS · p. 15

E FUSELLM VS. PREVIOUS MODEL FUSION METHODS

The section compares FuseLLM with LLM-Blender and tests its use with instruction-tuned source models.

E FUSELLM VS. PREVIOUS MODEL FUSION METHODS

  • LLM-Blender (Rank) remained below the best-performing source LLM, while Rank&Fuse showed notable performance deterioration after fusion.The comparison covers Big-Bench Hard and MultiPL-E; CommonSense cannot use LLM-Blender because it relies on perplexity-based evaluation.Source: E FUSELLM VS. PREVIOUS MODEL FUSION… · pp. 15–16, Table 9 · p. 15

F INCORPORATING INSTRUCTION-TUNING MODELS WITH FUSELLM

  • For instruction-tuning fusion, Llama-2 Evol-Instruct was the target, and the continual-training loss masked the instruction portion.The continual-training samples must follow the instruction-tuning format.Source: F INCORPORATING INSTRUCTION-TUNING… · p. 16
  • On the Vicuna Benchmark, FuseLLM achieved the best GPT-4-judged performance, surpassing each individual source instruction-tuning LLM.The sources were fine-tuned on 20k samples each, and another 5k samples per dataset formed the continual-training corpus.Source: Table 10 · p. 16, F INCORPORATING INSTRUCTION-TUNING… · p. 16

G CAUSE OF PERFORMANCE IMPROVEMENT

RedPajama results support the authors’ view that FUSELLM’s gains reflect knowledge integration rather than continual training alone.

  • Above 95% of examples in Arxiv, StackExchange, Wikipedia, and Github showed lower BPB for OpenLLaMA or MPT than Llama-2 when FUSELLM beat Llama-2 CLM.The figure reports the percentage of test examples with reduced perplexity across domains.Source: G CAUSE OF PERFORMANCE IMPROVEMENT · p. 16
  • Across RedPajama domains, when FUSELLM outperformed Llama-2 CLM, OpenLLaMA or MPT typically also outperformed Llama-2.The authors interpret this pattern as evidence that FUSELLM’s gains come from integrating knowledge from multiple LLMs.Source: G CAUSE OF PERFORMANCE IMPROVEMENT · p. 16
  • The RedPajama evaluation used BPB to mitigate tokenizer differences and compared the percentage of samples with decreased BPB across model transitions.The transitions were Llama-2 CLM to FUSELLM and Llama-2 to OpenLLaMA or MPT.Source: G CAUSE OF PERFORMANCE IMPROVEMENT · p. 16

H WEIGHT FOR LOSS COMBINATION

The section compares static and dynamic loss weights, tests OpenLLaMA as a target, and presents task case studies.

H WEIGHT FOR LOSS COMBINATION

I USING STRONGEST LLM AS TARGET

  • With OpenLLaMA as target, FUSELLM improved on OpenLLaMA CLM but remained behind the original OpenLLaMA on code generation.The authors attribute the gap to StarCoder data in OpenLLaMA’s pretraining and limited code data in the continual-training corpus.Source: I USING STRONGEST LLM AS TARGET · p. 17, Table 12 · p. 17

J CASE STUDIES

Loading 2401.10491v2…