PaperlayerView exact revision on arXiv

Source-linked AI summary

FuseChat: Knowledge Fusion of Chat Models

Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, Xiaojun Quan

arXiv:2408.07990v1cs.CL

TL;DR

Building LLMs from scratch is costly, while existing models have complementary strengths and differing architectures. FuseChat pairwise-fuses source chat models into common target models, then merges them in parameter space; it outperforms baselines across evaluated model scales.

  • Problem

    Existing chat LLMs have distinct strengths and architectures, while training new models is costly and fusion requires token alignment, source-distribution integration, and a diverse compact dataset.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    FuseChat pairwise-fuses source chat LLMs into same-sized target models using a pivot, then merges them in parameter space with SCE coefficients based on parameter-update magnitudes.

    Source: §1 · p. 2

  • Results

    FuseChat-7B showed superiority on AlpacaEval 2.0 and MT-Bench against chat LLMs at 7B, 10B, and 20B scales.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FuseChat supports plug-and-play integration of new source LLMs at any scale by fusing each with the pivot and merging the resulting target.

    Source: §3.5 · p. 6

    Constructing the diverse-domain fusion dataset requires substantial data engineering, and the study does not assess comprehension or hallucination mitigation.

    Source: §6 · p. 10

Abstract

from arXiv · show

While training large language models (LLMs) from scratch can indeed lead to models with distinct capabilities and strengths, it incurs substantial costs and may lead to redundancy in competencies. Knowledge fusion aims to integrate existing LLMs of diverse architectures and capabilities into a more potent LLM through lightweight continual training, thereby reducing the need for costly LLM development. In this work, we propose a new framework for the knowledge fusion of chat LLMs through two main stages, resulting in FuseChat. Firstly, we conduct pairwise knowledge fusion on source chat LLMs of varying structures and scales to create multiple target LLMs with identical structure and size via lightweight fine-tuning. During this process, a statistics-based token alignment approach is introduced as the cornerstone for fusing LLMs with different structures. Secondly, we merge these target LLMs within the parameter space, where we propose a novel method for determining the merging coefficients based on the magnitude of parameter updates before and after fine-tuning. We implement and validate FuseChat using six prominent chat LLMs with diverse architectures and scales, including OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B. Experimental results on two instruction-following benchmarks, AlpacaEval 2.0 and MT-Bench, demonstrate the superiority of FuseChat-7B over baselines of various sizes. Our model is even comparable to the larger Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench. Our code, model weights, and data are public at \url{https://github.com/fanqiwan/FuseAI}.

1 Introduction

The introduction motivates knowledge fusion as a way to combine existing chat LLMs’ strengths without the cost of training from scratch. FuseChat uses pairwise fusion followed by parameter-space merging, and supports diverse source models and plug-and-play additions.

1 Introduction

  • The authors motivate knowledge fusion as a way to integrate existing LLMs’ distinct strengths while reducing the high time and computational costs of building new models.They note that LLMs can have distinct strengths despite similar capabilities across tasks.Source: §1 · p. 2
  • Ensembling aggregates multiple models’ outputs but incurs memory and inference costs, while parameter-space merging typically assumes uniform architectures and needs merging coefficients.Knowledge fusion instead aims to transfer capabilities across architectures through lightweight continual training.Source: §1 · p. 2
  • FuseChat combines pairwise knowledge fusion into same-sized target models with parameter-space merging, using SCE to set fine-grained coefficients from parameter-update magnitudes.The pairwise fusion uses a pivot and lightweight fine-tuning; SCE assigns matrix-level coefficients without additional training.Source: §1 · p. 2
  • Unlike FUSELLM, FuseChat supports source chat LLMs of varying sizes and adds new sources plug-and-play by fusing them into a target and merging it.The comparison also reports that pairwise fusion empirically mitigates the challenges of distillation relative to many-to-one fusion.Source: §1 · p. 2
  • FuseChat-7B was evaluated using six open-source chat LLMs and showed superiority across instruction-following benchmarks against chat LLMs at 7B, 10B, and 20B scales.The benchmarks were AlpacaEval 2.0 and MT-Bench.Source: §1 · p. 2

2 Related Work

Related work covers output ensembling, parameter-space merging, mixture-of-experts models, and knowledge fusion. Knowledge fusion distills across model architectures, but token alignment, combining source distributions, and constructing a diverse compact dataset remain challenges.

Model Fusion

  • Ensemble methods combine model outputs by weighted averaging, voting, sequence-level ranking and consolidation, token-level gating, or prompt perplexity.These approaches combine outputs rather than model parameters.Source: §2 · p. 3
  • Parameter-space merging combines models of identical structure and scale through averaging or computed coefficients, while some methods use task vectors, sparsification, or expanded depth.Some coefficient-learning methods require additional forward or backward propagation and data.Source: §2 · p. 3
  • Mixture-of-experts approaches combine specialized dense or LoRA experts using sparse activation or token- and sequence-level routing.The cited approaches may require further fine-tuning to learn token-level routing.Source: §2 · p. 3

Knowledge Distillation

  • Knowledge distillation trains a student guided by teacher outputs, hidden states, relations, generated distributions, or generated sequences.For multiple teachers, prior approaches average distributions or blend sequences.Source: §2 · p. 3

Knowledge Fusion

  • Knowledge fusion transfers knowledge across architectures without parallel deployment, but requires token alignment, a fusion function for source distributions, and a diverse compact dataset.FUSELLM uses source-model distribution matrices to transfer collective knowledge into a target model.Source: §2 · p. 3

3 FUSECHAT

FUSECHAT first transfers knowledge pairwise from source chat LLMs into same-size targets, then merges those targets in parameter space. Its methods combine token alignment for differing tokenizers with SCE, which sets matrix-level merging coefficients from fine-tuning updates.

3.1 Overview

  • FUSECHAT selects a pivot whose structure and scale set the targets’ structure and scale, then pairwise-fuses it with each remaining source before merging the resulting targets.The pivot choice also considers the candidate model’s capabilities and performance.Source: §3.1 · p. 4

3.2 Preliminaries

  • SFT minimizes negative log-likelihood on instruction-response examples, using the instruction and preceding response tokens to predict each next token.The model’s predicted probability for that token is conditioned on the instruction and preceding response tokens.Source: §3.2 · p. 4

3.3 Pairwise Knowledge Fusion

  • FUSECHAT treats output distribution matrices as representations of models’ knowledge and uses minimum cross-entropy to fuse pivot and source distributions.The target LLM is fine-tuned to align its predictions with the fused representation.Source: §3.3 · p. 5
  • Mapping statistics align distributions by choosing the most frequent mapping for 1-1 and 1-n cases, and frequency-weighting source distributions before alignment for n-1 cases.The method is described as better reflecting dataset mapping statistics while preserving significant information and minimizing alignment errors.Source: §3.3 · pp. 5–6

3.4 Model Merging

  • SCE selects the top τ% update elements per parameter matrix, calculates matrix-level coefficients from their squared magnitudes, and erases elements whose directions are in the minority.The filtered updates are merged using the calculated coefficients and added to the pivot parameters.Source: §3.4 · p. 6

3.5 Discussions

  • FUSECHAT uses pairwise rather than many-to-one fusion because direct fusion is difficult, and it supports plug-and-play addition of new sources at any scale.Adding a source requires fusing it with the pivot to obtain a target and merging that target with the existing FUSECHAT model.Source: §3.5 · p. 6

4 Experiments

Experiments fuse six chat LLMs of varied architectures and scales into FuseChat-7B, then evaluate it and its components on AlpacaEval 2.0 and MT-Bench. The reported results favor FuseChat-7B over a range of baselines, with analyses varying the pivot, data scale, merging method, and token alignment.

4.1 Experimental Setup

  • Experiments use six source chat LLMs to create five same-structure targets from OpenChat-3.5-7B, then merge them; training uses about 95,000 dialogues drawn half from the pivot dataset and half from open sources.Target-model training runs for three epochs; evaluation uses AlpacaEval 2.0 and MT-Bench, and baselines span proprietary, source, ensemble, and fused LLMs.Source: §4 · p. 7, §4.1 · p. 7

4.2 Overall Results

  • FuseChat-7B improves on OpenChat-3.5-7B SFT by 18.34% and 7.27% in the two metrics, and beats most 162B ensembles except Top1-GPT4 on MT-Bench while being 23x smaller and independent of GPT-4.Table 1 compares FuseChat-7B and baselines on AlpacaEval 2.0 and MT-Bench; the reported percentages are relative to OpenChat-3.5-7B SFT.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Different Pivot LLM

  • With Starling-LM-7B-alpha as pivot, FuseChat-Starling-7B improves over the pivot by 17.62% on AlpacaEval 2.0 and 2.14% on MT-Bench.Starling-LM-7B-alpha SFT alone did not improve performance, while pairwise fusion and merging improved results using the same training data.Source: §4.3 · p. 9

4.4 Dataset Scaling

  • The final fused LLM’s MT-Bench performance consistently improves as pairwise-fusion training data scales from 10k to 95k.Source: §4.4 · p. 9

4.5 Analysis of Model Merging

  • FuseChat-7B SCE outperforms Linear, TA, TIES, and DARE merging on both AlpacaEval 2.0 and MT-Bench.On MT-Bench, Average Score rises from 7.24 to 7.38 as the number of integrated target LLMs increases from one to five, then remains stable after adding OpenChat-3.5-7B SOLAR.Source: §4.5 · p. 9

4.6 Ablation Studies for SCE

  • Removing SCE’s select operation substantially degrades performance on both benchmarks, while removing both select and erase decreases MT-Bench performance.The authors associate selection with retaining high-variance salient fusion-vector elements and erasure with resolving parameter interference.Source: §4.6 · pp. 9–10

4.7 Analysis of Token Alignment

  • For InternLM2-Chat-20B aligned to OpenChat-3.5-7B, the statistics-based MS method consistently outperforms exact-matching EM and minimal-edit-distance MinED.The authors attribute the improvement to using token-mapping statistics in the dataset for distribution-level alignment.Source: §4.7 · p. 10

5 Conclusion

FUSECHAT fuses structurally and scale-varied chat LLMs through pairwise knowledge transfer into same-sized targets, then merges those targets in parameter space. It outperforms baselines across model scales and is comparable to Mixtral-8x7B-Instruct on MT-Bench while approaching GPT-3.5-Turbo-1106.

5 Conclusion

  • FUSECHAT first creates same-structure, same-size targets through lightweight pairwise fine-tuning, then merges them using SCE coefficients based on parameter-update magnitudes.The coefficients use update magnitudes before and after fine-tuning.Source: §5 · p. 10
  • FUSECHAT is comparable to Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench, with superiority across model scales on two instruction-following benchmarks.The benchmarks are AlpacaEval 2.0 and MT-Bench.Source: §5 · p. 10
  • Unlike mixture-of-experts approaches, knowledge fusion integrates diverse LLMs into a single model without additional inference-time or memory overhead.The paper notes that MoEs load multiple experts at inference, raising time and memory requirements.Source: §5 · p. 10

6 Limitations and Future Work

The approach relies on constructing a diverse knowledge-fusion dataset, which requires substantial data-engineering effort and may constrain scalability. Its evaluation does not address knowledge comprehension or hallucination mitigation.

6 Limitations and Future Work

  • Substantial data-engineering effort to construct a diverse knowledge-fusion dataset may limit the methodology’s scalability.The authors suggest more efficient data-synthesis techniques to expand the dataset’s scope.Source: §6 · p. 10
  • The study improves chat-model capabilities but does not address knowledge comprehension or hallucination mitigation.The authors call for further investigation of the approach’s applicability and effectiveness in these areas.Source: §6 · p. 10
  • Future work should investigate more efficient data-synthesis techniques to expand the knowledge-fusion dataset’s scope.The current process relies on substantial data-engineering effort.Source: §6 · p. 10

A Details of Token Alignment

The token-alignment strategies represent source-to-pivot token mappings as matrices, with weights determined by exact matches, edit distance, or observed mapping frequency. The MS method handles mapping cases where EM and MinED may produce inaccurate alignments, and also supports many-to-one mappings.

A Details of Token Alignment

  • The mapping matrices encode source-token alignment probabilities to pivot tokens, using exact matches for EM, inverse edit distance for MinED, and mapping frequency for MS.The matrix values derive from each strategy’s alignment method.Source: A Details of Token Alignment · p. 14
  • For 1-1 or 1-n mappings, EM and MinED may be inaccurate: EM can align “flow_” with “<unk>”, while MinED can map “flow_” to “flown” or “belo_” to “below”; MS achieves more accurate alignments.Source: A Details of Token Alignment · p. 14
  • For n-1 mappings, MS combines multiple source distributions with a WMS-weighted average before processing the unified distribution like a 1-1 mapping.The passage identifies n-1 mapping as applicable only to MS.Source: A Details of Token Alignment · p. 14

B Details of Model Merging

The merging baselines use fixed or searched coefficients, while SCE selects salient elements and derives coefficients from delta-parameter magnitudes. The selected settings are specified for each method.

B Details of Model Merging

  • SCE searches salient-element thresholds from 10% to 90%, selects 10%, and derives merging coefficients automatically from delta-parameter magnitudes.The threshold search uses increments of 10%.Source: B Details of Model Merging · p. 14
  • TIES and DARE use the optimal trim/drop rate of 0.4, zeroing the bottom or random 40% of delta parameters, and average target-LLM coefficients.The search range is 0.1 to 0.9.Source: B Details of Model Merging · p. 14
  • Linear averages target-LLM parameters, while TA searches scaling coefficients from 0.3 to 0.5 and selects 0.3.The stated search values are 0.3, 0.4, and 0.5.Source: B Details of Model Merging · p. 14

C Details of Training Dataset

FUSECHAT-MIXTURE combines human-written and model-generated examples spanning general instruction-following and specialized skills. After cleaning and length-based splitting, the final dataset contains 95,000 samples.

C Details of Training Dataset

  • It also uses 9,500 human-annotated No-Robots examples, 6,200 GPT-4-generated ShareGPT dialogues, and 5,000 examples from human-annotated Oasst-Top1.Oasst-Top1 is described as a refined version of Oasst1.Source: C Details of Training Dataset · p. 15
  • Specialized data includes 10,000 MetaMathQA examples for math, 10,000 OSS-Instruct examples synthesized from open-source code, and 10,000 GPT-4-generated Evol-Alpaca code instructions.MetaMathQA is augmented from GSM8K and MATH; Evol-Alpaca uses evol-instruct.Source: C Details of Training Dataset · p. 16
  • The mixture includes 10,000 Python-Code examples generated by GPT-3.5 and GPT-4, 20,000 filtered Orca-Best examples, and all 16,000 Capybara conversations.Capybara is a collection of high-quality multi-turn synthetic conversations; Orca-Best is filtered from Orca's GPT-4 (1M) partition by length and instruction clustering.Source: C Details of Training Dataset · pp. 15–16
  • The cleaned FUSECHAT-MIXTURE contains 95,000 samples, with long conversations split into blocks of at most 2048 tokens.Processing also removes instances containing non-English or special characters.Source: C Details of Training Dataset · pp. 14, 16

D Details of Baselines

The experiments compare FuseChat against proprietary, source, ensemble, and fused LLM baselines. The fused baselines include single-source fine-tuning, simultaneous multi-source fusion, and pairwise-fusion targets.

D Details of Baselines

  • Proprietary baselines are GPT-3.5-Turbo-1106, Claude-3-Opus, and GPT-4-1106-Preview.The source-model baselines are OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B.Source: D Details of Baselines · p. 16
  • Ensemble baselines select a top-ranked source response using perplexity, an LLM-Blender ranker, or GPT-4 judgment.Only the LLM-Blender ranker is used because its fuser is constrained by maximum sequence length; Top1-GPT4 is an upper-bound comparison because evaluation also relies on GPT-4.Source: D Details of Baselines · p. 16
  • Fused baselines include OpenChat-3.5-7B SFT, a single-source fine-tuning scenario, and OpenChat-3.5-7B Multi, which fuses multiple sources simultaneously.The simultaneous approach is described as resembling FUSELLM.Source: D Details of Baselines · p. 16
  • Five additional fused baselines are OpenChat-3.5-7B targets created by pairwise fusion with Starling, SOLAR, InternLM, Mixtral, or Qwen.OpenChat-3.5-7B is the pivot model for these pairwise-fusion targets.Source: D Details of Baselines · p. 16

E Evaluation of Additional Benchmarks

The paper additionally evaluates six general benchmarks, while noting that FuseChat-7B's training data emphasizes alignment rather than general knowledge. Improvements on these benchmarks are reported as less significant than on AlpacaEval 2.0 and MT-Bench.

E Evaluation of Additional Benchmarks

  • FuseChat-7B's improvements are less significant on six general benchmarks than on AlpacaEval 2.0 and MT-Bench, consistent with its alignment-focused training data.The benchmarks are MMLU-Pro, PIQA, BoolQ, GPQA, GSM8K, and IFEval; Table 6 presents their comparison results.Source: E Evaluation of Additional Benchmarks · p. 17, Table 6 · p. 17
  • The six benchmarks assess knowledge understanding, question-answering, common-sense reasoning, and instruction-following.The experiments primarily focus on alignment training data and chat-model benchmarks.Source: E Evaluation of Additional Benchmarks · pp. 16–17

F Case Studies

The case studies compare pairwise-fused target LLMs and the further-merged FuseChat-7B against OpenChat-3.5-7B SFT. Separate MT-Bench tables show outputs from the individual targets and FuseChat-7B.

F Case Studies

  • The case studies examine individual strengths of five pairwise-fused targets and the collective strengths of FuseChat-7B, using OpenChat-3.5-7B SFT as the baseline.The targets pair OpenChat-3.5-7B with Starling, SOLAR, InternLM, Mixtral, or Qwen.Source: F Case Studies · p. 17
  • MT-Bench case-study tables show answers generated by OpenChat-3.5-7B SFT, Starling, and SOLAR targets.These outputs are presented in Tables 7, 8, and 9, respectively.Source: Table 7 · p. 18, Table 8 · p. 19, Table 9 · p. 20
  • Further MT-Bench case-study tables show answers from the InternLM, Mixtral, and Qwen targets, followed by FuseChat-7B.These outputs are presented in Tables 10–13.Source: Table 10 · p. 21, Table 11 · p. 22, Table 12 · p. 23

Related papers

Loading 2408.07990v1…