PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li

arXiv:2311.03099v3cs.CLcs.LG

TL;DR

The paper addresses whether language models can acquire task-specific abilities by absorbing parameters from homologous models without retraining or GPUs. It introduces DARE to sparsify redundant SFT delta parameters before merging models, and reports substantial parameter removal alongside preserved or improved capabilities in merged models.

  • Problem

    The paper examines how to obtain language models with additional task-specific abilities without retraining or GPU resources.

  • Method

    DARE randomly drops and rescales SFT delta parameters, then supports merging multiple homologous task-specific models through existing parameter-fusion methods.

  • Results

    DARE removes 90% or even 99% of redundant SFT delta parameters, while merged models retain or improve capabilities across encoder- and decoder-based evaluations.

  • Takeaways & Limitations

    DARE provides a CPU-only, plug-and-play route to reducing SFT parameter redundancy and creating a single model with diverse capabilities.

  • Takeaways & Limitations

    DARE works well when SFT delta parameters are relatively small, such as within 0.002, but may fail otherwise.

Abstract

from arXiv · show

In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio $p$ And REscales the remaining ones by $1 / (1 - p)$ to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.

1. Introduction

The paper presents DARE, which sparsifies redundant SFT delta parameters and enables homologous task-specific language models to be merged into one model with diverse abilities. Experiments across encoder- and decoder-based models show substantial parameter reduction and improved or preserved merging performance without retraining or GPUs.

  • Method: DARE randomly drops delta parameters at rate p and rescales the remainder by 1/(1−p) to approximate the original embeddings.The method is designed as a plug-in for sparsifying SFT parameter differences before model merging.
  • Model merging: DARE enables multiple homologous SFT LMs to be fused into one model with diverse capabilities while preserving the abilities unlocked by SFT.Sparsifying each model before merging can potentially mitigate parameter interference.
  • Findings: 90% or even 99% of SFT delta parameters can be removed without significantly affecting performance across multiple LM backbones.This redundancy appears in BERT, RoBERTa, LLaMA, Llama 2, and Code Llama models.
  • Results: 3.10% for LM & Math & Code vs. LM on AlpacaEval, 3.18% for LM & Math vs. Math on GSM8K, and 19.57% for LM & Code vs. Code on MBPP.These are reported improvements from merging decoder-based LMs with DARE and Task Arithmetic.
  • Scope: SFT delta parameters usually stay within 0.002, whereas continuous pre-training can raise them to around 0.03 and make DARE infeasible.Dropping only 10% fine-tuned parameters in that setting can cause performance to approach zero.

2. Related Work

The related work situates DARE among supervised fine-tuning, network pruning, and model-merging methods. These areas respectively provide task adaptation, parameter reduction, and strategies for combining task-specific models.

  • Supervised Fine-Tuning: Supervised fine-tuning imparts particular abilities to pre-trained LMs by optimizing them on task-specific data.The paper distinguishes full fine-tuning from parameter-efficient fine-tuning.
  • Network Pruning: Network pruning eliminates unnecessary neural-network parameters while maintaining model performance to reduce computational costs.Magnitude-based pruning is identified as a classical pruning approach.
  • Model Merging: Model merging combines multiple task-specific models into a single model with diverse abilities.The paper reviews Fisher Merging, RegMean, and TIES-Merging as representative methods.

3. Methodology

DARE sparsifies redundant SFT delta parameters by randomly dropping them and rescaling those retained, then supports merging homologous task-specific models. Its rescaling preserves embeddings in expectation, while sparsification is intended to reduce interference during parameter fusion.

  • SFT delta parameters are defined as the parameter difference between a fine-tuned LM and its shared pre-trained backbone.
  • DARE: A Simple Approach for Reducing Delta Parameter Redundancy: DARE randomly drops delta parameters at rate p, rescales the remainder by 1/(1 − p), and adds them to the pre-trained parameters for inference.
  • DARE: A Simple Approach for Reducing Delta Parameter Redundancy: By setting the rescaling factor to 1/(1 − p), DARE matches the expected embeddings of the original model despite removing most delta parameters.
  • The method is practical only when the drop rate is properly set; its tolerance increases with model parameter size, whereas removing fine-tuned parameters can catastrophically reduce performance.
  • Merging Models with DARE: DARE is proposed as a plug-in that sparsifies delta parameters before established merging methods fuse multiple SFT models into one model.
  • Merging Models with DARE: DARE can be applied to Average Merging, Fisher Merging, RegMean, and TIES-Merging to reduce delta-parameter redundancy before fusion.

4. Experiments

Experiments across encoder- and decoder-based LMs evaluate DARE for sparsifying SFT delta parameters, preserving model abilities, and merging task-specific models. DARE generally supports high drop rates and improves merging, especially for decoder-based LMs, but requires small delta parameters and well-fine-tuned source models.

  • Experimental Setup: DARE was evaluated on encoder- and decoder-based LMs for reducing SFT delta-parameter redundancy and merging models.The experiments cover instruction following, mathematical reasoning, code generation, and GLUE tasks.
  • Extreme Redundancy in SFT Delta Parameters: 90% of SFT delta parameters can be removed without significantly decreasing performance, and the drop rate can reach 99% in some cases.Larger LMs tolerate higher drop rates; WizardMath-70B performs well at p = 0.99 while WizardMath-7B and WizardMath-13B fail.
  • Merging Models with DARE on SFT LMs: DARE often improves model merging, with decoder-based gains of 3.10% on AlpacaEval, 3.18% on GSM8K, and 19.57% on MBPP over specified source-model baselines.For encoder-based LMs, average improvements were 0.58% for Average Merging, 0.36% for Task Arithmetic, 0.37% for Fisher Merging, -0.03% for RegMean, and 0.84% for TIES-Merging.
  • Prerequisites and Limitations: Effective model merging requires each source model to be well fine-tuned.The paper reports that llama-2-13b-code-alpaca was not well fine-tuned for code generation, which affected the merging performance.
  • Merging Models with DARE on SFT LMs: DARE’s merging improvements are more pronounced for decoder-based LMs than encoder-based LMs.The paper suggests this may relate to decoder-based LMs accommodating more abilities because of their substantially larger sizes.
  • Merging Models with DARE on SFT LMs: A merged 7-billion-parameter LM achieved first place on the Open LLM Leaderboard as of January 28, 2024, using only CPUs.The merged models also improved over the source models they were built upon.
  • Importance of the Rescale Operation: DARE preserves original embeddings with cosine similarities above 0.95 after removing 90% of delta parameters, whereas DropOnly degrades sharply at higher drop rates.Without rescaling, WizardMath-7B similarities decrease to about 0.85 at p = 0.5 and 0.68 at p = 0.9, and performance generally worsens.

5. Conclusion

The work identifies extreme redundancy in SFT delta parameters and introduces DARE to reduce that redundancy without data, retraining, or GPUs. DARE also supports merging task-specific models into one model with diverse abilities.

  • DARE reduces the number of SFT delta parameters without data, retraining, or GPUs.
  • 90% or even 99% of SFT delta parameters can be dropped without sacrificing much performance.
  • DARE enables multiple task-specific fine-tuned models to be merged into one model with diverse abilities.
  • Experiments with encoder- and decoder-based LMs demonstrate DARE’s effectiveness in reducing delta-parameter redundancy and facilitating model merging.

Impact Statement

The paper situates model merging as a way to obtain one model with diverse capabilities without accessing original training data. It also notes that harmful information may persist, motivating careful regulation.

  • Harmful information such as gender bias and racial discrimination may still exist when using this approach.
  • The paper advocates careful regulation by communities and authorities.

A.4. Details of Grid Search on Hyperparameters of Model Merging Methods for Encoder-based LMs

The encoder-based model-merging experiments search specified hyperparameter ranges, while the 7-billion-parameter Super Mario models use particular backbones, drop rates, and scaling settings.

  • DARE’s drop rate p is searched over [0.1, 0.2, · · ·, 0.9], selecting the setting with the best performance.
  • Super Mario v1 merges NeuralBeagle14-7B and Turdus with DARE drop rate p=0.3 and Task Arithmetic scaling term 0.8.
  • The Open LLM Leaderboard ranks models by average score across six benchmarks.
  • Turdus leaderboard results are unavailable, so Beagle14-7B performance is reported instead.

B.1. Additional Results of Delta Parameter Redundancy of Decoder-based LMs

Additional decoder-based experiments examine how different DARE drop rates affect AlpacaEval, MATH, and MBPP performance. The results show task-dependent sensitivity to aggressive dropping.

  • At drop rate 0.9, WizardLM-70B’s AlpacaEval performance drastically declines.
  • WizardMath-70B and WizardCoder-Python-34B show different behavior from WizardLM-70B under the same observation.
  • The paper suggests that harder instruction-following may require more SFT delta parameters and exhibit stronger parameter dependencies.
  • Performance is evaluated on AlpacaEval, MATH, and MBPP across various DARE drop rates.

B.2. Additional Results of Merging Encoder-based LMs

Additional results report encoder-based LM merging performance on GLUE, as shown in Figure 13.

  • Figure 13 presents the performance of merging encoder-based LMs on GLUE.

B.3. Additional Results of Comparisons between DARE and DropOnly

Additional experiments compare DARE with DropOnly across decoder-based LM benchmarks and encoder-based LM results on GLUE.

  • Figure 14 compares DARE and DropOnly on AlpacaEval, MATH, HumanEval, and MBPP using decoder-based LMs.
  • Figure 15 compares DARE and DropOnly on GLUE using encoder-based LMs.

B.4. Additional Results of Comparisons between DARE and MP

Additional results compare DARE with magnitude-based pruning and examine delta-parameter ranges and fine-tuned-parameter removal across encoder- and decoder-based LMs.

  • Figures 16 and 17 compare DARE with magnitude-based pruning on decoder-based benchmarks and encoder-based GLUE results, respectively.
  • Figures 18–20 show SFT delta-parameter ranges for decoder- and encoder-based LMs.Decoder-based results use a random 10% sample of delta parameters, while encoder-based results include all delta parameters.
  • Table 6 reports decile statistics for delta-parameter ranges across decoder- and encoder-based LMs.The deciles are obtained by sorting the full ranges and indexing positions from 0 through 100% in 10% increments.
  • Figure 21 shows results from removing fine-tuned parameters on GLUE using encoder-based LMs.
Loading 2311.03099v3…