PaperlayerView exact revision on arXiv

Source-linked AI summary

EMR-Merging: Tuning-Free High-Performance Model Merging

Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, Wanli Ouyang

arXiv:2405.17461v2cs.LGcs.CV

TL;DR

Existing merging methods can leave a performance gap from individual models or MTL, and may require data or training to tune. EMR-MERGING combines an elected unified model with lightweight task-specific masks and rescalers; the paper reports strong performance across classical and newly established settings, comparable to MTL or individual models. Its scope is pretrained models fine-tuned for downstream tasks.

  • Problem

    Model merging still faces performance gaps from individual models or MTL, while existing methods can require data-based tuning or additional training.

    Source: §1 · p. 2

  • Method

    EMR-MERGING elects a unified model and generates lightweight task-specific masks and rescalers to align its direction and magnitude with each specific model.

    Source: §1 · p. 2

  • Results

    EMR-MERGING is reported as comparable to MTL or individual models and effective across classical and newly established vision, NLP, PEFT, and multimodal settings.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    The method offers a tuning-free model-merging approach validated across multiple model types and task settings.

    Source: §1 · p. 3, §5 · p. 10

    EMR-MERGING requires some additional memory for its modulators and, as a task-vector method, cannot generalize to models trained from scratch.

    Source: G Limitations and future works · p. 22

Abstract

from arXiv · show

The success of pretrain-finetune paradigm brings about the release of numerous model weights. In this case, merging models finetuned on different tasks to enable a single model with multi-task capabilities is gaining increasing attention for its practicability. Existing model merging methods usually suffer from (1) significant performance degradation or (2) requiring tuning by additional data or training. In this paper, we rethink and analyze the existing model merging paradigm. We discover that using a single model's weights can hardly simulate all the models' performance. To tackle this issue, we propose Elect, Mask & Rescale-Merging (EMR-Merging). We first (a) elect a unified model from all the model weights and then (b) generate extremely lightweight task-specific modulators, including masks and rescalers, to align the direction and magnitude between the unified model and each specific model, respectively. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance. We find that EMR-Merging shows outstanding performance compared to existing merging methods under different classical and newly-established settings, including merging different numbers of vision models (up to 30), NLP models, PEFT models, and multi-modal models.

1 Introduction

The paper argues that a single merged model cannot fully simulate all task-specific models, while existing methods still face performance gaps and often require data or training for tuning. EMR-MERGING combines a unified model with lightweight task-specific modulators and is reported to approach MTL and individual-model performance across a range of settings.

1 Introduction

  • Model merging combines existing weights to reduce deployment costs, while MTL requires joint training and can face high compute costs and data unavailability.The introduction situates merging in the growth of released pretrained and finetuned checkpoints.Source: §1 · p. 1
  • Existing methods retain a gap from individual models or MTL, and their performance gains may depend on data-based tuning or training.The introduction groups prior approaches as weighted averaging, task-vector methods, and preprocessing techniques.Source: §1 · p. 2
  • The paper argues that one model weight is suboptimal for simulating all task-specific models, particularly as model count or task difficulty increases.Its alternative stores a shared unified weight alongside significant but lightweight task-specific parts.Source: §1 · p. 2
  • EMR-MERGING is the only merging method comparable to MTL and individual models in average accuracy across eight vision tasks.The paper also reports significant improvement over the strongest baseline across classical and newly established settings.Source: Figure 1 · p. 2, §1 · p. 2
  • The approach elects a unified model and adds lightweight task-specific masks and rescalers to align direction and magnitude with each original model.The method requires no data, tuning, or additional training.Source: §1 · pp. 2–3
  • The paper evaluates EMR-MERGING on ViTs, PEFT and GPT-2 models, then extends tests to 30 ViTs, RoBERTa, and BEiT3 models.The settings cover vision, language, and multimodal tasks.Source: §1 · pp. 2–3

2 Related Work

Related work frames model merging as an alternative to joint multi-task training and reviews methods that trade interference reduction against tuning, additional matrices, or training. It also positions EMR-MERGING as applicable to full finetuned models and PEFT modules.

2 Related Work

  • Fisher-Merging and RegMean use additional information matrices, while Task Arithmetic, Ties-Merging, and AdaMerging rely on task vectors with tuning or training requirements.DARE [90] drops and rescales task-vector elements, but its validation covers limited task counts and its performance gain is limited.Source: §2 · p. 3
  • Unlike MTL, which trains jointly on task data, model merging combines existing weights without training data or additional training.The paper associates MTL with high computational costs and data-privacy-related availability limits.Source: §2 · p. 4
  • EMR-MERGING is based on the pretrain-finetune paradigm and is presented as applicable to both full finetuned models and PEFT modules.The paper concludes that it requires no tuning while showing performance across various settings.Source: §2 · p. 4

3 Method

EMR-Merging addresses the mismatch between one merged model and multiple task models by electing a unified vector and applying lightweight task-specific masks and rescalers. Its analyses and visualizations report reduced distance and closer task-model alignment.

3.1 Motivation

  • A single merged model can poorly represent all task-specific weights, motivating a unified vector with lightweight task-specific modulators to approximate them more closely without tuning.The modulators are described as much smaller than a model.Source: §3.1 · p. 4

3.2 ELECT, MASK & RESCALE-MERGING

  • EMR-MERGING elects a unified task vector using shared sign direction and maximum compatible parameter magnitudes, then applies task-specific masks and rescalers before adding it to pretrained weights.Masks align direction; rescalers align magnitude, and the process requires no data, tuning, or additional training.Source: §3.2 · pp. 4–5

3.4 Empirical analysis

  • Figure 3 visualizations show each EMR-MERGING procedure bringing results closer to individual models, with clearer t-SNE separation and more precise Grad-CAM targets.The visualizations cover partial results for eight ViT-B/32 models across tasks.Source: Figure 3 · p. 5, §3.4 · p. 5

3.3 Theoretical analysis

  • The analysis states that masks reduce the distance between the merged model and each task model, while rescalers minimize that distance after masking.The mask result is framed relative to the distance before masking.Source: §3.3 · p. 5

3.4 Empirical analysis

  • EMR-MERGING significantly reduces sign conflicts and L2 distance while improving cosine similarity versus other merging methods, indicating effective task-specific model-weight approximation.Figure 4 compares merged-model weights with task-specific model weights.Source: Figure 4 · p. 6, §3.4 · p. 6

4 Experiment Validation

Experiments compare EMR-MERGING with established merging baselines across vision, NLP, PEFT, and multimodal settings. The reported results favor EMR-MERGING across these settings, including when merging 30 vision models.

4.1.1 Merging 8 ViTs.

  • 7.6% and 2.7%: EMR-MERGING improves performance over existing merging methods for ViT-B/32 and ViT-L/14, respectively.Its t-SNE and Grad-CAM visualizations are reported as closest to those of individual models.Source: §4.1.1 · pp. 6–7, Figure 5 · p. 7

4.1.2 Merging 30 ViTs.

  • 3.48%: EMR-MERGING reduces performance decay to this level from nearly 25% when merging 30 vision models.RegMean performs best among existing methods in this setting, yet still has a performance decay of nearly 25%.Source: §4.1.2 · pp. 7–8, Table 4 · p. 7

4.2.1 Merging fully finetuned RoBERTa models

  • EMR-MERGING outperforms existing methods on every task when merging fully finetuned RoBERTa models on GLUE.The authors report that DARE may not improve performance in this multiple-model setting, possibly because random dropping no longer resolves task-vector conflicts.Source: §4.2.1 · p. 8, Table 5 · p. 8

4.2.2 Merging fully finetuned GPT-2 models

  • Over 10%: EMR-MERGING outperforms other merging methods and reduces GPT-2 merging’s performance degradation from 12% to 1.6%.The experiment merges fully finetuned GPT-2 models on seven text-classification tasks.Source: §4.2.2 · p. 8, Table 6 · p. 8

4.2.3 Merging PEFT models

  • 2.2% and 0.7%: EMR-MERGING improves average accuracy over methods without validation data and methods using validation data, respectively, on PEFT tasks.It outperforms all merging methods in the reported comparison.Source: §4.2.3 · pp. 8–9, Table 7 · p. 9

4.3 Merging multi-modal models

  • EMR-MERGING performs best across all five vision-language tasks, regardless of the evaluation metric used.The comparison covers BEiT3 models finetuned for image classification, visual question answering, visual reasoning, image captioning, and image-text retrieval.Source: §4.3 · p. 9, Table 8 · p. 9

4.4 Merging different number of models

  • As the number of merged models increases, EMR-MERGING outperforms Ties-Merging and its performance decreases more slowly.Its task-specific modulators add some storage over Ties-Merging but substantially less than storing individual models; a 1-bit mask is 32 times smaller than a single 32-bit model, and a rescaler is a single parameter. Figure 6 compares parameter counts and average normalized performance across model counts.Source: §4.4 · pp. 9–10, Figure 6 · p. 9

4.5 Ablation Study

  • In ablations, electing outperforms replacing that procedure with other task-vector merging methods, while masking and rescaling improve those alternatives by 6.8%, 5.0%, and 6.6%, respectively.With election alone, adding masking and rescaling improves performance by 27.5% and 12.6%, respectively; applying both yields up to 29.4% improvement over applying either alone.Source: §4.5 · p. 10, Table 9 · p. 10, Table 10 · p. 10

5 Conclusion

The conclusion presents EMR-MERGING as a tuning-free approach validated across vision, NLP, PEFT, and multimodal benchmarks. The accompanying algorithm takes finetuned models and a pretrained model as input and produces a unified task vector with task-specific masks and rescalers.

5 Conclusion

  • EMR-MERGING requires no data access or additional training for tuning, and its effectiveness is validated across classical and newly established vision, NLP, PEFT, and multimodal benchmarks.The conclusion attributes existing methods’ severe performance degradation to the difficulty of using a single model to simulate all models’ performance.Source: §5 · p. 10

A Algorithm flow of EMR-MERGING

  • Algorithm 1 takes finetuned models and a pretrained model as input and outputs a unified task vector, task-specific masks, and task-specific rescalers.Its listed procedure includes creating task vectors, electing a unified task vector, and generating the masks and rescalers.Source: A Algorithm flow of EMR-MERGING · p. 15

B Theoretical analyses

The theoretical analysis considers models finetuned from a shared pretrained model and frames merging as reducing the distance from a unified model to each task-specific model. It analyzes how masks and rescalers affect that distance.

B Theoretical analyses

  • The analysis assumes all models were finetuned from the same pretrained model and defines the goal as minimizing the unified model’s distance to each task-specific model without dataset use.The shared-pretraining premise lets the formulation be rewritten in terms of task vectors.Source: B Theoretical analyses · p. 15
  • Applying task-specific masks reduces the distance between the unified model and individual models, with DisM ≤ Dis.The masks disable unified-vector elements whose signs conflict with the corresponding task-specific elements.Source: B Theoretical analyses · pp. 15–16
  • The rescaler setting minimizes the distance between the masked unified task vector and an individual model, yielding DisM,λ ≤ DisM.The analysis derives the rescaler by setting the distance’s derivative with respect to λ_i to zero.Source: B Theoretical analyses · pp. 16–17
  • After masking and rescaling, sign conflicts and L2 distance decrease while cosine similarity improves between merged and task-specific models.Figure 7 compares these measures across merging methods and EMR-MERGING procedures.Source: Figure 7 · p. 16, B Theoretical analyses · p. 17

C Baseline Methods

The section contrasts common merging baselines: joint training, weight averaging, weighted averaging, task-vector methods, and parameter dropping. Their methods differ in the information or operations used to form a merged model.

C Baseline Methods

  • Traditional multi-task learning jointly trains one model on datasets from all tasks, while weight averaging element-wise averages the model weights.The cited prior work verifies weight averaging for fine-tuned models from the same pretraining.Source: C Baseline Methods · p. 18
  • Fisher Merging weights parameters by importance from Fisher information matrices [46], whereas RegMean uses a closed-form solution and requires inner-product matrices before merging.RegMean’s formulation considers merging K linear model weights.Source: C Baseline Methods · p. 18
  • Task Arithmetic adds task vectors—the differences between fine-tuned and pretrained weights—while Ties-Merging removes redundant parameters and resolves symbol conflicts.AdaMerging learns coefficients per task or layer, and AdaMerging++ applies Ties-Merging before learning them.Source: C Baseline Methods · pp. 18–19
  • DARE randomly drops most task-vector parameters before merging to potentially mitigate interference, using drop rates of 90% or even 99% [90].The section also identifies DARE-preprocessed RegMean and Task Arithmetic as evaluated combinations.Source: C Baseline Methods · p. 19, Table 13 · p. 19

D.1 Merging ViT-B/16 models on 8 tasks

On eight vision tasks, EMR-Merging improves the performance of merged ViT-B/16 models over AdaMerging++.

D.1 Merging ViT-B/16 models on 8 tasks

  • 5.6% performance improvement over AdaMerging++ [85] is reported for EMR-Merging when merging ViT-B/16 models on eight vision tasks.The authors state that the experiment follows Section 4.1.1 settings and report accuracy in Table 11.Source: §D.1 · p. 19
  • The comparison merges ViT-B/16 models on eight vision tasks using the settings from Section 4.1.1.The reported measure is accuracy, shown in Table 11.Source: §D.1 · p. 19
  • The authors describe the 5.6% improvement over AdaMerging++ [85] as further evidence of EMR-Merging’s effectiveness.This is the authors’ characterization of the result.Source: §D.1 · p. 19

D.2 Merging ViT-B/32 models on 9 tasks (ImageNet-1K added)

Across these supplementary experiments, EMR-Merging reports an improvement of up to 20% on nine ViT-B/32 tasks, while the accompanying material examines DARE, hyperparameter settings, model counts, and modulator statistics.

D.2 Merging ViT-B/32 models on 9 tasks (ImageNet-1K added)

  • Following Section 4.1.1 settings, EMR-Merging was compared with existing merging methods across nine tasks after adding ImageNet-1K, showing a much more significant improvement of up to 20%.Source: §D.2 · p. 19

D.3 DARE’s experimental results and causes

  • When merging eight models, DARE generally fails, which the authors attribute to random dropping’s inapplicability as conflicts become harder to avoid across multiple models.At low drop rates, DARE works on a few tasks; with two or three models, dropping can significantly reduce interference.Source: §D.3 · p. 19

D.4 Results under different hyper-paramerter settings

  • Table 14 presents Task Arithmetic [30], Ties-Merging [84], Ties-Merging with DARE [90], and RegMean [33] under different hyperparameter settings.The table caption identifies λ as the task-vector merging coefficient, P as DARE’s drop rate, and a as RegMean’s non-diagonal multiplier.Source: §D.4 · p. 20, Table 14 · p. 20

D.5 Detailed information for merging different number of models

  • Quantified, task-specific results for merging different numbers of ViT-B/32 models are provided in Table 15.The supplied passage does not state the table’s numerical outcomes.Source: §D.5 · p. 20, Table 15 · p. 21

D.6 Sparsity of masks and values of rescalers.

  • Table 16 reports mask sparsity and rescaler values for ViTs on eight vision tasks and RoBERTa models on eight language tasks.The table caption defines sparsity as the ratio of non-zero mask items.Source: Table 16 · p. 21, §D.6 · p. 21

E More visualization results

The appendix adds t-SNE and Grad-CAM visualizations and specifies how the weight-comparison metrics in Figures 4 and 7 are calculated.

E More visualization results

  • Additional visualizations compare existing merging methods with EMR-Merging using t-SNE and Grad-CAM.The t-SNE results appear in Figure 8 and Grad-CAM results in Figure 9.Source: E More visualization results · p. 21

F Configuration of Fig. 4 and Fig. 7

  • Figures 4 and 7 compare sign conflicts, L2 distance, and cosine similarity between merged-model weights and individual-model weights.The passage specifies reporting average values across individual models.Source: F Configuration of Fig. 4 and Fig. 7 · p. 22
  • Sign conflicts are calculated as the average ratio of elements whose signs differ, while L2 distance and cosine similarity are calculated between flattened weight vectors.Each metric compares merged weights with each individual model’s weights.Source: F Configuration of Fig. 4 and Fig. 7 · p. 22

G Limitations and future works

EMR-Merging requires extra memory for lightweight task-specific modulators and, like other task-vector methods, is limited to pretrained-and-finetuned models. Future work includes improving merging performance, broadening model coverage, and exploring low-bit-width quantization.

G Limitations and future works

Loading 2405.17461v2…