PaperlayerView exact revision on arXiv

Source-linked AI summary

TIES-Merging: Resolving Interference When Merging Models

Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, Mohit Bansal

arXiv:2306.01708v2cs.LGcs.AIcs.CLcs.CV

TL;DR

Existing model-merging methods can lose performance because redundant parameter changes and sign disagreements interfere across task-specific models. TIES-Merging trims redundant changes, resolves signs, and merges aligned values, outperforming prior methods across diverse settings.

  • Problem

    Existing merging methods ignore interference among parameters from different models, which can cause performance drops when multiple task-specific models are merged.

  • Method

    TIES-Merging trims low-magnitude task-vector changes, resolves sign conflicts, and averages only values aligned with the elected sign.

  • Results

    TIES-Merging outperforms existing merging methods across modalities, domains, model sizes, architectures, task counts, and fine-tuning settings, including 1.0% and 4.4% gains for T5-base and T5-large out-of-domain evaluation.

  • Takeaways & Limitations

    TIES-Merging enhances merged multitask-model performance across varied settings while using a simple recipe with fixed hyperparameters.

  • Takeaways & Limitations

    Merging relies on common initialization and architecture, still lags behind simultaneous multitask training, and lacks a clear checkpoint-selection strategy for domain-specific multitask models.

Abstract

from arXiv · show

Transfer learning - i.e., further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference. Our code is available at https://github.com/prateeky2806/ties-merging

1 Introduction

Model merging combines task-specific fine-tuned models into one multitask model without additional training, but parameter interference can cause performance loss. TIES-Merging addresses redundant changes and sign disagreements before merging, and outperforms prior methods across diverse settings.

  • Fine-tuned models improve task performance and efficiency but require separate storage and deployment and cannot leverage related-task information.
  • Existing merging methods combine task vectors but can ignore parameter interference, reducing magnitudes and eliminating distinctions among values.
  • TIES-Merging trims redundant parameters, resolves sign conflicts, and averages only values aligned with the elected sign.
  • 2.3% and 1.7% absolute average gains over the strongest baseline occur for in-domain NLP and vision evaluations, respectively.
  • TIES-Merging outperforms prior merging methods across modalities, model sizes, task domains, fine-tuning settings, and validation-set conditions.

2 Related Work

Prior work studies weight interpolation and model merging, including methods for combining fine-tuned models and improving multitask or out-of-domain performance. The paper also examines how retaining only high-magnitude task-vector values affects performance.

  • Weight interpolation can preserve accuracy between some training runs because their parameter values may be mode-connected.
  • Fine-tuned models sharing initialization can often be merged without accounting for permutation symmetry, supporting multitask and generalization use cases.
  • Keeping only the top-20% largest-magnitude task-vector values does not degrade average performance across eleven tasks.

3 Background and Motivation

The motivation for TIES-Merging is that fine-tuning introduces redundant parameter changes and sign conflicts that interfere during merging. Trimming preserves influential changes, but sign conflicts remain and become more likely as more models are merged.

  • Task vectors represent task-specific directions and movement magnitudes relative to the shared initialization.
  • Many fine-tuning changes are redundant: retaining only the top-20% largest-magnitude values delivers comparable performance to retaining all parameters.
  • TIES-Merging constructs task vectors, trims redundant parameters, elects final signs, performs a disjoint merge, and adds the scaled result to the initialization.
  • Sign conflicts persist after trimming and increase as the number of merged models grows, including when merging only two models.

4 TIES-MERGING: TRIM, ELECT SIGN & MERGE

TIES-Merging addresses interference through three sequential operations: trim low-magnitude task-vector entries, elect the dominant sign for each parameter, and merge only sign-aligned values. The resulting task vector is scaled and added to the pretrained initialization.

  • TIES-Merging is designed to address redundant-parameter and sign-conflict interference before model merging.
  • Trim: Trim keeps each task vector’s top-k% magnitude values and resets the remaining entries to zero.
  • Elect: Elect assigns each parameter the sign with the greatest total movement across the relevant trimmed task vectors.
  • Disjoint Merge: Disjoint Merge averages only values whose signs agree with the elected sign, ignoring zero values.
  • The final merged model adds a scaled merged task vector to the initial parameters using λ as a scaling hyperparameter.

5 Experimental Setup

The experiments compare TIES-MERGING with several model-merging baselines across fine-tuning settings and modalities, including conditions with and without a validation set. The setup also evaluates a fixed-hyperparameter recipe for validation-free merging.

  • Baseline Methods: The study compares TIES-MERGING with Simple Averaging, Fisher Merging, RegMean, and Task Arithmetic.Simple Averaging computes element-wise means, Fisher Merging reweights parameters using approximate Fisher information, RegMean matches activations, and Task Arithmetic adds scaled task vectors.
  • Experimental Scope: Table 1 compares merging methods across NLP and Vision under multiple fine-tuning settings with and without a validation set.
  • Experimental Scope: The evaluation includes individual fine-tuned models and a multitask model trained on the concatenation of all task datasets.
  • Validation-Free Setting: Prior methods use validation data for Fisher estimation or hyperparameter tuning, while RegMean requires storing and transmitting inner-product matrices that scale with model size.
  • Validation-Free Setting: The fixed recipe retains the top-20% of task-vector parameters, resets the rest to zero, and sets λ = 1.It is applied to unseen full-finetuning settings for ViT and T5 models.

6 Main Results

Across language, vision, parameter-efficient, out-of-domain, same-task, and initialization settings, TIES-MERGING consistently improves model merging performance. Its advantage persists as more tasks are merged, where competing methods degrade more rapidly.

  • Main results: TIES-MERGING improves performance across diverse language, vision, fine-tuning, and domain-shift settings.The evaluations cover T5 and ViT models, fully fine-tuned and PEFT models, in-domain and out-of-domain tasks, and settings with or without validation.
  • Parameter-efficient models: 2.5% average enhancement across 11 tasks compared to the top baseline is achieved when merging (IA)3 models with a validation set.TIES-MERGING exceeds all other merging methods in this setting.
  • Vision models: 1.8% and 1.5% average improvements over 8 tasks are obtained for ViT-B/32 and ViT-L/14, respectively, with validation.Without validation, the improvements are 6.6% and 2.7% over other methods for ViT-B/32 and ViT-L/14.
  • NLP models: 0.7% and 3.6% improvements over 7 tasks are obtained for T5-base and T5-large, respectively, compared to the state-of-the-art.For T5-large, TIES-MERGING without validation also outperforms all baselines with validation by 1.1%.
  • Out-of-domain generalization: 1.0% and 4.4% improvements over the strongest baseline are achieved for T5-base and T5-large on six held-out tasks, demonstrating better out-of-domain generalization.The models are merged on seven in-domain datasets and evaluated on six held-out datasets.
  • Scaling with task count: As the number of merged tasks increases, all methods lose performance, but TIES-MERGING degrades more slowly than Task Arithmetic.When merging two tasks, TIES-MERGING and Task Arithmetic achieve average normalized accuracy close to one, while Simple Averaging drops by 10%.

7 Additional Results and Analysis

The analyses isolate the roles of parameter magnitude, sign direction, and each TIES-Merging component in merged-model performance. They show that influential parameter directions matter, while trimming, electing signs, disjoint averaging, and scaling each contribute to the method’s results.

  • Removing Redundant Parameters: The analysis separates redundant parameters from parameters influential to one or multiple models to assess how trimming changes merged parameter values.The experiment uses a 20% trimming threshold and compares direct merging with trimming followed by disjoint merging.
  • Resolving Sign Interference: Sign agreement is measured from 0.5 for an equal positive-negative split to 1 when all parameter signs agree.The analysis compares direct merging with merging after sign disagreement is resolved by election.
  • Importance of Parameter Directions: Flipping the directions of the top-20/30% parameters monotonically decreases performance as the flip probability increases.Flipping directions among the bottom-80/70% parameters has a different effect.
  • Ablation of TIES-Merging Components: TIES-Merging ablations show that all components are crucial for optimal performance.Removing scaling decreases performance by 2.5% in T5-base and 5.2% in (IA)3, while removing the disjoint mean decreases it by 1.9% and 3.2%, respectively.
  • Oracle Sign Vector: An oracle multitask sign vector is evaluated to measure the performance available when the elected signs come from a multitask model.The experiment trains a multitask (IA)3 model on eleven tasks and uses its sign vector during TIES-Merging.

8 Conclusion

The paper concludes that TIES-Merging addresses interference in model merging by trimming low-magnitude changes and resolving sign disagreements. Experiments find improved merged multitask-model performance across varied settings and emphasize the importance of parameter signs.

  • Conclusion: TIES-Merging trims low-magnitude fine-tuning changes and resolves sign disagreements across the models being merged.The method combines these operations to address interference in model merging.
  • Conclusion: TIES-Merging improves merged multitask-model performance across various settings and domains despite using simple fixed hyperparameters.The study also analyzes different interference types and the importance of signs in merging.

A Limitations and Future Works

The paper identifies scope and methodological limitations of model merging, including dependence on shared initialization and architecture, weaker performance than simultaneous multitask training, and uncertainty about checkpoint selection and multitask sign estimation.

  • Limitations: Model merging relies on a common initialization and model architecture.This is listed as a general limitation shared with existing merging methods.
  • Limitations: Merging individual-task models into a multitask model still lags behind simultaneous multitask training.The paper frames estimating multitask signs as a potential way to bridge this gap.
  • Future Works: It remains unclear how to select checkpoints for merging to create multitask models useful for specific domains.The limitation concerns checkpoint choice for domain-specific multitask models.
  • Future Works: Using signs from a multitask model performs better than the signs selected by the proposed merging procedure.The paper proposes estimating multitask signs through limited validation data and few-shot multitask training.
  • Hyper-Parameters: TIES-Merging is less sensitive to λ changes than Task Arithmetic in the reported GLUE experiments.TIES-Merging has an accuracy range of 68-75% across considered λ values, compared with 55-75 for Task Arithmetic.
  • Hyper-Parameters: As k increases, performance drops and then saturates, although the curve may vary with the task-vector parameter distribution.The sign-conflict analysis also reports that keeping more parameters can raise conflict to almost 80%.

B.4 Sign Conflicts Exists Between Different Checkpoints for the Same Task

Sign conflicts occur even among checkpoints trained on the same task, not only when merging different tasks. Their frequency increases with the number of checkpoints and can reach a similar degree to cross-task interference.

  • Same-Task Checkpoints: Sign conflicts exist among checkpoints trained on the same task for RTE, MRPC, and WNLI.The analysis compares ten Huggingface checkpoints for each dataset with checkpoints from different tasks.
  • Same-Task Checkpoints: As the number of same-task checkpoints increases, sign conflict increases.The comparison reports a similar degree of interference for same-task and different-task checkpoint merging.
  • Parameter Magnitudes: The parameter-magnitude analysis compares different merging methods with TIES-Merging and reports standard deviations to represent diversity across parameter values.A high standard deviation indicates diversity in magnitude values across parameters.
  • Evaluation: The appendix includes test-set performance for merging IA3 models across eleven tasks.The cited table provides the evaluation setting rather than a specific numerical outcome.

B.5 Detailed Results for Types of Interference and Their Effect on Merging

The analysis examines how trimming and electing affect merged parameter magnitudes, alongside results across model families, task settings, and evaluation configurations.

  • Trimming and electing usually produce higher parameter magnitudes and standard deviations than simple averaging.Simple averaging decreases the magnitude of non-redundant parameters and influential parameters in the merged model.
  • The evaluation covers merged models across T5-base, T5-large, ViT-B/32, and ViT-L/14 configurations.The supplied results include in-domain and out-of-domain evaluation tables for these model families.
  • The experiments also include different task counts, parameter-efficient settings, and validation-set conditions for selecting merging hyperparameters.Figure 13 studies increasing numbers of merged tasks, while the validation-free recipe uses preliminary k = 20 and λ close to 1.

C.5 Merging Different Number of Tasks

The study evaluates merging different numbers of tasks by sampling subsets and averaging their merged-model performance. It also reports training and evaluation procedures for the T5 experiments.

  • For each number of tasks, the experiment samples at most 10 distinct subsets of the considered tasks.The x-axis represents the number of merged tasks, and the solid line averages performance across sampled runs.
  • The reported curves average merged-model performance across random task subsets, with optimal λ values selected separately for TIES-MERGING and Task Arithmetic.For averaging, λ is fixed to 1 multiplied by the number of tasks.
  • The T5 models were trained for up to 75,000 steps with batch size 1024, learning rate 0.0001, early stopping patience 5, and maximum sequence length 128.Training used bfloat16 to reduce GPU memory expenditure.
  • Evaluation uses rank classification, counting a prediction as correct when the highest-ranked label string matches the correct answer.This procedure supports both classification and multiple-choice tasks.
Loading 2306.01708v2…