PaperlayerView exact revision on arXiv

Source-linked AI summary

Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors

Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, Chun Yuan

arXiv:2503.08099v2cs.LG

TL;DR

Model merging must reduce interference without requiring retraining data or test-time computation. WUDI-Merging uses the approximate input-subspace structure of linear-layer task vectors to guide data-free merging. It achieves state-of-the-art data-free performance and outperforms mainstream test-time adaptation methods, while LoRA introduces a reported performance gap linked to task-vector noise.

  • Problem

    Parameter interference degrades merged-model performance, while resolving it without additional data, retraining, or test-time computation remains challenging.

  • Method

    WUDI-Merging minimizes interference using linear-layer task vectors, which approximately form subspaces of their corresponding inputs, without additional data or rescaling coefficients.

  • Results

    WUDI-Merging achieves state-of-the-art data-free model-merging performance and outperforms mainstream test-time adaptation methods across evaluated vision and language tasks.

  • Takeaways & Limitations

    The approach provides a simple data-free model-merging technique requiring no additional data, extra storage, or rescaling coefficients.

  • Takeaways & Limitations

    For LoRA-finetuned models, random LoRA initialization can introduce task-vector noise that degrades input reconstruction and creates a performance gap, although WUDI remains state of the art.

Abstract

from arXiv · show

Model merging seeks to integrate task-specific expert models into a unified architecture while preserving multi-task generalization capabilities, yet parameter interference between constituent models frequently induces performance degradation. Although prior work has explored many merging strategies, resolving interference without additional data for retraining or test-time computation remains challenging. In this paper, we theoretically demonstrate that the task vectors of the linear layer constitute an approximate linear subspace for its corresponding input. Therefore, we can minimize interference under the guidance of task vectors. Based on this insight, we propose \textbf{WUDI-Merging} (\textbf{W}hoever started the interference sho\textbf{U}ld en\textbf{D} \textbf{I}t), a simple yet effective model merging method that eliminates interference without any additional data or rescaling coefficients. Comprehensive empirical evaluations across vision and language benchmarks demonstrate our method's superiority, achieving state-of-the-art performance in data-free model merging scenarios (average 10.9\% improvement versus baseline methods) while even outperforming mainstream test-time adaptation approaches by 3.3\%, and only very few computing resources are required. The code will be publicly available soon.

1. Introduction

Model merging combines expert models without costly multi-task retraining, but interference can degrade task performance. WUDI-Merging uses the approximate input-subspace structure of task vectors to reduce interference without extra data or storage.

  • Model merging integrates multiple expert models into one multi-task model without expensive retraining on multi-task datasets.
  • Interference among expert models creates a performance gap between merged models and their corresponding experts on specific tasks.
  • Task vectors in linear layers approximately form a linear subspace of the corresponding inputs, enabling implicit use of training-data information.
  • WUDI-Merging minimizes interference to task vectors without additional data, extra storage, or rescaling coefficients.
  • WUDI-Merging achieves state-of-the-art performance in data-free model merging and surpasses mainstream test-time adaptation merging methods.

2. Related Work

Model-merging research includes data-free, test-time adaptation, and MoE-like approaches. Data-free methods avoid extra data and storage, while alternatives mitigate interference using test data or task-specific components but introduce practical costs.

  • Model-merging research primarily comprises data-free, test-time adaptation, and MoE-like methods.
  • Data-Free Model Merging: Data-free methods combine expert models without additional retraining data, using approaches such as averaging, Fisher weighting, prediction matching, and task-vector arithmetic.
  • Test-time adaptation Model Merging: Test-time adaptation methods use unlabeled test data to resolve interference through learned coefficients or representation alignment.
  • MoE-like Model Merging: MoE-like methods retain or route task-specific knowledge through masks or routers when merging specialized models.
  • Test-time adaptation and MoE-like methods face practical limitations involving data privacy, extra storage, or reduced parallelism in unified models.

3. Methodology

The method defines task and interference vectors, proves that linear-layer task vectors approximate subspaces of corresponding inputs, and reconstructs inputs from task vectors to optimize merging. It solves layers independently and uses empirical weighting because actual inputs are inaccessible.

  • 3.1. Preliminary of Model Merging: A task vector is the difference between an expert model’s weights and the pretrained model’s weights.
  • 3.1. Preliminary of Model Merging: The merged model parameters equal the pretrained parameters plus the merged task vector.
  • 3.1. Preliminary of Model Merging: The method focuses on linear layers because their task vectors encapsulate most expert-model capabilities.
  • 3.1. Preliminary of Model Merging: For each task, the interference vector is the remaining component of the merged task vector after accounting for that task’s task vector.
  • 3.1. Preliminary of Model Merging: Linear-layer interference is defined using expert and merged layer parameters, their task vectors, and the corresponding input.
  • 3.2. Towards Understanding the Task Vector: Under small learning rates and limited iterations, corresponding linear-layer inputs remain highly consistent across fine-tuning iterations.
  • 3.2. Towards Understanding the Task Vector: Most ViT-B/32 layers show pretrained-to-fine-tuned input direction changes below 0.4, supporting high input consistency.
  • 3.2. Towards Understanding the Task Vector: Each linear-layer neuron is approximated as a weighted sum of corresponding inputs, so task vectors form an approximate input subspace.

4. Experiment

WUDI-Merging is evaluated across vision and language tasks using common model families and data-free baselines, with experiments targeting performance, efficiency, robustness, and ablations. It consistently improves over strong merging methods, while revealing limitations in code-task performance and LoRA settings.

  • Experimental Setup: WUDI-Merging is evaluated on eight vision datasets, GLUE discriminative language tasks, and generative instruction-following, mathematical-reasoning, and code-generation tasks.The experiments use CLIP-derived ViT-B/32, ViT-B/16, and ViT-L/14 models, RoBERTa-Base and RoBERTa-Large, and Llama2.
  • Experimental Setup: WUDI-Merging is compared with data-free methods including Weight Averaging, Fisher Merging, RegMean, Task Arithmetic, Ties-Merging, Consensus Merging, and PCB Merging, plus AdaMerging and Surgery.The method uses two hyperparameters, Adam optimization with learning rate 1e-5, 300 iterations, and applies only to linear layers.
  • Main Results: 8.9% improvement over PCB-Merging and 4.1% over AdaMerging++ are achieved on ViT-B/32, while ViT-L/14 gains are 5.1% and 1.6%, respectively.WUDI-Merging also outperforms existing methods without additional data or extra storage.
  • Main Results: 19.7% and 16.0% improvements over Ties-Merging are achieved on RoBERTa-Base and RoBERTa-Large, while generative language tasks improve 4.3% over Ties-Merging with DARE.Code-task performance remains average, with substantial conflicts between math and code tasks, although the overall average remains ahead of the current state of the art.
  • LoRA-Finetuned Models: 0.99% improvement over Ties-Merging with DARE is observed for Qwen-14B LoRA models, and 2.99% over AdaMerging++ for Flan-T5 LoRA models.The improvement is smaller than in conventional settings, but WUDI-Merging still attains state-of-the-art performance for LoRA-finetuned models.
  • Efficiency and Robustness: Accuracy increases consistently across datasets and models, typically converging within 100 to 200 iterations, while WUDI-Merging maintains higher accuracy as task count increases.Task arithmetic and Ties-Merging show more pronounced performance drops as task numbers increase; larger task-vector magnitudes also show greater resilience to interference.

5. Conclusion

The paper argues that task vectors approximate the input subspace of linear layers, enabling WUDI-Merging to reduce expert-model interference without additional data or rescaling coefficients.

  • Task vectors in linear layers constitute an approximate linear subspace of their corresponding inputs.
  • WUDI-Merging minimizes interference under task-vector guidance in a data-free model-merging setting.
  • WUDI-Merging resolves interference without additional data or rescaling coefficients.
  • Extensive empirical evaluations demonstrate the effectiveness of WUDI-Merging.

A.1.1. PROOF OF LEMMA 1

The proof analyzes how fine-tuning updates affect a linear layer’s inputs, using gradient descent, Lipschitz continuity, and bounded gradients to establish input consistency.

  • The lemma studies inputs to a linear layer after two fine-tuning iterations, under learning-rate and Lipschitz-continuity assumptions.
  • The proof represents the preceding network as a mapping from the input to the linear-layer input.
  • Gradient descent updates the parameters, and the cumulative parameter change is derived from those updates.
  • The proof formulates and bounds the difference between linear-layer inputs using the ℓ2-norm.

A.1.2. PROOF OF PROPOSITION 1

The proposition proof connects fine-tuning gradients with task-vector structure by bounding gradient-related quantities for individual neurons and samples.

  • The proof considers a task-vector neuron and a collection of input samples for a linear layer.
  • It assumes a bound on the gradient of the loss with respect to the relevant parameter product.
  • The quantity Φ_l,k is defined using the sample count, Lipschitz constant, gradient bound, and neuron-specific bound.
  • The task vector is calculated as an accumulation of gradients, after which the proof applies norm operations and the preceding lemma.

A.1.3. PROOF OF THEOREM 1

The theorem proof bounds interference by reconstructing linear-layer inputs from task vectors and relating reconstruction error to expected interference.

  • The theorem defines each task’s linear-layer input, interference vector, and task vector before deriving an interference bound.
  • Each task-vector neuron is interpreted as a weighted sum of corresponding input samples, motivating input reconstruction from task vectors.
  • The proof introduces reconstruction error and formulates the expected interference over the input domain.
  • It reconstructs the input, applies the triangle inequality, and reformulates the resulting expressions.
  • With fixed task vectors and inputs, the reconstruction error and coefficient are fixed, yielding constants for the interference analysis.

A.1.4. THE DETAILED DERIVATION FOR THE CLOSE-FORM SOLUTION.

The paper connects task-vector structure and input consistency to interference analysis, then evaluates linear-layer merging across models, tasks, layers, and optimization steps. Additional experiments report multi-task settings, datasets, and implementation choices.

  • A.1.4. THE DETAILED DERIVATION FOR THE CLOSE-FORM SOLUTION.: The close-form solution derives an objective with regularization, computes its gradient, sets it to zero, rewrites the equation, assumes invertibility, and substitutes the resulting minimizer.The derivation proceeds from the regularized objective through stationarity and back-substitution.
  • A.2. Towards understanding the conflict in Model Merging: Interference arises from interaction between interference vectors and inputs, with larger input-domain overlap potentially exacerbating conflicts across experts.The passage contrasts limited overlap among visual tasks with larger overlap across language tasks.
  • A.3. The impact of linear layer: Applying task vectors only to linear layers preserves most model ability, indicating that the linear layer is central to the model-merging problem.The result is reported from the experiment corresponding to Figure 7.
  • A.4. Additional Experiments: Additional experiments evaluate ViT-B/16 and discriminative language tasks, with tables covering eight-task ViT-B/16 and RoBERTa-based GLUE merging.The tables also include RoBERTa-Large on the eight-task GLUE benchmark and report normalized scores there.
  • A.5. Consistency of the input between pretrained model and fine-tuned model: The change in input direction is below 0.4 for most layers, while task vectors are described as more consistent than pretrained-to-fine-tuned input changes.The passage states that larger models show less consistency, although direction and magnitude changes remain small.
  • A.6. The detailed results of interference in different layers and tasks.: Relative error evaluates layer-wise interference, with Figures 10–12 showing errors across models, layer counts, and datasets and Figure 3 summarizing dataset averages.Figures 10, 11, and 12 correspond to ViT-B/32, ViT-B/16, and ViT-L/14, respectively.
  • A.7. Performance comparison when using different steps for the solution process.: Figure 13 compares performance for ViT-B/32, ViT-B/16, and ViT-L/14 when using different numbers of optimization steps.The figure reports results at different stages of the solution process.
  • A.8. Experimental setup: Experiments span eight vision tasks, GLUE discriminative language tasks, and generative language tasks, using NVIDIA A100 40GB with two stated hyperparameters.The stated hyperparameters are learning rate 1e-5 and 300 iterations, with the method applied only to linear layers.
Loading 2503.08099v2…