PaperlayerView exact revision on arXiv

Source-linked AI summary

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D. Bagdanov, Joost van de Weijer

arXiv:2502.04959v3cs.LG

TL;DR

Model merging still leaves a performance gap between combined and single-task models. The paper measures how task-specific and merged subspaces align, then proposes isotropic merging through flattened singular spectra and task-specific subspaces. Iso-C and Iso-CTS achieve strong results, including state-of-the-art performance across task sets and model scales, while the common subspace remains dependent on Task Arithmetic.

  • Problem

    A significant performance gap remains between merged multi-task models and single-task models, motivating analysis of what makes task matrices effective for merging.

  • Method

    The paper uses subspace alignment analysis and proposes Iso-C, which equalizes singular values, plus Iso-CTS, which adds task-specific directions to a common subspace.

  • Results

    Iso-CTS achieves state-of-the-art results across multiple model scales and task sets, while subspace alignment strongly correlates with merged-model performance.

  • Takeaways & Limitations

    Subspace alignment is a critical factor in effective model merging, and isotropic methods improve merging without requiring additional training.

  • Takeaways & Limitations

    The common subspace is determined by Task Arithmetic, which can be suboptimal; the methods could also exploit task-matrix low-rank structure further.

Abstract

from arXiv · show

Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training. Code is available at https://github.com/danielm1405/iso-merging .

1. Introduction

Model merging seeks to combine task-specific experts, but a substantial gap remains between merged and single-task performance. This paper links effective merging to subspace alignment and proposes isotropic methods that improve alignment by balancing common and task-specific directions.

  • A significant performance gap remains between combined and single-task models despite methods addressing sign conflicts, gradient alignment, and magnitude selection.
  • The Subspace Alignment Ratio measures similarity between subspaces spanned by the top singular vectors of task matrices.
  • Iso-C equalizes singular values in the merged matrix, improving alignment and outperforming most baselines without changing fine-tuning.
  • Iso-CTS adds task-specific directions to the common subspace, retaining unique task features while preserving shared knowledge.
  • Alignment between principal subspaces of task-specific and merged matrices positively correlates with merged-model performance.
  • The methods achieve state-of-the-art results on vision and language merging benchmarks for fully and LoRA fine-tuned models.

2. Related Work

Prior work combines task-specific experts through static or dynamic model-merging procedures and increasingly uses singular-value structure. This paper differs by making singular values uniform in a common subspace and combining shared with task-specific subspaces.

  • Task Arithmetic aggregates task vectors formed from differences between fine-tuned and pre-trained weights to create a multi-task model.
  • Dynamic merging methods alter inference using test-time composition, per-task masks, or rescalers, whereas this paper considers static merging exclusively.
  • SVD-based fine-tuning and compression methods motivate analyzing singular components of model weight updates.
  • KnOTS and Task Singular Vectors use SVD structure for merging, focusing respectively on low-rank adaptation concatenation and orthogonalization to reduce interference.
  • This paper instead makes singular values uniform in a common subspace and combines shared and task-specific subspaces for improved performance.

3. Background and Motivation

The paper frames model merging as combining task-specific updates into one model, then argues that subspace alignment—not vector cosine similarity—better explains performance improvement. The proposed Subspace Alignment Ratio measures this relationship through dominant singular directions.

  • Model Merging: Model merging seeks one model that performs all tasks represented by individually fine-tuned models starting from a shared pre-trained network.Task Arithmetic sums task matrices into the pre-trained weights without additional training or original training data.
  • Cosine Similarity and Performance: Task vectors are typically close to orthogonal, but cosine similarity with the merged task vector has low correlation with normalized accuracy improvement.Figure 2 reports similar cosine similarities for models with substantially different NAI values.
  • Subspace Alignment: The authors hypothesize that performance depends on overlap between dominant singular-vector subspaces of individual task matrices and the merged matrix.The merged subspace is proposed to approximate the union of task-specific subspaces.
  • Subspace Alignment: SAR quantifies overlap between task-matrix subspaces using the top singular vectors of a merged matrix, while SARavg averages alignment across layers.The number of dominant merged components is selected by minimizing approximation error.
  • Subspace Alignment: 0.94 is the Pearson correlation between SARavg and NAI for Task Arithmetic, with task-to-merged alignment ranging from 0.75 to 0.87.Highly mutually aligned task groups tend to align better with the merged model, whereas DTD, Cars, and SUN397 are less aligned and perform worse.

4. Isotropic Merging in Common and Task-specific Subspaces

The paper introduces Iso-C, which equalizes the singular spectrum of the common Task Arithmetic subspace, and Iso-CTS, which additionally preserves task-specific directions. These designs target more balanced alignment and improve performance, especially as the number and diversity of tasks grow.

  • Iso-C: Iso-C equalizes the skewed singular-value spectrum of the Task Arithmetic matrix to improve average subspace alignment and balance task representation.The skewed spectrum concentrates on early singular values and favors highly aligned task groups.
  • Iso-C: Iso-C reconstructs each layer using common singular directions scaled to their average singular value, producing an isotropic transformation.This replaces direction-specific singular values with a fixed scaling factor.
  • Iso-CTS: Iso-CTS combines common and task-specific singular directions across tasks, orthogonalizes them, applies isotropic scaling, and reconstructs the merged matrix.Algorithm 2 specifies this pipeline from task-matrix summation through SVD, projection, combination, whitening, and reconstruction.
  • Limitation and Extension: Iso-C’s common subspace can underrepresent weaker task directions orthogonal to shared directions, especially when tasks become more numerous and diverse.Iso-CTS is introduced to address this limitation by expanding the directions represented in the merged singular basis.
  • Iso-CTS: Iso-CTS extends Iso-C by incorporating task-specific directions orthogonal to the common subspace into the final singular basis.It retains top common directions, projects task matrices orthogonally, selects task-specific components, and then reconstructs an isotropic matrix.

5. Experimental Results

Across vision and language benchmarks, Iso-C and Iso-CTS improve model merging by increasing subspace alignment, while task-specific directions become especially valuable as the number of merged tasks grows.

  • Main results: Iso-CTS achieves state-of-the-art results across the evaluated multi-task model-merging settings.The evaluation covers sets of 8, 14, and 20 datasets.
  • Main results: Iso-CTS improves over Iso-C by up to 2.8% in absolute accuracy when merging 14 and 20 models.The two methods achieve very similar results in the 8-task scenario.
  • LoRA-adapted vision models: Iso-C and Iso-CTS significantly outperform competing methods when merging LoRA-fine-tuned vision models.The general-purpose Iso methods outperform KnOTS, which is specifically designed for LoRA merging.
  • NLP results: Both isotropic methods significantly outperform competing approaches on the evaluated NLP task sets.Iso-CTS performs similarly to Iso-C, suggesting the common space captures the directions needed for these 7- and 8-task settings.
  • Analysis and ablations: Interpolating toward Iso-C flattens the merged spectrum, increases preserved components and SARavg, and improves performance across tasks.The observed trend follows the reported strong correlation between alignment and merged-model performance.
  • Analysis and ablations: Task-specific directions raise SARavg and accuracy relative to Iso-C, while the least significant Task Arithmetic directions contribute negligibly.Iso-CTS remains better than Iso-C across tasks in this analysis.
  • Analysis and ablations: Iso-CTS improves over Iso-C for every tested common-subspace fraction, with values from 0.5 to 0.9 within 0.5% of optimal.The default fraction is 0.8, and these results are reported for the 20-task scenario.

6. Conclusion

The paper concludes that isotropic merging improves multi-task performance by aligning task-specific and merged subspaces. Iso-C balances shared representations, while Iso-CTS additionally preserves task-specific directions.

  • Conclusion: Iso-C equalizes singular values through SVD to create a more balanced representation across tasks.Iso-CTS further incorporates task-specific directions while preserving shared knowledge.
  • Conclusion: Iso-CTS achieves state-of-the-art results across multiple model scales and task sets.The conclusion identifies subspace alignment as a critical factor in effective model merging.
  • Limitations: The common subspace is determined by Task Arithmetic, which can be suboptimal.The authors note that better methods for determining this subspace could be developed.
  • Limitations: The methods could be adapted to exploit low-rank task-matrix structure for improved performance and efficiency.This limitation is noted despite state-of-the-art results in LoRA merging.

Impact Statement

The paper frames model merging around how singular subspaces represent task updates and how spectral structure affects alignment. Its analysis connects spectrum flattening with larger effective subspaces, higher SAR, and lower interference.

  • Spectral structure: The effective rank kM is controlled by singular-spectrum skewness: flatter spectra require more preserved components than skewed spectra.Task Arithmetic therefore has smaller effective rank than Iso-C under the reported approximation criterion.
  • Iso-C mechanism: Iso-C expands the merged model’s effective subspace from kTA to kIso-C by flattening Task Arithmetic’s spectrum.The singular vectors remain unchanged while the singular values are modified.
  • Subspace alignment: Adding lower-significance directions increases alignment because these directions remain relevant to task matrices rather than being orthogonal to them.The analysis reports that enriching the basis consistently increases the alignment ratio.
  • Inter-task interference: SAR measures how well a task matrix’s principal directions align with those of the merged model.The paper defines interference as degradation in a task’s internal representation after merging.
  • Inter-task interference: Higher SAR limits the irreducible orthogonal component of interference, whereas Task Arithmetic retains interference that cannot be removed by scaling alone.The paper reports experimentally lower interference for Iso-C than for Task Arithmetic.

A.4. Limitations of Iso-C that motivate Iso-CTS

Iso-C can underrepresent task directions when their dominant singular components have lower intensity or are orthogonal to shared directions. This limitation worsens as task count and diversity increase, motivating Iso-CTS’s inclusion of overlooked task-specific directions.

  • A.4. Limitations of Iso-C that motivate Iso-CTS: Lower-intensity task matrices contribute less to the variance maximized by Task Arithmetic, so their directions may be underrepresented in Iso-C.Their individual variance contributions and cross terms are comparatively small.
  • A.4. Limitations of Iso-C that motivate Iso-CTS: Directions orthogonal to the principal common directions contribute minimally or not at all to Iso-C’s dominant singular subspace.The same issue can affect subsequent singular vectors defining the common subspace.
  • A.4. Limitations of Iso-C that motivate Iso-CTS: As the number and diversity of tasks increase, distinct smaller-magnitude directions become increasingly likely to be underrepresented or absent.This is the central limitation of Iso-C identified in the paper.
  • A.4. Limitations of Iso-C that motivate Iso-CTS: Iso-CTS addresses the limitation by incorporating orthogonal directions overlooked by the Task Arithmetic spectrum.The strategy is reported to yield the greatest improvements for many diverse tasks.

B. Computational complexity analysis

The computational analysis compares SVD costs for Iso-C, Iso-CTS, and TSV-M under squared n × n layer matrices. Iso-CTS and TSV-M share asymptotic complexity, while Iso-C is most efficient with respect to task count.

  • B. Computational complexity analysis: A single SVD on an n × n task matrix has complexity O(n^3), and the analysis counts SVDs as the dominant cost.The setup assumes T tasks and L network layers.
  • B. Computational complexity analysis: Iso-CTS has total complexity O((T + 3)Ln^3) = O(TLn^3).Its terms include one SVD of the task-arithmetic matrix, one per task, and two additional per layer.
  • B. Computational complexity analysis: TSV-M performs T SVDs per layer plus two additional SVDs per layer, giving the same asymptotic scaling as Iso-CTS.The comparison is based on the number of SVDs performed by each algorithm.
  • B. Computational complexity analysis: Iso-CTS incurs slightly more overhead than TSV-M because it applies an additional SVD to ΔTA.Both methods can reduce cost by using truncated SVDs because only a few components are retained.
  • B. Computational complexity analysis: Iso-C is the most computationally efficient algorithm because its complexity is constant with respect to the number of tasks T.The passage contrasts this task-count behavior with the scaling of Iso-CTS and TSV-M.

C.1. Datasets

The experiments use vision benchmarks with 8, 14, and 20 datasets, alongside comparisons, interference analyses, scaling-factor studies, and ablations. Iso-CTS achieves state-of-the-art performance in the reported additional-baseline evaluation and improves over Iso-C by up to 2.8% on 20 tasks.

  • C.1. Datasets: The 8-dataset benchmark contains Cars, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN.These datasets define the smallest listed benchmark.
  • C.1. Datasets: The 14-dataset benchmark adds CIFAR100, STL10, Flowers102, OxfordIIITPet, PCAM, and FER2013 to the preceding eight datasets.It therefore extends the 8-dataset benchmark with six datasets.
  • C.1. Datasets: Iso-CTS achieves state-of-the-art performance in additional-baseline comparisons on ViT-B/32 and ViT-L/14 across 8 tasks.The comparison includes Fisher Merging, RegMean, PCB, MaTS, and CART, with absolute accuracy reported.
  • C.1. Datasets: Iso-C produces lower merging interference than Task Arithmetic for 8 tasks on ViT-B/16, measured by L1 distance between task-specific and merged final embeddings.The figure caption identifies this comparison as Figure 7.
  • C.1. Datasets: Iso-C and Iso-CTS are more robust to scaling-factor α selection than Task Arithmetic, whose validation accuracy is more sensitive to α.Optimal α values are reported on a held-out validation set.
  • C.1. Datasets: Iso-CTS improves over Iso-C by up to 2.8% on 20 tasks, while isotropic scaling matters mainly when a common subspace exists.With only task-specific subspaces, isotropic scaling does not make a significant difference.

D.5. Applying Iso to individual task matrices

The paper tests whether flattening singular-value spectra benefits individual task matrices, comparing original task-specific models with modified versions. The experiment treats single-model spectrum flattening as Iso-C applied to one model.

  • D.5. Applying Iso to individual task matrices: Figure 11 compares original task-specific models with versions whose task-matrix singular-value spectra have been flattened.Flattening the spectrum is described as equivalent to applying Iso-C to a single model.
  • D.5. Applying Iso to individual task matrices: The additional visualizations include task-matrix spectra and Subspace Alignment Ratio analyses, including results for the larger ViT-L/14 model.These visualizations complement the individual-task experiment.

E.1. Visualization of task matrix spectra

The appendix examines task-matrix singular-value spectra and layerwise subspace alignment in ViT-B/16, while testing how merging behavior changes with model scale and α selection. Iso-C consistently improves alignment, and higher average alignment is associated with better normalized accuracy improvement.

  • Visualization of task matrix spectra: Singular-value spectra are visualized across multiple ViT-B/16 layers for task-specific models, TA, Iso-C, and Iso-CTS.The output projection matrix W_O from layer ℓ = 4 serves as an illustrative example in earlier visualizations.
  • Layerwise subspace alignment: Iso-C has higher subspace alignment than the alternatives across all ViT-B/16 transformer layers.Alignment is computed between fine-tuned and merged weight matrices and the layerwise average of all matrices.
  • Scaling to ViT-L/14: Normalized Accuracy Improvement strongly correlates with average Subspace Alignment Ratio on ViT-L/14, where Iso-C’s increased alignment accompanies better performance.This replicates the ViT-B/16 experiment at the larger ViT-L/14 scale.
  • Sensitivity to α: TA is sensitive to α selection, whereas Iso-C and Iso-CTS are more robust to α selection and resemble task-specific models.α is selected using the best average validation performance across tasks.
  • Sensitivity to α: Even at optimal α values, modified models can retain an accuracy gap relative to the original models when task matrices are scaled.The comparison is shown for a single task using Eq. (3).
Loading 2502.04959v3…