PaperlayerView exact revision on arXiv

Source-linked AI summary

Task Singular Vectors: Reducing Task Interference in Model Merging

Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà

arXiv:2412.00081v3cs.LGstat.ML

TL;DR

Task Arithmetic represents whole networks as flat parameter vectors, overlooking layer structure and relying on coarse-grained measures of task interactions. This paper analyzes per-layer task matrices through SVD and uses their singular vectors for compression and interference-aware merging; TSV-Merge significantly outperforms existing methods across multiple computer vision datasets.

  • Problem

    Task Arithmetic represents whole networks as high-dimensional vectors, overlooks structural information, and uses coarse-grained measures such as cosine similarity to assess task interactions.

    Source: §1 · p. 1

  • Method

    The paper analyzes per-layer task-matrix SVDs to identify low-rank Task Singular Vectors, then uses selected vectors for compression and whitening-based interference reduction in merging.

    Source: §1 · pp. 1–2

  • Results

    Up to 97% normalized accuracy, TSV-Merge significantly outperforms existing model-merging methods across multiple computer vision datasets.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The approach is presented as a storage-efficient alternative to MoEs, ensembles, and multi-task learning.

    Source: §7 · p. 8

    The method currently uses a uniform rank across tasks, while selecting an optimal rank for each task individually remains a possible future direction.

    Source: §7 · p. 8

Abstract

from arXiv · show

Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.

1. Introduction

The paper analyzes task changes as per-layer matrices and uses their low-rank singular-vector structure to motivate compression and interference-aware model merging. TSV-Merge combines compression with interference reduction and outperforms existing methods across several benchmarks.

Introduction

  • Task Arithmetic (TA) [22] and subsequent methods flatten networks into high-dimensional vectors, overlooking structure and relying on coarse measures such as cosine similarity to assess task interactions.The introduction contrasts this with preserving layer-level matrix structure.Source: §1 · p. 1
  • Per-layer task matrices are low-rank, so only a small portion of their singular vectors can represent each layer’s function with high fidelity.The paper calls these singular vectors Task Singular Vectors (TSV) and uses their alignment or divergence across tasks to assess layer-level interference.Source: §1 · pp. 1–2
  • TSV-Compress reduces task vectors to 10% of their original size while retaining 99% of their original accuracy.It does so by retaining only a fraction of each task’s most relevant singular vectors.Source: §1 · pp. 1–2
  • ∼15% accuracy points is TSV-Merge’s average advantage over existing methods across several benchmarks, establishing a new state of the art.Figure 1 shows mean accuracy for a ViT-L-14 merged over 8, 14, and 20 tasks.Source: Figure 1 · p. 1, §1 · p. 2
  • TSV-Merge discards irrelevant singular vectors, then applies whitening over their similarity matrix to reduce interference across tasks.The paper reports that interference reduction is complementary to compression.Source: §1 · p. 2

2. Related Work

Prior model merging combines existing models through alignment and averaging or uses task vectors and other multi-task strategies to merge fine-tuned models. The paper distinguishes TSV methods by using singular-vector geometry to reduce task interference and isolate task-specific components for compression.

Related Work

  • Task Arithmetic [22] merges task vectors by averaging weight differences from a pre-trained base model, while negating a vector enables forgetting that task.Other approaches use parameter selection or weighting: TIES [51] keeps significant changes and merges disjointly by sign, DARE [53] resets redundant parameters, and Fisher Merging [32] and RegMean [24] use weighted averaging.Source: §2 · p. 2
  • TwinMerging [31] and SMILE [44] apply SVD to layer task matrices but neglect singular-vector interference and require an inference-time router.The router selectively activates a subset of components during inference.Source: §2 · p. 2
  • TALL-Mask activates important task-specific weights with binary masks and extra storage, while Consensus Merging removes weights important to fewer than two tasks [46].Model Breadcrumbs instead merges significant weights after discarding outliers and minor or large perturbations.Source: §2 · p. 2
  • TSV methods decorrelate singular vectors to reduce cross-task interference, rather than averaging or selectively merging parameters.They also isolate each task’s unique components, store only essential parts, and target compression with minimal interference beyond traditional SVD-based compression approaches.Source: §2 · pp. 2–3

SVD for Model Compression

  • Prior SVD compression factorizes trained layers to minimize the difference between original and low-rank weight matrices, using SVD [12] [16] [23] or iterative optimization [23] [26].The paper uses SVD both for low-rank approximation and compression and to analyze and mitigate interference.Source: §2 · p. 3

3. Task Singular Vectors

The section analyzes task-specific layer matrices through their singular vectors, finding that they are low-rank and that cross-task singular-vector overlap can indicate interference. It introduces Singular Task Interference (STI) to measure this interaction.

3.1. Background

  • Task Arithmetic is formulated at the layer level using task-specific weight differences; matrix-structured layers use per-layer task matrices, while other layers default to standard Task Arithmetic.Source: §3.1 · p. 3

Decomposing layer task matrices

  • The SVD of layer task matrices yields Task Singular Vectors (TSVs), whose cross-task overlaps make concatenated left and right vectors non-orthogonal and can bias merging toward tasks with larger singular values.The paper represents the aggregate matrix using concatenated TSVs and block-diagonal singular values.Source: Decomposing layer task matrices · p. 3

3.2. Low-rank nature of layer task matrices

  • Retaining 3% of singular components per task reduces mean accuracy by only 1.5%; the proposed compression preserves 99% of accuracy at 10% of the original task-vector size.The low-rank result is averaged over 20 tasks; the compression procedure is introduced in Section 4.1.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Singular Task interference

  • STI measures interference from interactions among tasks’ TSVs; overlapping singular vectors may indicate shared features that introduce interference and degrade individual-task performance.STI uses the concatenated singular decompositions and treats identity Gram matrices as the ideal of minimal interference.Source: §3.3 · p. 4

4. Approach

The approach uses task-specific singular components for low-rank compression when task identity is known, and decorrelates task singular vectors before reconstructing a joint model when it is not.

4.1. TSV for compression

  • For a known or inferred task h, compression zeros other-task components and retains the task matrix’s top k singular components; larger k improves approximation but reduces compression.Source: §4.1 · p. 4

4.2. TSV for model merging

  • When task identity is unknown, TSV merging combines Task Singular Vectors to create one model that performs well jointly across tasks.Source: §4.2 · p. 4
  • To reduce task interference, the method whitens task-wise columns of U and V, using an SVD-based orthogonal Procrustes solution for improved numerical stability.Source: §4.2 · p. 4
  • After decorrelating the rank-reduced TSV matrices U and V, the method reconstructs the merged layer as in Eq. (5).Source: §4.2 · p. 5

5. Results

Results evaluate merging on 8-, 14-, and 20-task suites across three CLIP ViT encoders; TSV-M leads reported benchmark comparisons, while TSV-C preserves accuracy with fixed storage.

5.1. Model merging results

  • +15.45% and +10.71% average absolute accuracy: TSV-M outperforms Task Arithmetic [22] and Consensus TA [46], respectively, on ViT-B-32; it reaches 96.98% normalized accuracy on ViT-L-14.TSV-M achieves state-of-the-art results across benchmarks, encoder sizes, and task counts. Evaluations use CLIP ViT-B-32, ViT-B-16, and ViT-L-14 on 8-, 14-, and 20-task benchmarks.Source: §5.1 · pp. 5–6, Table 2 · p. 6

5.2. Compression results

  • More than 99% of original accuracy is retained by TSV-C across benchmarks and models, with results differing from TALL-masks [46] by less than 1% accuracy on average.TSV-C slightly underperforms TALL-masks for small ViTs, while TALL-masks performs better at larger scale; TSV-C leverages the most significant task singular vectors.Source: §5.2 · p. 6
  • Approximately twice the original model size is required to store TSV-C's top 1 T singular vectors per task, regardless of task count, unlike task-count-dependent TALL-masks [46].TALL-masks storage varies with mask compressibility and increases with the number of tasks, potentially reducing compression benefits when masks are less compressible.Source: §5.2 · p. 6

6. Analysis

The analysis finds that combining low-rank approximation with interference reduction improves merging performance, while also examining orthogonalization error, layer-wise interference, and the scaling coefficient. Across evaluated task sets, lower interference is associated with higher accuracy, and the method performs best without additional scaling.

6.1. Ablation study

  • TSV-C retains over 99% of the original finetuned models’ accuracy while reducing storage requirements across the compression benchmarks.The table caption describes average absolute accuracy results across models and varying numbers of tasks.Source: Table 3 · p. 7
  • Combining low-rank approximation and interference reduction yields +15.8% to +17.5% gains; interference reduction alone yields +3.1% to +6.1%, while low-rank approximation alone underperforms vanilla Task Arithmetic.Table 4 compares versions using either or both components; the method using both is TSV-Merge.Source: §6.1 · pp. 6–7, Table 4 · p. 7
  • Procrustes orthogonalization incurs greater reconstruction error on full-rank than low-rank matrices, making interference reduction less costly when applied after compression.For ViT-B-32 with 8 tasks, full-rank errors have a wider, higher-average distribution, while low-rank errors are more compact and lower; the analysis attributes this pattern to better consistency across layers.Source: Figure 5 · p. 7, §6.1 · p. 7
  • For T > 4 and k ≤ n(T−2√T)/T, Theorem 6.1 states that full-rank Procrustes error is at least the error for truncated matrices.The paper interprets this as a general property explaining why orthogonalizing concatenated truncated matrices introduces less approximation error than orthogonalizing the original full-rank matrices.Source: §6.1 · p. 7

6.2. Per-layer task interference

  • Task interference is highest in early transformer layers and decreases significantly in deeper, more specialized layers.In Fig. 6, each plotted layer groups two attention matrices and two MLP matrices within a transformer block.Source: §6.2 · pp. 7–8, Figure 6 · p. 8

6.3. Choice of the interpolation coefficient

  • The method consistently achieves its best results at α = 1.0, avoiding extra scaling-coefficient tuning and validation data.The paper describes this as an empirical finding based on Fig. 7.Source: §6.3 · p. 8, Figure 7 · p. 8

6.4. Effect of task interference reduction

  • Across three task sets of different sizes, reduced interference is closely associated with significant accuracy gains for TSV-Merge over Task Arithmetic.Fig. 8 reports Singular Task Interference and average normalized accuracy for merges of 8, 14, and 20 tasks on ViT-B-32.Source: §6.4 · p. 8, Figure 8 · p. 8

7. Conclusions

The paper uses the low-rank structure of per-layer task matrices and their singular vectors to measure task interference, then combines compression and interference reduction for model merging. It also identifies deeper-layer and rank-allocation observations that inform evaluation and future work.

Conclusions

  • The analysis confirms per-layer task matrices are low-rank and uses their singular vectors to define task interference.The paper analyzes the SVD of per-layer task matrices.Source: §7 · p. 8
  • Up to 97% normalized accuracy, TSV-M combines low-rank approximation with interference reduction as a storage-efficient alternative to MoEs, ensembles, and multi-task learning.The approach is presented as a storage-efficient alternative to those methods.Source: §7 · p. 8
  • TSV-C reduces task vectors to 10% of their original size while retaining 99% of original accuracy, with storage independent of the number of tasks.The method maintains a constant storage requirement regardless of task count.Source: §7 · p. 8
  • Interference reduction contributes most to performance improvements, but optimal results combine it with compression; interference decreases in deeper layers, and the method needs no scaling-coefficient tuning.These observations come from the paper’s extensive evaluation.Source: §7 · p. 8
  • Future work could replace the uniform rank across tasks with individually selected ranks, using approaches from Idelbayev and Carreira-Perpin´an or Gavish and Donoho.The current method allocates 1/ntasks of a layer’s dimensionality to each task vector.Source: §7 · p. 8

A. Illustrative example: merging two tasks with rank-1 approximation · B. Additional details

In the rank-1 example, interactions between two tasks depend on the alignment of their singular vectors: orthogonal vectors act independently, collinear vectors cause maximal interference, and partial alignment produces intermediate interaction. The example illustrates why singular-vector alignment matters when merging models.

A. Illustrative example: merging two tasks with rank-1 approximation

  • The example models each task and layer using only the first singular vector and singular value from that task’s SVD.It assigns u1Li and u2Li to tasks 1 and 2, respectively, with analogous assignments for v and σ.Source: A. Illustrative example: merging two… · p. 12
  • When the tasks’ singular vectors are orthogonal, the zero off-diagonal similarities indicate no interference and independent operation.The passage applies this condition to both u and v singular vectors.Source: A. Illustrative example: merging two… · p. 12
  • When singular vectors align in the same direction, their similarity is 1, producing a double-scaling effect between the tasks.The example states that aligned vectors are equal and have unit inner product.Source: A. Illustrative example: merging two… · p. 12
  • When singular vectors point in opposite directions, their similarity is −1 and the tasks completely cancel one another.The example describes opposite alignment as complete interference.Source: A. Illustrative example: merging two… · p. 12
  • For partially collinear vectors, the degree and constructive or destructive direction of interaction are proportional to the cosine of their angle.The example considers angles between slightly above 0 degrees and below 180 degrees.Source: A. Illustrative example: merging two… · p. 12
  • Singular-vector alignment determines whether two tasks interact independently, interfere maximally, or influence one another partially.The paper characterizes partial interaction as constructive or destructive according to the vectors’ alignment.Source: A. Illustrative example: merging two… · p. 12

B.1. Implementation details and computational resources … C.2. Observations

The appendices specify implementation, hyperparameter, and storage choices for task-vector merging, and establish properties of the similarity matrices used in the method.

B.1. Implementation details and computational resources

  • The appendix reports normalized accuracy as multi-task accuracy divided by individually fine-tuned accuracy for each task, averaged across tasks, alongside absolute accuracy.Benchmarks reuse existing model and classification-head checkpoints without further fine-tuning; matrix layers use SVD, non-matrix layers use standard Task Arithmetic, and averaging uses Knut’s algorithm [47]. Merging and evaluation ran on an NVIDIA 4060Ti GPU with 16GB memory and an Intel i7-6800K CPU with 64GB RAM, with a shared HTCondor cluster used when more RAM was needed.Source: §B.1 · pp. 12–13

B.2. Hyperparameter Settings

  • The scaling factor α is selected from 0.0 to 3.0 in 0.1 steps using average validation performance, but TSV-Merge’s marginal tuning gains do not justify validation evaluation.This removes the need for validation datasets and labels; evaluation uses batches of 32 images, and Figure 7 uses whitened subsets of 8 tasks selected from 20.Source: §B.2 · p. 13

B.3. Storage cost calculation

  • Retaining k′ < 0.1 × min(d, m) is reported as sufficient to preserve most task performance, while k′ < dm/(d+m+1) guarantees fewer stored parameters than the original task vector.The storage expression is L × (d × k′ + k′ + k′ × m) + N × c; choosing k′ = k/T satisfies the storage inequality for T ≥ 3.Source: §B.3 · p. 14

C. Proofs

  • Appendix C states that it proves the claims outlined in the main manuscript.Source: C. Proofs · p. 14

C.1. Characterization of the similarity matrices

  • Under the full-column-rank assumption, the similarity matrix U^T Û is positive definite and therefore invertible.The proof establishes positivity by expressing x^T Û^T Û x as the squared norm of Ûx for every nonzero x.Source: §C.1 · pp. 14–15

C.2. Observations

  • Because Û^T Û is real symmetric positive definite, its inverse is also positive definite, with eigenvalues 1/λ_i.Source: §C.2 · p. 15

C.3. Proof of Theorem 6.1 · D. Additional experimental results

For T > 4, Theorem 6.1 gives a sufficient upper bound on the truncation rank k under which the Procrustes distance for the full concatenated matrix is at least that for its truncated version. Figure 13 identifies additional ViT-B-16 evaluations of STI and normalized accuracy for Task Arithmetic and TSV-Merge.

C.3. Proof of Theorem 6.1

  • For T > 4 and k ≤ n(T − 2√T)/T, Theorem 6.1 states that ||U − X||_F ≥ ||Û − X̂||_F.Here U concatenates T orthogonal n × n matrices, Û retains each matrix’s first k columns, and X and X̂ are their respective Procrustes orthonormalizations.Source: §C.3 · p. 15
  • Because UU^T = T I_n, all singular values of U equal √T, giving ||X − U||_F = √n(√T − 1).This is the full-matrix distance used in the proof’s comparison.Source: §C.3 · p. 16
  • The proof bounds the truncated-matrix distance by √(n + kT − 2√(kT)) and compares it with the full-matrix distance √n(√T − 1).The bound uses the lower limit √(kT) on the sum of Û’s singular values.Source: §C.3 · p. 16

D. Additional experimental results

  • Figure 13 evaluates Singular Task Interference and average normalized accuracy for Task Arithmetic and TSV-Merge across merges of 8, 14, and 20 tasks on ViT-B-16.The supplied caption specifies the evaluation scope but does not provide plotted values or identify a winner.Source: Figure 13 · p. 16

D.1. Per-dataset performance metrics … D.2.7. Test-time adaptation

The extended analyses test TSV-M across model sizes and task counts, examining rank, interference, compression, and test-time adaptation. They also describe numerical differences between whitening and SVD implementations.

D.1. Per-dataset performance metrics

  • ViT-B-16 and ViT-L-14 radar-plot analyses show findings consistent with the ViT-B-32 results reported in the main text.The appendix presents the analogous per-task plots in Figures 9 and 10, respectively.Source: §D.1 · p. 16

D.2.5. Detailed per-layer task interference

  • Singular Task Interference is high in early ViT-B-32 layers and decreases later, with each attention-out layer showing an initial rise followed by a drop.Figure 14 provides a per-transformer-block view across 20 tasks.Source: Figure 14 · p. 17, §D.2.5 · p. 18

D.2.1. Whitening vs. SVD

  • Whitening task singular-vector matrices is mathematically equivalent to Orthogonal Procrustes, but eigendecomposition and SVD can yield slightly different numerical results.The implementation uses ϵ = 1e−12 to prevent division by zero and an absolute value to avoid small negative magnitudes below 1e−6 from numerical errors.Source: §D.2.1 · p. 17

D.2.2. Impact of rank

  • 1.3% mean accuracy decrease at 3% retained TSVs for ViT-B-16 and 1.1% at 2% retained TSVs for ViT-L-14 demonstrate low-rank approximation with few TSVs.These findings are analogous to those reported for ViT-B-32.Source: §D.2.2 · p. 17

D.2.6. Compression analysis

  • TSV outperforms TALL-Mask on ViT-L-14 benchmarks with 14 and 20 tasks, while rank selection enables flexible compression and more aggressive compression.The fixed-budget analysis contrasts TSV’s selectable compression rates with TALL-Mask’s fixed requirement based on model size and task count.Source: Figure 16 · p. 18, §D.2.6 · p. 18

D.2.3. Extended Ablation study

  • The extended ablation compares Task Arithmetic, low-rank approximation alone, interference reduction alone, and TSV-Merge combining both components.The study reports this setup for ViT-B-16 and ViT-L-14, following the experimental setup in Section 6.1.Source: §D.2.3 · pp. 17–18

D.2.4. Effect of task interference

  • Interference removal produces a significant accuracy gain in all reported instances for ViT-B-16 and ViT-L-14.The appendix reports the same finding as Figure 8 for these models.Source: §D.2.4 · p. 18

D.2.7. Test-time adaptation

  • 88.93% accuracy for TSV-M without test-time adaptation exceeds AdaMerging’s 85.43% by approximately 3.5%; adding AdaMerging-style adaptation raises TSV-M to 89.87%.The comparison uses a subset of 7 tasks from the 8-task benchmark.Source: §D.2.7 · p. 18

E. Theoretical motivations and analysis · E.1. Theoretical foundation - Empirical design · E.2. Heuristic interference measure

TSV-C draws on low-rank approximation, while TSV-M combines low-rank filtering with orthogonalization to reduce noise and task interference. The paper defines interference operationally as cross-task interaction that hinders merging and uses performance improvements from minimizing its measure as empirical validation.

E.1. Theoretical foundation - Empirical design

  • TSV-C is grounded in the established framework of low-rank approximation for compression [12].Source: §E.1 · p. 18
  • TSV-M uses low-rank approximation to filter insignificant variations and reduce noise.Source: §E.1 · p. 18
  • TSV-M orthogonalizes task-specific singular vectors to keep them independent and preserve individual task performance.Source: §E.1 · p. 18

E.2. Heuristic interference measure

  • Interference is operationally defined as cross-task interaction that hinders merging, and the proposed measure is empirically validated by consistent gains when minimized.Source: §E.2 · p. 18
  • Independent task-wise finetuning can produce destructive activation overlaps, unlike the coordinated knowledge sharing enabled by joint multi-task learning.Source: §E.2 · p. 18
  • Figure 17 plots ViT-B-32 accuracy across retained singular-component fractions, contrasting low-rank approximations with full-rank fine-tuned models.Source: Figure 17 · p. 19
  • Figure 18 presents confusion-matrix classification accuracy for one merged ViT-L-14 model across 20 tasks, omitting values too small to display.Source: Figure 18 · p. 20
Loading 2412.00081v3…