PaperlayerView exact revision on arXiv

Source-linked AI summary

Model merging with SVD to tie the Knots

George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman

arXiv:2410.19735v1cs.CV

TL;DR

Merging works well for fully finetuned models but transfers less well to LoRA models, whose task-updates show lower alignment. KnOTS uses SVD to align LoRA updates before merging, and improves prior work by up to 4.3% across vision and language settings, including a new joint-evaluation benchmark.

  • Problem

    Existing merging methods do not always transfer well to LoRA models, whose task-updates show considerably lower CKA alignment than full-finetuning updates.

    Source: §1 · p. 1

  • Method

    KnOTS applies SVD to LoRA task-updates to place them in a shared aligned space where existing merging methods can be applied.

    Source: §1 · pp. 1–2

  • Results

    Across vision and language settings, including the new benchmark, KnOTS improves prior work by up to 4.3%.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Joint evaluation tests generality by classifying across the union of eight vision datasets’ inputs and labels without dataset-source information.

    Source: §1 · p. 2

    The formal setup assumes layer-wise weights with optional biases concatenated to their corresponding weights.

    Source: §3 · p. 3

Abstract

from arXiv · show

Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.

1 INTRODUCTION

KnOTS addresses the weaker alignment of LoRA task-updates by mapping them into a shared space for merging. The paper also introduces joint evaluation to test whether merged models can classify across the union of eight vision datasets.

1 INTRODUCTION

  • Model merging combines task-specific model weights into one model, and prior work has shown strong results with full-rank finetuned models.In some cases, a linear sum of finetuned weights achieves strong performance without further training.Source: §1 · p. 1
  • LoRA task-updates have considerably lower CKA alignment than full-finetuning updates, motivating an alignment approach that does not require data or gradients.The authors hypothesize that aligning LoRA task-updates is important for improving their mergeability.Source: §1 · p. 1
  • KnOTS concatenates layer task-updates, applies SVD to obtain a shared basis, and merges the aligned task matrices before reconstructing an update for the pretrained model.The SVD yields a shared orthonormal basis U, scale values Σ, and aligned task-specific matrices Vᵀ; existing merging methods operate on those matrices.Source: §1 · pp. 1–2
  • KnOTS improves merging performance in the challenging joint-evaluation setting, which tests classification across all eight datasets without dataset-source information.The evaluation uses the union of all inputs and labels; KnOTS further improves merging performance in this setting.Source: §1 · p. 2
  • The new joint-evaluation benchmark classifies images against labels from all eight vision datasets without identifying each image’s source dataset.This contrasts with evaluation limited to labels from the dataset an example comes from.Source: §1 · p. 2

2 RELATED WORK

Related work traces model merging from interpolation and weight averaging to task-vector methods that address interference. Prior merging approaches have transferred less well to LoRA models.

Landscapes between models

  • Mode-connectivity findings show that independently trained models can sometimes be interpolated without increasing test loss, and models finetuned on the same task from the same initialization are often mode-connected.These findings accompany the broader practice of finetuning pretrained models and merging models trained on different tasks from the same initialization.Source: §2 · p. 3

Model merging methods

  • Task arithmetic sums task-vectors formed by subtracting pretrained parameters from finetuned models, while TIES prunes low-magnitude weights and averages weights with the dominant sign.DARE further explores merging by randomly dropping finetuned weights and rescaling the remainder.Source: §2 · p. 3

Merging LoRA models

  • Tang et al. (2024) attribute weak transfer of existing merging methods to LoRA to increased weight-entanglement and propose specialized finetuning; this paper instead presents a gradient-free framework.Other work explores merging improvements through finetuning and access to training data.Source: §2 · p. 3

3 BACKGROUND AND MOTIVATION

The paper studies gradient-free merging of LoRA models fine-tuned on distinct tasks from the same pretrained model. Unlike successful full-rank merging, existing gradient-free methods encounter challenges with LoRA models.

3 BACKGROUND AND MOTIVATION

  • Gradient-free merging combines LoRA task-updates into one model intended to solve all tasks, without requiring data or additional training.The setup uses models with the same architecture, fine-tuned on distinct tasks from a shared pretrained model.Source: §3 · p. 3
  • LoRA constrains each task-update to rank r, with r << min(I, O), limiting it to a subspace of the input and output dimensions.Here, I and O denote the input and output dimensions.Source: §3 · p. 3

3.1 LORA MODELS ARE DIFFICULT TO MERGE

  • Methods that merge full-rank finetuned models perform well across different tasks, but the same methods face significant challenges with LoRA models.The cited work reports LoRA merging challenges even when models are fine-tuned on the same tasks.Source: §3.1 · p. 4

2 ReLU(W (2)

LoRA task-updates have substantially lower representation alignment than full-rank updates, while KnOTS is associated with higher alignment. This motivates aligning LoRA updates as a step toward improving their mergeability.

Task-vector orthogonality

  • In a toy example, orthogonal task-vectors can still produce catastrophic merging that harms at least one model's predictions.The two task-vectors are [1, 1] and [−1, 1], yet a scaled sum preserves one model's predictions while flipping the other's.Source: §2 · p. 4

Representation alignment

  • CKA measures similarity between models' intermediate activations, with values near one indicating that the models process information similarly.The paper uses CKA as a measure of functional alignment between model weights.Source: §2 · p. 4
  • LoRA updates have drastically lower average CKA than full-rank updates, while KnOTS yields higher alignment than LoRA.The figure caption describes full-rank models as highly aligned, LoRA models as drastically less aligned, and KnOTS models as dramatically more aligned.Source: §2 · pp. 4–5

Alignment hypothesis

  • The authors hypothesize that aligning LoRA task-updates is a crucial first step toward better mergeability and performance of existing merging methods.They seek an alignment method that does not assume access to data or gradients.Source: §2 · p. 5

4 METHOD: KNOTS

KnOTS uses SVD to align LoRA task-updates layer-wise in a shared representation space. Existing merging methods can then combine the aligned task-specific components to form a merged update.

Subspace alignment with the SVD

  • KnOTS concatenates task-updates across tasks and applies SVD to align them in a shared space.The method is data-free and gradient-free, and is designed to improve existing merging methods.Source: §4 · p. 5
  • The decomposition gives each task-update a shared UΣ term and a task-specific [V(i)]^T component.UΣ defines the orthogonal basis for activations from all task-updates; the task-specific components act on the same information.Source: §4 · pp. 5–6

Applying merge methods

  • After aligning updates, existing merging functions can combine the task-specific components, which are reconstructed with the remaining SVD components and added to pretrained weights.The paper gives a weighted sum as an example of a merging function.Source: §4 · p. 6

5 EXPERIMENTS AND RESULTS

Across vision and language benchmarks, KnOTS improves existing LoRA merging methods; the section also evaluates merged-model generality with a new joint-task benchmark and analyzes how task count, LoRA rank, and update concatenation affect results.

5.1 EXPERIMENTAL DETAILS

  • Experiments use gradient-free merging methods, including TA, TIES, DARE, and RegMean; LoRA is applied to attention key, query, value, and output matrices, with rank 16 by default.The paper tunes baseline merging hyperparameters on held-out validation data.Source: §5.1 · pp. 6–7

5.2 PER-TASK EVALUATIONS ON VISION AND NLP SETTINGS

  • 4.3% average accuracy is the reported ViT-B/32 KnOTS-TIES gain over baselines; KnOTS-TIES also improves TIES by 3% on ViT-L/14 and up to 2.9% average normalized accuracy on Llama3-8B NLI.The per-task evaluations cover vision and language tasks.Source: §5 · p. 6, Table 1 · p. 7, §5.2 · p. 8

5.3 A NEW BENCHMARK TOWARDS BUILDING GENERAL MODELS

  • 3.2% on Hits@1 is KnOTS-TIES' reported gain on the 748-label Union, where the joint-task tests classification across labels from all eight datasets.The authors describe this setting as a more challenging test of whether a merged model is general.Source: §5.3 · pp. 8–9, Table 4 · p. 8

5.4 ADDITIONAL ANALYSIS & ABLATIONS

  • For merges of more than two tasks, KnOTS-TIES maintains a performance gap consistently above 4%; across LoRA ranks, it beats TIES by 6% at rank 4 (64.6% vs. 58.6%) and 4% at full rank.The ViTs have feature dimension 768, so rank 768 is full rank.Source: §5.4 · p. 9
  • Row-wise task-update concatenation yields 65.4% average normalized accuracy, 2.6% below column-wise concatenation; the authors suggest the row-wise decomposition may lower alignment.They identify column-wise concatenation as crucial to strong KnOTS performance.Source: §5.4 · p. 10

6 CONCLUSION

The paper attributes weak LoRA merging to parameter misalignment and introduces KnOTS, which uses SVD to place task updates in a shared space for merging. It also explains why merging LoRA factors separately can create mismatched products.

6 CONCLUSION

  • KnOTS uses SVD to map LoRA parameters into an aligned shared representation space, where existing merging methods can be applied.The conclusion reports improvements of up to 4.3% across vision and language settings, including a new benchmark for evaluating general models.Source: §6 · p. 10
  • Prior merging methods do not transfer well to LoRA models, which the paper associates with parameter misalignment between models.The paper studies LoRA models sharing a pretrained checkpoint and finetuned on different tasks without additional finetuning.Source: §6 · p. 10

A WHICH LORA REPRESENTATION IS BEST FOR MERGING?

  • Separately merging LoRA factors can produce mismatched products because factors from different models are not guaranteed to align.The passage notes that including such cross-model terms can cause a significant performance drop.Source: A WHICH LORA REPRESENTATION IS BEST… · p. 14
  • The paper therefore merges each model’s original update representation, avoiding factor mismatches when applying existing merging methods.It notes that the issue also persists with the cited alternative merging method.Source: A WHICH LORA REPRESENTATION IS BEST… · p. 14

B COMPUTING CENTERED KERNEL ALIGNMENTS ACROSS TASKS

The section describes how the authors compute pairwise CKA for full-rank, LoRA, and KnOTS-aligned models, then outlines how they tune merging baselines. The supplied passages specify procedures rather than comparative CKA outcomes.

B COMPUTING CENTERED KERNEL ALIGNMENTS ACROSS TASKS

  • For full-rank models, the authors subtract pretrained-model activations from each finetuned model’s activations, compute pairwise CKA by layer, and average across layers.They use intermediate activations from key, query, value, and projection layers across eight finetuned ViT-B/32 models.Source: B COMPUTING CENTERED KERNEL ALIGNMENTS… · p. 14
  • For LoRA models, they compute CKA directly on activations from corresponding layers across model pairs and average over layers.The LoRA comparisons use the same held-out data as the full-rank plots.Source: B COMPUTING CENTERED KERNEL ALIGNMENTS… · pp. 14–15
  • For KnOTS-aligned LoRA models, CKA is computed over outputs of the aligned [V(i)]T matrices at corresponding layers.The values are averaged across layers using the same data as the other LoRA CKA plots.Source: B COMPUTING CENTERED KERNEL ALIGNMENTS… · p. 15

C TUNING THE HYPERPARAMETERS FOR DIFFERENT MERGING BASELINES

  • Baseline tuning searches scaling coefficients from 0.1 to 1.0 and top-k values from 10 to 100; TIES and DARE are tuned by first selecting scaling, then the pruning parameter.The DARE pruning coefficient is searched from 0.99 to 0.1; the default top-k for TIES is 30 and the default DARE coefficient is 0.9.Source: C TUNING THE HYPERPARAMETERS FOR… · p. 15

D TRAINING DETAILS

The training details cover vision LoRA models, Llama-3 models for NLI, and the evaluation and compute setup. The section also identifies dataset-license information and cases where licenses could not be found.

D TRAINING DETAILS

  • Vision models use CLIP ViT-B/32 or ViT-L/14 with rank-16 LoRA on query, key, value, and output layers, while the text encoder remains frozen.The models use AdamW, cosine learning-rate scheduling, and cross-entropy loss.Source: D TRAINING DETAILS · p. 15
  • For NLI, the authors LoRA-finetune Llama-3 8B on attention query, key, value, and output weights, masking missing classes for datasets that use only two classes.The models use a linear learning-rate scheduler, a learning rate of 3e-5, and warm steps equal to 6% of training steps.Source: D TRAINING DETAILS · p. 15
  • During NLI evaluation, the merged Llama backbone is paired with a task-specific head for each dataset.This backbone-and-head arrangement is used for both merged and unmerged models.Source: D TRAINING DETAILS · p. 16
  • Experiments ran on machines with one 48GB Nvidia A40 GPU and an eight-worker CPU; KnOTS can also run entirely on CPU.The authors note that more efficient SVD algorithms could be used.Source: D TRAINING DETAILS · p. 16
  • The authors report that licenses could not be found for DTD, RESISC45, SVHN, and SUN397.They identify licenses for Cars and GTSRB as Creative Commons, EuroSAT as MIT, and MNIST as GNU General Public License.Source: D TRAINING DETAILS · p. 16

E COMPARISON WITH A FINETUNING BENCHMARK

The section compares KnOTS with Fisher Weight Averaging in the same setting as Table 1. The authors report that Fisher matches TIES but is outperformed by KnOTS, which does not require training.

E COMPARISON WITH A FINETUNING BENCHMARK

  • KnOTS outperforms Fisher Weight Averaging, while Fisher matches TIES in the reported setting.The authors characterize KnOTS as not requiring training.Source: E COMPARISON WITH A FINETUNING BENCHMARK · p. 16
  • The authors classify Fisher Weight Averaging as finetuning because it adds learnable parameters optimized with gradient descent to obtain the merged model.They select the best reported Fisher configuration using the same held-out validation data as the other methods.Source: E COMPARISON WITH A FINETUNING BENCHMARK · p. 16
  • The Fisher search varied the number of examples for computing Fisher information matrices and the parameter-scaling term.The reported best configuration used 256 examples and a scaling coefficient of 1.0.Source: E COMPARISON WITH A FINETUNING BENCHMARK · p. 16

F JOINT TASK FULL PERFORMANCES

On the joint task, KnOTS-TIES improves over merging baselines across nearly every evaluation, often trading top performance with the ensemble. The evaluation uses images from each dataset with a shared label set of 748 labels, for which the authors consider Hits@k suitable despite detected label synonyms.

F JOINT TASK FULL PERFORMANCES

  • Each joint-task dataset column evaluates images from that dataset using the shared label set of 748 labels.Source: F JOINT TASK FULL PERFORMANCES · p. 16
  • KnOTS-TIES improves over baselines by significant margins across nearly every joint-task evaluation, often trading best performances with the ensemble.The comparison covers eight LoRA-finetuned ViT-B/32 models merged across different datasets.Source: Table A2 · p. 17
  • A search using DistilBERT label embeddings and a cosine-similarity threshold of 0.8 finds 111 potential synonyms, leading the authors to regard Hits@k for k = {1, 3, 5} as suitable for joint-task evaluation.The authors report averages of 0.15 synonyms per label and 1.95 per labels that have synonyms.Source: F JOINT TASK FULL PERFORMANCES · p. 17

Related papers

Loading 2410.19735v1…