PaperlayerView exact revision on arXiv

Source-linked AI summary

Accurate and Efficient Low-Rank Model Merging in Core Space

Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Bartłomiej Twardowski, Andrew D. Bagdanov, Simone Calderara, Joost van de Weijer

arXiv:2509.17786v4cs.CVcs.AI

TL;DR

Merging low-rank adaptations remains difficult because existing methods can require costly full-space operations, undermining LoRA’s efficiency. Core Space provides a shared low-rank alignment basis with no information loss, and experiments report improved state-of-the-art performance across vision and language tasks at lower computational cost.

  • Problem

    Existing LoRA merging methods can abandon low-rank representations through full-space operations, reducing efficiency and becoming prohibitively expensive for large models.

  • Method

    Core Space projects task-specific LoRA updates into a compact common alignment basis that supports arbitrary merging techniques while preserving information.

  • Results

    Core Space improves tested merging methods across vision and language tasks, reaching 76.3% average normalized accuracy for TSV + Iso-C on ViT-B/32 and 94.16% for TSV on Llama 3 8B.

  • Takeaways & Limitations

    Core Space combines low-rank parameter efficiency with improved merging performance and is presented as scalable for practical multi-task adaptation of large models.

  • Takeaways & Limitations

    The theoretical derivation initially assumes the total LoRA rank does not exceed the target matrix dimensions, although a more general analysis preserves the zero-alignment-error result.

Abstract

from arXiv · show

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

1 Introduction

Large models make full fine-tuning and merging increasingly costly, motivating efficient low-rank adaptation and Core Space merging for LoRA models. Core Space preserves low-rank efficiency while improving merging performance across vision and language tasks.

  • 1 Introduction: LoRA reduces trainable parameters by reparameterizing model updates, making fine-tuning large architectures more accessible.This efficiency supports task-specific adaptation by a broader range of users.
  • 1 Introduction: Existing alignment-space merging improves compatibility but requires full-space SVD, eliminating low-rank efficiency and becoming expensive as models grow.The limitation motivates a representation that retains decomposed low-rank updates.
  • 1 Introduction: Across vision and language tasks, Core Space achieves the best results on ViT-B/32, ViT-L/14, and Llama 3 8B while preserving parameter efficiency.The paper evaluates direct low-rank merging, KnOTS-space merging, and Core Space merging.
  • 1 Introduction: Core Space merges LoRA-adapted models in a shared low-rank basis, avoiding costly full-space operations while improving accuracy.The framework supports integration with existing merging methods.
  • 1 Introduction: Projection into Core Space is proved information-preserving, and complexity analysis demonstrates efficiency gains from merging in the proposed space.These properties address both accuracy and computational-cost concerns.

2 Related Work

Prior work developed parameter-efficient fine-tuning and model merging, but methods for fully fine-tuned models do not necessarily transfer efficiently or effectively to LoRA-adapted models. KnOTS improves shared-subspace merging but incurs full-matrix SVD costs.

  • 2 Related Work: LoRA and related PEFT methods update only a small fraction of parameters, reducing the cost of adapting large pretrained models.The related-work discussion identifies LoRA as a widely used low-rank PEFT technique.
  • 2 Related Work: Model merging combines task-specific expert models into one multitask model, commonly by aggregating task vectors derived from expert and base parameters.Task Arithmetic is presented as a basic weighted-sum approach.
  • 2 Related Work: Methods designed for fully fine-tuned models do not necessarily transfer to LoRA models, while KnOTS improves shared-subspace merging at high computational cost.KnOTS performs SVD on concatenated full-size matrices rather than decomposed low-rank representations.

3 Preliminaries

LoRA represents each task adaptation as a low-rank update to a shared pretrained model, while conventional task-vector merging aggregates full parameter differences. Directly merging LoRA-derived weight matrices can be suboptimal because their updates are less aligned.

  • 3 Preliminaries: LoRA replaces full weight updates with W = W0 + BA, where A and B are low-rank matrices and r ≪ min(m, n).This parameterization substantially reduces the number of trainable parameters.
  • 3 Preliminaries: Basic model merging computes task vectors ΔWi = Wi − W0 and adds them to the base model using weighted aggregation.For LoRA-adapted models, the updates are represented as ΔWi = BiAi.
  • 3 Preliminaries: Directly merging weight matrices reconstructed from LoRA updates can produce suboptimal performance because LoRA models are less aligned with fully fine-tuned counterparts.This alignment issue motivates specialized merging methods for low-rank adaptations.

4 The Core Space Merging Framework

Core Space Merging reparameterizes LoRA updates in shared reference bases, enabling merging in a compact low-rank space while preserving information and improving efficiency and efficacy.

  • Core Matrix Alignment and Merging: Each task is projected into a core matrix M^(t) in the shared bases, and any merging operator can combine these matrices before reconstruction.The merged update is recovered by projecting the merged core matrix back through the reference bases.
  • No Information Loss: Core Space is lossless for individual task updates, and linear merging is exactly equivalent to applying the same merge in full model space.This establishes information preservation while allowing the computation to remain in the reduced representation.
  • Efficiency: The core matrices have size Tr × Tr rather than m × n, allowing costly merging algorithms to use a fraction of the time and memory footprint.The approach has complexity comparable to full-space Task Arithmetic and scales better than KnOTS, whose complexity includes a super-cubic dependence on weight-matrix size.
  • Efficacy: Core Space merging improves performance over full-space merging for nonlinear methods through better alignment and more compact task-direction representations.The framework is designed to integrate with existing merging functions, including arithmetic averaging and more advanced techniques.
  • Reference Bases: Core Space uses SVD-derived reference bases from stacked low-rank factors to represent all task updates in a shared coordinate system.The bases span the task-specific directions and enable aligned core matrices without requiring full-space reconstruction.

5 Experimental Results

Across vision and language experiments, Core Space makes LoRA model merging substantially more efficient while improving performance across merging methods and evaluation settings.

  • Complexity analysis: Core Space has time complexity comparable to Task Arithmetic in full space, while avoiding KnOTS’s super-cubic dependence on weight-matrix size.The additional Core Space terms are negligible unless T · r becomes significantly large.
  • LLMs merging: 94.16% average normalized accuracy elevates TSV in Core Space to state-of-the-art performance on Llama 3 8B NLI tasks.Core Space also provides up to 600× merging speed-up over Full or KnOTS space.
  • Per-task evaluation in vision setting: 76.3% average normalized accuracy makes TSV + Iso-C in Core Space state of the art, outperforming TIES in KnOTS space at 68.0%.This is reported for ViT-B/32 vision tasks; similar conclusions hold for ViT-L/14.
  • Per-task evaluation in vision setting: Eight of nine merging methods achieve their highest average vision accuracy in Core Space.The comparison uses ViT-B/32 and normalized accuracy across per-task vision evaluations.
  • Additional PEFT methods: Core Space retains applicability beyond standard LoRA, supporting heterogeneous ranks and producing improvements over baselines for VeRA.Heterogeneous-rank modules are concatenated into an aggregate basis, while VeRA scaling vectors are absorbed into its matrices.
  • Joint-task evaluation in vision setting: Core Space achieves state-of-the-art results in the joint-task vision setting when combined with Iso-C.The setting requires distinguishing classes across the union of all tasks without a task ID during inference.
  • Truncation: Removing up to 80% of full-space components causes no performance loss, whereas removing any Core Space component reduces accuracy.This indicates that full space contains redundant components while Core Space is information-dense.
  • Core Space improves subspace alignment: Core Space yields consistently higher Subspace Alignment Ratios than full space, and higher SAR correlates with lower merging interference.The shared basis is argued to filter task-specific noise and promote alignment.

6 Conclusion

Core Space merges LoRA modules through a common subspace while preserving efficiency, reducing alignment error, and improving accuracy across vision and language settings. Its theoretical development establishes orthonormal reference bases and zero alignment error under the stated construction.

  • 6 Conclusion: Core Space projects task-specific LoRA updates into a common subspace, reducing alignment error while delivering consistent accuracy improvements and state-of-the-art results across vision and language tasks.The conclusion also emphasizes scalability and computational efficiency for practical multi-task adaptation.
  • 6 Conclusion: The reference bases are formed from orthonormal singular-vector components of stacked low-rank factors, with dimensions governed by the number of tasks and LoRA rank.The construction uses shared bases for the A and B components and represents them with T · r dimensions.
  • 6 Conclusion: The least-squares alignment objectives are convex with unique global minimizers because the reference bases have full column rank.The derivation solves the optimal coefficients by setting the objective gradients to zero.
  • 6 Conclusion: The resulting reference bases satisfy orthonormality conditions and achieve zero alignment error at the optimum.The proof verifies the stationarity conditions and evaluates the optimal objective value.

A.3 Optimal Reference Bases

The optimal reference basis is obtained from singular vectors of stacked LoRA factors and yields perfect reconstruction with zero alignment error. These guarantees extend to multiple tasks and remain valid when the total nominal rank exceeds the feature dimensions.

  • A.3 Optimal Reference Bases: The optimal reference basis consists of top singular vectors of the LoRA factor matrix and achieves the maximum objective value r with zero alignment error.Equivalently, the basis contains the top r eigenvectors of B(t)B(t)⊤.
  • A.3 Optimal Reference Bases: The constrained optimization is solved by an orthonormal basis containing the relevant eigenvectors, with Lagrange-multiplier conditions enforcing stationarity.The basis is completed with orthonormal vectors spanning the remaining dimensions.
  • A.3 Optimal Reference Bases: For multiple tasks, horizontally stacking all B(t) matrices and taking their left singular vectors produces orthonormal bases that guarantee perfect reconstruction and zero alignment error.The construction uses the stacked matrix B across all tasks and defines the associated diagonal eigenvalue matrix.
  • A.3 Optimal Reference Bases: The theoretical guarantees remain unchanged when T · r exceeds m or n because the stacked factors span the relevant update spaces through their intrinsic ranks.Truncated bases retain the columns spanning the full LoRA update space.

B Computational Complexity Analysis

Core Space reduces complexity by operating on low-rank factors and a task-rank-sized core instead of repeatedly decomposing full-dimensional updates. Under m = n and T, r ≪ n, its dominant terms depend on T, r, and lower-dimensional operations rather than the cubic task-scaled full-space costs of competing methods.

  • KnOTS Complexity: KnOTS simplifies to O(n^3T^2) under m = n, retaining cubic feature dependence and adding a quadratic factor in the number of merged tasks.Its cost is driven by full-space SVD, merging, and reconstruction.
  • Iso-C Complexity: Iso-C remains approximately cubic in feature dimension, with complexity O(n^3 + T · r · n^2) when m = n.Its cost includes full-space summation, SVD, and isotropization.
  • TSV Complexity: TSV has overall cost dominated by O(T · n^3) when m = n and T, r, k ≪ n.The method includes full-space conversion, per-task and global SVD computations, concatenation, and merging.
  • Core Space Complexity: When m = n and T, r ≪ n, Core Space simplifies to O(Tr(2n + 5n^2 + r^2) + T^2r^2(3n + T) + T^3r^3), dominated by its low-rank terms.The simplified expression follows from the general complexity formula.

C Additional Analysis

The additional analysis examines interference and the effective rank of merged updates when combining LoRA models. Core and KnOTS preserve a constrained T · r-dimensional structure, whereas full-space TIES can lose that low-rank structure.

  • C.1 High subspace alignment leads to lower interference: Interference is measured as the L1 distance between task-specific and merged final embeddings, comparing TSV + Iso-C in Full Space and Core Space.Activations are collected from the final layer’s projection into a common vision-language space.
  • C.2 Rank of the merged update matrices: Core Space and KnOTS operate directly in a constrained T · r-dimensional space, preserving the intended low-rank structure of merged updates.For the analyzed setting, T = 8 and r = 16, so T · r = 128.
  • C.2 Rank of the merged update matrices: Full-space TIES can produce effective ranks approaching d = 768 because trimming reconstructed updates destroys their low-rank structure.This comparison concerns eight ViT-B/32 LoRA models with rank 16.

Licenses of Used Datasets and Models

The paper uses publicly available datasets and models with stated licenses or research-use restrictions, and documents its experimental implementation and tuning procedures. Core Space merging improves vision results while retaining practical efficiency.

  • Experimental setup: Experiments use KnOTS LoRA checkpoints, validation-set linear hyperparameter search, and separately specified hardware for Llama 3 8B language and vision evaluations.Language experiments use one 48G NVIDIA L40S, while vision experiments use one 16G NVIDIA RTX 4080.

E.2 Experiments with Heterogeneous Ranks

Core Space handles heterogeneous LoRA ranks by spanning their combined task subspaces with SVD-derived reference bases and aligning each task without reconstruction loss. The framework also extends beyond LoRA to VeRA, where it consistently outperforms other approaches.

  • E.2 Experiments with Heterogeneous Ranks: SVD of stacked task matrices produces orthonormal reference bases spanning the union of task subspaces, regardless of individual rank differences.The aggregate matrices have rank equal to the dimension of the union of the task subspaces.
  • E.2 Experiments with Heterogeneous Ranks: Mixed LoRA ranks incur no additional performance degradation in Core Space because projection and task-specific alignment preserve lossless reconstruction.Baselines without alignment struggle when task subspace dimensions differ.
  • E.2 Experiments with VeRA: Core Space consistently outperforms other approaches on VeRA after its trainable scaling vectors are absorbed into effective low-rank matrices.The subsequent stacking, SVD, projection, and alignment steps are unchanged because they are agnostic to how the matrices were obtained.
Loading 2509.17786v4…