PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks

Avinash Amballa, Yashas Malur Saidutta, Wenbo Li, Lazar Valkov, Srinivas Chappidi

arXiv:2609.22237v1cs.LGcs.AI

TL;DR

Existing LoRA merging methods often allocate rank uniformly across layers and tasks, despite evidence that task and layer needs vary. Net Utility scores singular directions from adapter weights and selects them globally under a rank budget. Across vision and language tasks, this allocation improves performance over uniform allocation on average.

  • Problem

    Many merging methods allocate rank uniformly across layers, and some also divide it equally among tasks, although layer importance and spectral energy can vary.

    Source: Introduction · p. 1

  • Method

    Net Utility scores each task’s singular directions by own-task benefit against cross-task interference, then globally selects directions under a shared budget using adapter weights alone.

    Source: Introduction · p. 1

  • Results

    Across merging methods and spaces, Net Utility allocation improves performance by an average of +2.1% on vision tasks and +2.2% on language tasks.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The reported results identify uniform rank allocation as a major source of the performance gap between merged and per-task LoRAs.

    Source: Introduction · p. 2

    The derivation is restricted to linear merging, and λ is not tuned, though validation-set tuning could improve performance.

    Source: Net-Utility Allocation · p. 4, Limitations · p. 7

Abstract

from arXiv · show

Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too. We show this uniform-budget assumption is a major source of the performance gap between merged and per-task LoRAs. However, rank selection is an NP hard problem. To this end, we introduce Net Utility, a data free metric that first decomposes every task LoRA by its Singular Value Decomposition (SVD) and scores each of those singular directions by its task utility and its interference with other tasks directions. Next, we globally pool these scores to select singular directions with the highest values with a constraint on the total number of directions selected. The proposed Net Utility metric is applied on top of five different merging methods across three different merging spaces. The merging is done over two sets of tasks, vision and language tasks. Net utility based rank allocation outperforms its counterparts without that allocation. On average, over vision tasks it achieves +2.1% improvement in performance, and +2.2% improvement over the language tasks.

Introduction

LoRA merging can replace task-specific adapter swapping with one adapter for multiple tasks, but existing methods often allocate rank uniformly across layers and tasks. The paper proposes Net Utility to select singular directions under a shared budget, addressing uneven task and layer needs.

Introduction

  • Across seven vision datasets, non-uniform rank allocation improves average performance by nearly +3.4% with Task Arithmetic in the full space.The reported rank profiles vary across layers and tasks.Source: Introduction · p. 1, Figure 1 · p. 1
  • Uniform rank budgets can underrepresent important layers and overrepresent others because LoRA spectral energy and task importance vary across layers.Some methods also explicitly divide the budget equally among tasks.Source: Introduction · p. 1
  • Net Utility scores each task’s singular directions by their contribution to their own update reconstruction minus their interference with other tasks, using adapter weights alone.The optimal budget allocation is NP-hard, motivating a greedy criterion.Source: Introduction · p. 1
  • The method globally pools direction scores and retains the highest-scoring directions under the shared rank budget.This allocates rank across both layers and tasks.Source: Introduction · pp. 1–2
  • At matched budgets, Net Utility improves prior merging methods across three spaces by an average of +2.1% absolute on vision and language tasks, with some combinations reaching +3.8%.The evaluation covers seven vision tasks and six language tasks.Source: Introduction · p. 2

Related Work

Prior merging work spans task-vector, spectral, subspace, and LoRA-specific approaches, while rank-selection methods differ in their data and optimization requirements. Net Utility is presented as a data-free, closed-form allocation criterion that selects ranks globally across modules, layers, and tasks.

Related Work

  • Task-vector methods combine fine-tuned weight differences, often sparsifying by magnitude, random dropping, outlier removal, or parameter importance before summing.Net Utility scores individual singular directions to assign ranks, rather than setting a fixed sparsity ratio.Source: Related Work · p. 2
  • Spectral and subspace methods use shared, orthogonalized, flattened, or aligned singular bases, but commonly fix retained rank in advance and share it across layers.ACE-Merging is described as a merge operator orthogonal to the rank-allocation question.Source: Related Work · p. 2
  • LoRA-specific methods address adapter merging through magnitude and direction separation, rank-one clustering, or a common alignment basis.Core Space is described as merging within an alignment basis without information loss.Source: Related Work · p. 2
  • AdaRank uses unlabelled calibration data and gradient descent, while PRIME shares one rank across layers and CtM applies a bottleneck before merging.Net Utility instead scores adapter weights without data or an optimization loop and derives the rank profile through global top-K selection.Source: Related Work · p. 2

Methodology: Net Utility and Rank Selection

Net Utility decomposes task adapters into singular components, scores their source-task preservation and cross-task interference, then allocates a shared component budget across tasks and layers.

Methodology: Net Utility and Rank Selection

  • Net Utility assigns each rank-one adapter component a data-free score balancing source-task preservation against cross-task interference before allocating a shared budget.The method decomposes each task adapter into singular components and scores them at each adapted layer.Source: Methodology: Net Utility and Rank… · p. 2

Preservation Objective

  • The preservation objective seeks merged-model outputs that match each task’s individual adapter on that task’s inputs, using a surrogate based on linearization around a shared reference model.The surrogate assumes comparable sensitivity across adapted layers.Source: Preservation Objective · p. 2

Data-Free Input Geometry

  • Because task input statistics are unavailable, the method uses a scaled input-side Gram proxy derived from adapter weights; an alternative heuristic weights active right-singular directions equally.The Gram proxy gives greater weight to directions on which the adapter acts strongly, and no task data, validation set, or forward pass is used.Source: Preservation Objective · p. 2, Data-Free Input Geometry · p. 3

Algorithm

  • Selection is binary and unrescaled; per-layer selected count upper-bounds realized additive rank, while SVDs come directly from LoRA factors without materializing dense updates.The realized rank may be lower than the selected-component count.Source: Net-Utility Allocation · p. 3, Algorithm · p. 4

Net-Utility Allocation

  • A separable upper bound turns coupled component selection into dimensionless scores balancing normalized source-task benefit against cross-task interference, which vanishes for directions orthogonal to other tasks’ row spaces.The bound separates interference contributed by different tasks.Source: Net-Utility Allocation · pp. 3–4
  • Across tasks and layers, the method retains the largest positive scores under RG = L * R, leaving capacity unused when needed; nonlinear merges are empirical extensions without the same guarantee.The selection is exact for the separable surrogate, not the original model-level objective.Source: Net-Utility Allocation · p. 4

Experiments

The experiments allocate LoRA rank by scoring SVD directions and selecting positive-utility components across tasks and layers. They evaluate this approach on vision and language tasks using multiple merging methods, spaces, and rank budgets.

Experiments

  • Net Utility computes utilities for SVD components and selects the top positive scores across tasks and layers within a global component budget, then reconstructs the selected components for merging.The algorithm computes each adapted layer’s SVD and component utilities before selection.Source: Experiments · p. 4
  • Vision experiments use CLIP ViT-B/32 on seven tasks, while language experiments use Qwen 3-4B on six NLI tasks.The reported metric is normalized accuracy relative to the original LoRA on each task.Source: Experiments · p. 4
  • Evaluation covers Task Arithmetic, TIES, DARE, TSV, and Iso-C across budgets of 64, 32, and 16 directions per layer in full, core, and KnOTS spaces.TIES and DARE use local equal budgets because their sample-specific operations occur during merging or preprocessing.Source: Experiments · pp. 4–5
  • Rank budgets are below full capacity: 112 directions for vision and 96 for language, with each budget counting retained directions per layer across tasks.Full capacity is the number of tasks multiplied by the per-task rank.Source: Experiments · p. 5
  • Vision adapters have heterogeneity h = 1.83 versus h = 0.63 for NLI adapters, so the experiments set α = 0 for vision to avoid dominance by large-magnitude tasks.The study uses variance of logarithms as its measure of heterogeneity.Source: Experiments · p. 5

Results

Net Utility rank allocation generally outperforms uniform allocation across the reported vision and language merging experiments. The gains also appear with Iso-C and transfer across merging methods and spaces, while at full vision rank the algorithm leaves much of the budget unused.

Vision

  • Vision allocation beats uniform allocation across all three spaces and four merging methods, reaching +3.8 over DARE in full space at budget 64.Gains are largest for DARE (+2.3 to +3.8) and TA (+2.3 to +3.4), while TSV improves by +0.3 to +0.8.Source: Results · p. 5

Language

  • Language allocation improves over uniform in most methods and spaces, with a maximum gain of +3.5 on TSV at budget 16.DARE in core and KnOTS space is the exception, staying within 0.4 of uniform and slightly below it at budgets 64 and 32.Source: Results · p. 5
  • In language experiments, gains tend to grow as budgets shrink; task homogeneity is offered as a possible reason TSV benefits especially strongly.The authors describe the homogeneity explanation as a possibility.Source: Results · pp. 5–6

Iso-C

  • At R=16 in full space, allocation improves over uniform in all four Iso-C settings: +1.3 and +1.2 on vision, and +1.0 and +0.5 on language.The table covers Iso-C results for both domains.Source: Table 3 · p. 6, Results · p. 6

Budget use

  • At full vision rank R=112, only 55% of directions carry positive net utility; discarding the rest raises accuracy by 3.19 while leaving over half the budget unspent.At lower ranks, the entire budget is spent.Source: Table 4 · p. 6

Transfer across methods

  • Net Utility derived for task arithmetic improves other merging methods across spaces, although the decomposition is inexact for nonlinear TSV, DARE, and TIES.The reported transfer includes truncation, random masking, and trimming with sign election.Source: Results · p. 6

Analysis

The analysis finds that Net Utility can leave harmful directions unselected, choose non-prefix sets, and allocate ranks unevenly across layers and modules. On MATH topics, mean Net Utility is inversely correlated with relative accuracy.

The budget need not be spent

  • At budget 112, the method uses 48% of the budget and achieves 3.19% more accuracy than using all the budget.It retains an average of 61.5 directions per layer; at lower budgets it uses all available slots when positive-utility directions outnumber them.Source: The budget need not be spent · p. 6

The optimal set need not be a prefix

  • At R=64, 88% of vision sets are non-prefix, and on language 64.5% are non-prefix; on vision, the strongest direction is retained 12.8% of the time versus 75.8% for the weakest.A low-interference direction can outrank a dominant but heavily shared direction, so the optimal selection need not be a prefix.Source: The optimal set need not be a prefix · p. 6, Figure 3 · p. 7

Not all Layers/Modules are equally important

  • For ViT-B/32 on seven vision tasks using Task Arithmetic in full space, later layers—and especially v-proj and out-proj—receive more ranks than early layers.The allocation deviates substantially from uniform rank across module and layer combinations.Source: Not all Layers/Modules are equally… · p. 6, Figure 4 · p. 7

Analyzing the net utility

  • Across seven rank-32 topic adapters for Hendrycks MATH, mean Net Utility is inversely correlated with relative accuracy (−0.78), associating higher utility with harder topics.Relative accuracy is adapter accuracy minus base-model accuracy on the same topic; the adapters share an identical training setup.Source: Analyzing the net utility · pp. 6–7

Conclusion

The conclusion presents Net Utility as a greedy, global allocation of singular directions that weighs task benefit against cross-task interference. Across the evaluated merging methods and spaces, it improves average performance on vision and language tasks.

Conclusion

  • The conclusion also notes a negative correlation between Net Utility and relative task performance on Hendrycks Math.Figure 5 presents the Net-Utility versus performance comparison.Source: Figure 5 · p. 7, Conclusion · p. 7
  • Across five merging methods and three merging spaces, Net Utility allocation improves average performance by 2.1% on vision tasks and 2.2% on language tasks.The evaluation covers seven vision tasks and six language tasks.Source: Conclusion · p. 7
  • Net Utility scores each task LoRA singular direction by its task benefit relative to cross-task interference, then globally pools directions for budget-constrained selection.Selected directions are merged to form the merged LoRA.Source: Conclusion · p. 7

Limitations

The authors identify two limitations: the Net Utility derivation is restricted to linear merging, and λ is not tuned.

Limitations

  • The Net Utility derivation is restricted to linear merging.The authors say a derivation for a specific merging algorithm could further boost performance.Source: Limitations · p. 7
  • The authors do not tune the hyperparameter λ.They state that tuning λ with access to a validation set would improve performance.Source: Limitations · p. 7
  • The proposed performance improvement from λ tuning is conditional on access to a validation set.The limitation concerns the tuning setup, not a reported comparison across validation-set choices.Source: Limitations · p. 7

A.1 Layer-Wise Preservation Surrogate

The appendix motivates a layer-wise preservation surrogate from a bound, then states a separable upper bound and a net-utility decomposition under specified conditions.

A.1 Layer-Wise Preservation Surrogate

  • The layer-wise preservation surrogate is motivated when Jacobian norms are approximately constant across adapted layers and inputs of interest.Under that condition, their common scale does not affect the minimizer.Source: §A.1 · p. 9

A.2 Separable Upper Bound

  • The separable upper bound holds for λ > 0 when either G_i or its alternative is used consistently and the corresponding normalization energy is nonzero.The proposition states equality for M = 1.Source: §A.2 · p. 9

A.3 Net-Utility Decomposition

  • Under the stated Gram geometry and positive task weights, the net-utility decomposition equates normalized source-task residual and cross-task interference terms with the expression in Equation (7).Zero-energy updates are omitted because they contain no candidates.Source: §A.3 · p. 9

A.4 Heuristic Row-Space Utility

The heuristic variant changes the weights used in Algorithm 1 while leaving its other steps unchanged. The appendix also gives the cost of computing each adapter’s thin SVD and pooling selected components.

A.4 Heuristic Row-Space Utility

  • Its justification follows the same bilinear-form orthogonality argument as Proposition 2.Source: §A.4 · p. 9
  • The heuristic variant replaces Algorithm 1’s w terms with their heuristic counterparts; all other steps are unchanged.Source: §A.4 · p. 10

A.5 Computational Complexity

  • A rank-r adapter’s thin SVD costs O(r^2(din + dout)) per adapter, without materializing the dense update.When C components are pooled, selecting them costs O(C log C).Source: §A.5 · p. 10

B LoRA Training Details for QWEN3-4B adapters

The experiments train one LoRA adapter for each of six NLI datasets using Qwen3-4B-Instruct, with a shared three-class output space. Adapters use attention projections only, and training selects checkpoints by validation accuracy before a single held-out test evaluation.

LoRA configuration

Tasks and label space

  • Six NLI datasets use a unified three-class output space so their adapters and classification heads remain shape-compatible for merging.RTE and QNLI map not-entailment to contradiction; SciTail retains neutral as class 1, and binary-task unused classes are masked.Source: B LoRA Training Details for QWEN3-4B… · p. 10

Optimization

  • Training uses AdamW with a 3 × 10^-5 learning rate for at most 10 epochs, retaining the best-validation-accuracy checkpoint for one held-out test evaluation.Validation is checked every 4,000 steps, with early stopping after three evaluations without improvement.Source: B LoRA Training Details for QWEN3-4B… · p. 10
Loading 2609.22237v1…