PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

Yixuan Tang, Yi Yang

arXiv:2604.16826v1cs.CL

TL;DR

LoRA merging often degrades task capabilities because existing methods treat ΔW = BA as a single object despite asymmetric behavior in A and B. Pico calibrates over-shared directions in B before merging, and improves three downstream mergers across eight datasets and four domains, with gains of 3.4–8.3 points over corresponding baselines.

  • Problem

    Existing LoRA merging methods treat ΔW = BA as a single object without distinguishing A and B, despite their different roles and contributions to adapted change.

  • Method

    Pico is a data-free pre-merge calibration that downscales over-shared directions in task-specific B matrices, then rescales the merged update and plugs into existing merge rules.

  • Results

    Pico improves average scores over corresponding uncalibrated baselines by 3.4 points for Task Arithmetic, 4.7 points for TIES, and 8.3 points for TSV-M across eight datasets from four domains.

  • Takeaways & Limitations

    Treating A and B separately improves multiple LoRA merging methods and yields a stronger multi-domain adapter than a jointly trained multi-task LoRA.

  • Takeaways & Limitations

    The formulation absorbs the usual LoRA scaling factor into B_t throughout the paper.

Abstract

from arXiv · show

Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update $ΔW = BA$ as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix $B$. Across tasks, $B$ repeatedly uses a small set of shared directions, while $A$ remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates $B$ before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.

1 Introduction

LoRA merging combines compact task adapters but often degrades their individual capabilities because existing methods treat ΔW = BA as one object. The paper identifies B as the main source of interference and proposes calibrating shared B directions before merging.

  • Motivation: Merging independently trained LoRA adapters can degrade individual task performance despite offering a way to combine capabilities without joint retraining.Existing methods reduce sign conflicts, prune weights, or align updates, but still operate on the full update.
  • Motivation: A and B play different roles in ΔW = BA: A maps inputs into the low-rank space, while B maps that representation back to output space.Prior methods also freeze or share A while retaining more task-specific change in B.
  • Findings: Cross-task overlap is consistently larger in B than A, while B retains low effective rank as rank increases across math, coding, finance, and medical LoRAs.These patterns indicate repeated reuse of a small set of output-space directions.
  • Method: Pico identifies over-shared directions in task-specific B matrices, downscales them before merging, and rescales the merged update to preserve its magnitude.It is data-free and plugs into Task Arithmetic, TIES, and TSV-M rather than replacing them.
  • Results: Pico improves average scores over uncalibrated baselines by 3.4 points for Task Arithmetic, 4.7 points for TIES, and 8.3 points for TSV-M.Across eight datasets from four domains, it also outperforms several other merging methods and can exceed a jointly trained LoRA.

2 Background and Problem Setup

The paper formalizes merging as combining independently trained low-rank updates while diagnosing how shared components can dominate and dilute task-specific components. It measures this interference through overlap and concentration in A and B.

  • Setup: For adapter t, the layer update is ΔW_t = B_tA_t, with A_t and B_t having dimensions determined by the LoRA rank r.The merged update combines T independently trained adapters with the same dimension.
  • Setup: LoRA merging aims to combine adapter capabilities without retraining on the union of task data.The usual LoRA scaling factor is absorbed into B_t for the paper’s formulation.
  • Merge interference: Under a linear merge, shared components retain their magnitude while task-specific components are diluted, allowing aligned directions to dominate the merged update.Under simple averaging, the shared-to-specific ratio grows by a factor of T.
  • Diagnostics: Overlap scores O_B(i,j) and O_A(i,j) measure shared output- and input-side subspace directions, respectively, with larger values indicating greater overlap.Effective rank measures spectral concentration, where lower values mean fewer directions contribute substantially.

3 Why Does LoRA Merging Fail?

The analysis compares four domain-specific LoRA adapters across query and value projections and ranks 8–64. It finds that B has greater cross-task overlap, remains concentrated, and can be dominated by a few shared components.

  • Experimental setup: The study analyzes math, coding, finance, and medical LoRAs trained on Llama-3.1-8B, using 50k examples per domain and ranks 8, 16, 32, and 64.LoRA targets the query and value projection matrices in attention, with LoRA alpha fixed at 16.
  • Overlap: At rank 8, B overlap exceeds A overlap for 99.5% of query-projection and 95.3% of value-projection layer pairs; at higher ranks, it holds for every measured pair.In query projections, mean B overlap rises from 0.0166 to 0.0839, while A overlap rises from 0.0035 to 0.0172.
  • Concentration: At rank 64, average effective rank is 2.94 for B versus 6.55 for A in query projections and 2.59 versus 5.30 in value projections.Thus B remains concentrated in relatively few directions while A becomes more spread out.
  • Shared-spectrum dominance: In the representative query-projection example, the first three shared B components contain 53.7% of energy and the first five contain 68.8%.Finance, math, and medical contribute more strongly to the dominant shared components than coding.

4 Pico: Pre-merge interference calibration in output-space

Pico calibrates output-side LoRA matrices before merging by identifying heavily shared directions, reducing them, and restoring the merged update’s magnitude. It uses a shared SVD-based basis and remains compatible with standard merger rules.

  • Pico adds a data-free calibration step before standard LoRA merging, reducing over-reused directions in B and then rescaling the merged update.The method is plug-in rather than a replacement merger.
  • Shared Basis: Pico builds a shared output-space basis by stacking task-specific B matrices and computing a joint singular value decomposition.The columns of U provide shared basis vectors, while σj records singular values.
  • Sharing Score: Each basis component receives a sharing score and scaling coefficient, with heavily shared components reduced toward 1/T while lightly shared components remain near 1.The score reflects how much joint B-space energy lies in each component.
  • Calibration Operator: Pico changes only B, leaving the more task-specific A matrix untouched, and requires one stacked-B SVD plus task-specific left multiplications per layer.The calibrated update is g∆W_t = eB_tA_t.
  • Merge and Rescaling: The calibrated updates are passed to Task Arithmetic, TIES, TSV-M, or another merger, after which Pico rescales the result to match source-adapter magnitude.Rescaling preserves the calibrated direction while restoring overall update magnitude.

5 Experiment

Experiments evaluate Pico across four domains, eight benchmarks, three merge rules, robustness settings, and comparisons with specialized and jointly trained LoRAs. Pico consistently improves merging, with gains tied to calibrating B and restoring merged-update magnitude.

  • Experimental Setup: The evaluation uses four domain-specific adapters, eight benchmarks, and Task Arithmetic, TIES, and TSV-M as downstream merge rules.The domains are math, coding, finance, and medical reasoning; the main metric is the average across eight benchmarks.
  • Main Results: 3.4, 4.7, and 8.3 points are the absolute gains from Pico over Task Arithmetic, TIES, and TSV-M, respectively.Average scores rise from 0.4093 to 0.4430, 0.3859 to 0.4328, and 0.3473 to 0.4305.
  • Main Results: Pico’s gains persist across different downstream mergers because it calibrates over-shared B directions before the merger operates.The method improves all three tested merge rules rather than replacing them.
  • Comparison with Single-Task and Jointly Trained LoRA: The best merged adapter reaches 0.4430 overall, exceeding the jointly trained multi-task LoRA’s 0.3688 by 7.4 points.Single-domain LoRAs remain specialized, whereas the jointly trained adapter is more balanced but has a lower overall average.
  • Ablations and Analysis: Calibrating only B raises Task Arithmetic’s overall average from 0.4093 to 0.4430, while calibrating A or the full update lowers performance.These ablations support concentrating calibration on the output-side matrix rather than modifying task-specific A structure.
  • Ablations and Analysis: Magnitude restoration is necessary after B-space calibration: removing it lowers the overall average to 0.3908, despite finance improving from 0.5431 to 0.5646.Coding and medical performance drop sharply without restoration.
  • Ablations and Analysis: Pico remains strongest across ranks 8, 16, 32, and 64, on Qwen3-4B-Base, and as the coding merge pool grows from two to six adapters.The merged B spectrum also becomes less dominated by leading shared components after calibration.

6 Related Work

Related work spans LoRA adaptation, model and LoRA merging, and methods that reduce interference in shared bases or low-rank subspaces. Pico extends these directions by focusing calibration on the output-side LoRA factor before merging.

  • LoRA adaptation and asymmetry: LoRA represents each task update as a low-rank product ∆W = BA, and prior work treats A and B asymmetrically during adaptation.HydraLoRA shares A across task-specific B heads, while other work analyzes asymmetric update changes.
  • Model and LoRA merging: Task Arithmetic composes finetuned models directly in parameter space, while TIES, DARE, and DELLA reduce interference through trimming, sparsification, rescaling, or magnitude-based sampling.LoRA Soups frames adapter merging as skill composition, and LoRA-LEGO decomposes merging more finely.
  • Shared-basis and low-rank merge methods: KnOTS, TSV-M, and Core Space merge updates in shared bases or low-rank subspaces, while SVC studies over-accumulated aligned singular directions.These methods are closest to Pico’s shared-space perspective, but Pico calibrates task-specific B matrices before downstream merging.

7 Conclusion

The paper identifies LoRA merge interference as concentrated in shared output-side B directions and proposes calibrating B before merging. Pico improves existing merging methods across domains and produces a stronger multi-domain adapter than joint multi-task LoRA in this setting.

  • Merge interference is concentrated in a few repeatedly shared B components, while A remains more task-specific.
  • Pico calibrates only B before merging, integrates with existing merge rules, and consistently improves strong baselines across four domains and three downstream mergers.
  • Pico yields a stronger multi-domain adapter than a jointly trained multi-task LoRA in the reported setting.
  • The results suggest diagnosing interference inside the LoRA factorization rather than only at the full-update level.

A Additional Motivation Analysis

The motivation analysis explains why shared directions can dominate linear adapter merges and reports evidence that output-side B overlap is consistently greater than input-side A overlap. It also defines the overlap and concentration diagnostics used to quantify this asymmetry.

  • Toy derivation: In a linear average, a shared direction retains coefficient a, whereas each task-specific direction is reduced to b/T.
  • Toy derivation: The shared-to-specific ratio grows by a factor of T, motivating Pico’s magnitude rescaling after merge.
  • Toy derivation: The derivation shows why widely reused directions can dominate a linear merge without implying that every shared direction is harmful.
  • Diagnostics: Overlap scores measure shared directions in B’s column spaces and A’s row spaces, with larger values indicating greater subspace overlap.
  • Diagnostics: Lower effective rank indicates concentration of singular-value mass in fewer directions, while component energy measures one component’s fraction of total squared magnitude.
  • Empirical evidence: The gap between OB and OA grows with rank, B’s effective rank remains lower than A’s, and the asymmetry becomes universal from rank 16 onward.
  • Empirical evidence: At rank 64, every domain pair has OB > OA in both query and value projection matrices, so the pattern is not driven by one unusually similar task pair.

C Benchmark-Level Results

Benchmark-level results show that Pico’s gains extend across multiple benchmarks and domains rather than arising from one dataset. The best merged adapter achieves the highest overall average among the compared adapters while remaining competitive beyond individual specialists’ source domains.

  • Benchmark-level results: Pico improves strong downstream mergers across multiple benchmarks within each domain, with especially clear gains for Task Arithmetic, TIES, and TSV-M.
  • Multi-domain comparison: Pico + Task Arithmetic gives the highest overall average among the compared adapters while remaining competitive on several individual out-of-domain benchmarks.
  • Spectral diagnostics: The diagnostics assess whether merged B energy is concentrated in a few over-shared components or distributed more evenly.
  • Spectral diagnostics: Lower omax, higher effective rank, and higher stable rank indicate less spectral concentration, whereas Frobenius norm serves only as a reference magnitude.

E Additional Rank-Robustness Results

Rank-sweep experiments show that Pico’s advantage persists across LoRA ranks 8, 16, 32, and 64 for Task Arithmetic, TIES, and TSV-M. The benefit is especially clear for TSV-M at higher ranks.

  • Pico remains the strongest method for Task Arithmetic, TIES, and TSV-M at ranks 8, 16, 32, and 64.
  • At higher ranks, uncalibrated TSV-M drops sharply while Pico stays clearly stronger throughout the rank sweep.

F Additional Results on Qwen3-4B-Base

Appendix experiments show that Pico remains strongest across backbone transfer, LoRA ranks, and progressive multi-adapter merging. As the merge grows, coding performance can fluctuate, while Pico’s calibration makes it more stable.

  • Transfer to Qwen3-4B-Base: Pico again achieves the best overall average after transferring the comparison to Qwen3-4B-Base.The same eight-benchmark setup is used, with especially large gains in math and medical while coding remains strong.
  • Robustness across LoRA ranks: Pico remains the strongest method across downstream mergers at LoRA ranks 8, 16, 32, and 64.Each entry in the robustness table reports the overall average across all eight benchmarks.
  • Progressive LoRA merging: Coding performance is not monotonic as independently trained adapters are added, because a new adapter can either hurt or help transfer.Baselines can improve when the merge grows from three adapters to four instead of declining at every step.
  • Progressive LoRA merging: Pico remains strongest at every progressive merge stage as the pool grows from two to six adapters.The protocol starts from coding and adds finance, medical, math, and law adapters one at a time while tracking coding average.
  • Progressive LoRA merging: Pico’s stability advantage is attributed to calibrating shared B directions before merging, which reduces repeated counting across tasks.Baseline methods become less stable as more independently trained updates are merged.
Loading 2604.16826v1…