Source-linked AI summary
LoRA Superposition across Pretrained Models
Keigo Watanabe, Hiroshi Kera, Kazuhiko Kawamoto
TL;DR
Because LoRA is usually tied to the backbone on which it was trained, it is unclear whether same-task adapters learned on different backbones can be combined directly. The paper sums those adapters and applies the same update unchanged across contributing backbones; the exact sum improves all evaluated pairs and remains close to individually trained adapters.
Problem
Because LoRAs are typically tied to their pretrained backbone, it is unclear whether independently trained same-task updates can be reused across different pretrained backbones.
Source: §1 · p. 1Method
The paper sums same-task solo LoRAs trained on different backbones and applies the shared update unchanged to all contributing backbones.
Source: §1 · pp. 1–2Results
30.21 percentage points mean accuracy gain; the exact sum improves all 56 backbone–task pairs and averages 0.28 percentage points below Solo.
Source: §1 · p. 2, §4.2 · p. 7Takeaways & Limitations
The results support backbone-specific adaptations remaining functionally separable after addition, with the frozen backbone acting as an implicit selector.
Source: §6 · p. 10The study considers same-architecture ViT-B/16 backbones, and whether weak interactions accumulate as the number and diversity of backbones grow remains unresolved.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Low-rank adaptation (LoRA) is usually tied to the pretrained backbone on which it is learned. We ask whether LoRAs trained independently for the same task on different pretrained backbones can be summed and applied unchanged to all contributing backbones. We call this construction LoRA superposition. Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, the exact sum improves all 56 backbone-task pairs and achieves a mean accuracy gain of 30.21 percentage points over the unadapted backbone. The exact sum achieves a mean accuracy only 0.28 percentage points below that of individually trained solo LoRAs. In contrast, a leave-one-model-out (LOMO) sum produces a mean gain of only 0.06 percentage points over the unadapted backbones. Backbone-specific solo LoRAs are nearly orthogonal in parameter space, while pairwise cross-backbone responses are strongly concentrated on the diagonal. These results support a component-wise non-interference hypothesis, under which the frozen backbone acts as an implicit selector of its own component without explicit routing or cross-model alignment.
1 Introduction
LoRA is usually tied to its pretrained backbone, so supporting multiple tasks across multiple backbones typically requires separate adapters. This section introduces LoRA superposition: summing same-task, backbone-specific adapters and applying the shared update unchanged, with component-wise non-interference as the central hypothesis.
1 Introduction
- LoRA superposition sums same-task solo LoRAs trained on different pretrained weights and applies the resulting shared update unchanged to every contributing backbone.The shared adapter requires no additional optimization, calibration data, model identifier, or target-specific transformation after the solo LoRAs are trained.Source: §1 · pp. 1–2
- The central hypothesis is that nearly orthogonal solo LoRAs interfere little because each frozen backbone responds mainly to its own component, acting as an implicit selector.The hypothesis predicts that the full sum retains target solo-LoRA performance, while a sum excluding that backbone’s component has little effect.Source: §1 · p. 2
- Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, exact sum improves all 56 backbone–task pairs, gaining 30.21 percentage points in mean accuracy; LOMO sum gains 0.06 points over Backbone.Source: §1 · p. 2
- The study characterizes non-interference using parameter-space similarity, pairwise cross-backbone response, and target-excluded LOMO analysis across seven backbones and eight tasks.The reported geometric and response analyses support the component-wise non-interference hypothesis.Source: §1 · p. 2
2 Related Work
Related work combines or transfers parameter updates in several ways, often relying on a shared backbone, explicit alignment, or a selection mechanism. LoRA superposition instead asks whether same-task updates from different backbones can be directly summed and used unchanged.
2.1 Task Vectors and LoRA Composition
- Task-vector and LoRA-library methods combine or select reusable updates, but generally define them relative to a common pretrained backbone.Task-vector merging methods include TIES-Merging and DARE, while LoRAHub and Arrow compose or select adapters.Source: §2.1 · pp. 2–3
2.2 Cross-Model Adaptation
- Cross-model adaptation transports updates through projection, re-basing, or orthogonal alignment, typically for a particular source–target relationship.These approaches address mismatches between parameter spaces or representations of independently pretrained models.Source: §2.2 · p. 3
2.3 Model and Parameter Superposition
- Parameter-superposition methods store multiple configurations but use contexts, masks, or task-selection mechanisms to retrieve the desired component.LoRA superposition instead hypothesizes that the frozen backbone responds selectively to its own component.Source: §2.3 · p. 3
2.4 LoRA Merging
- LoRA merging methods address interference or incompatibility among updates generally defined for a common backbone, whereas superposition sums same-task updates trained on different backbones.The summed update is applied unchanged without parameter alignment, context vectors, or explicit routing.Source: §2.4 · p. 3
3 LoRA Superposition
For a fixed task, LoRA superposition sums independently trained, backbone-specific updates for same-architecture backbones and applies the shared update unchanged to each. The non-interference hypothesis is that each frozen backbone effectively retains its own component while other components have little effect.
3.1 Problem Formulation
- Each backbone in a fixed same-architecture set receives its own solo LoRA trained for the same downstream task, while all backbone parameters remain frozen.The pretrained weights may differ across backbones.Source: §3.1 · p. 3
- A solo LoRA update is the effective matrix product Bm,t,jAm,t,j, and the complete adapter is represented by vectorizing and concatenating its module updates.Using effective updates rather than LoRA factors avoids factorization non-uniqueness.Source: §3.1 · p. 4
- LoRA superposition sums the solo-LoRA vectors, equivalently summing corresponding module updates into one shared adapter.The saved matrices have the dimensions of their corresponding backbone weights, and their rank can reach Kr, or 112 when K = 7 and r = 16.Source: §3.1 · p. 4
- The same summed update is applied unchanged to every contributing backbone, with no additional data, optimization, model identifier, or target-specific transformation after training.A same-architecture backbone outside the contributing set serves as a target-excluded control.Source: §3.1 · p. 4
3.2 Non-Interference Hypothesis
- The non-interference hypothesis attributes separability to nearly orthogonal solo LoRAs and each frozen backbone's sensitivity to its own useful update directions.The projection is a conceptual interpretation of backbone responses, not an operation explicitly computed by the method.Source: §3.2 · p. 4, Figure 2 · p. 5
- The hypothesis predicts weak effects from another backbone's solo LoRA and preservation of the target backbone's solo accuracy under the sum; the target-excluded case tests for model-invariant directions.For included backbones, the sum without that backbone's component is also predicted to leave accuracy near the unadapted level.Source: §3.2 · p. 5
4 Experiments
Across seven ViT-B/16 CLIP backbones and eight image-classification tasks, the experiments compare post-hoc LoRA superposition with individual and jointly trained adapters. They also examine whether backbone-specific updates are geometrically and functionally non-interfering.
4.1 Setup
- The evaluation uses seven ViT-B/16 CLIP backbones, eight classification datasets, frozen backbone parameters, and rank-16 LoRA updates to Q and V projections.Joint shared trains one common LoRA with equal weighting across the seven backbones; Exact sum and LOMO sum require no additional optimization after solo training.Source: §4.1 · pp. 6–7
4.2 Main Results
- Exact sum improves all 56 backbone–task pairs over Backbone, averages 0.28 percentage points below Solo, and reaches 90.06% accuracy versus 89.34% for Joint shared.The Exact sum–Joint shared comparison is not rank-matched: their per-module ranks are at most 112 and 16, respectively.Source: §4.2 · pp. 7–8, Table 2 · p. 7
4.3 Geometric Relation of Solo LoRAs
- Cross-backbone solo LoRAs have mean absolute cosine similarity 1.34 × 10−3, versus 0.424 for same-backbone, cross-seed solo LoRAs.The paper notes that low cosine similarity alone does not establish functional independence.Source: §4.3 · p. 8
4.4 Pairwise Cross-Backbone Response
- Pairwise cross-backbone accuracy gains averaged over tasks are interpreted through their diagonal pattern: diagonal responses indicate stronger effects on the backbone used to train each solo LoRA.Small off-diagonal responses support pairwise non-interference.Source: §4.4 · p. 8
5 Discussion and Limitations
The discussion supports component-wise coexistence but leaves its mechanism and generality unresolved. It also notes a practical trade-off: a shared adapter avoids backbone-specific selection, while its rank may grow and compression remains unevaluated.
5 Discussion and Limitations
- Figure 4’s pairwise similarities and Figure 5’s cross-backbone responses support coexistence, but near-orthogonality alone does not explain the behavior.Small off-diagonal responses may reflect both backbone-specific sensitivity and the separate training of solo LoRAs.Source: §5 · p. 8, Figure 4 · p. 9, Figure 5 · p. 10
- Each backbone’s effective update subspace is uncharacterized, and near-orthogonality may depend on the particular backbones, tasks, and training conditions.The implicit-projection interpretation describes the observed behavior but does not explicitly characterize those subspaces.Source: §5 · p. 9
- Weak interactions may accumulate as the backbone set grows; the study uses same-architecture ViT-B/16 backbones with directly compatible LoRA modules.Whether analogous effective subspaces can be identified across architectures remains unresolved.Source: §5 · p. 9
- A shared adapter removes backbone-specific selection after training, but its rank can reach 112 in this setting and rank-16 compression remains unevaluated.Whether compression preserves adaptation performance across all contributing backbones remains to be evaluated.Source: §5 · p. 9
6 Conclusion
The conclusion supports summing same-task, backbone-specific LoRAs and applying the shared update unchanged to a closed set of pretrained models. It presents this as a route to modular adapter reuse, while leaving the mechanism and broader scope open.
6 Conclusion
- The results support summing same-task, backbone-specific LoRAs and applying the update unchanged to a closed set of pretrained models.The conclusion describes this construction as LoRA superposition.Source: §6 · p. 10
- Near-orthogonal updates, pairwise cross-backbone responses, and target-excluded results together suggest adaptations can coexist without explicit routing or cross-model alignment.Source: §6 · p. 10
- The findings suggest modular reuse of adapter libraries across pretrained models without joint retraining, while the mechanism and extension beyond identical architectures remain open.Source: §6 · p. 10