Source-linked AI summary
Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection
Ziyu Yang, Guibin Chen, Yuxin Yang, Aoxiong Zeng, Xiangquan Yang
TL;DR
Multi-task LoRA saves parameters by sharing an adapter, but conflicting gradients and the low-rank bottleneck can degrade task performance. Ortho-LoRA projects gradients orthogonally within LoRA’s adapter space and, on GLUE, approaches single-task performance while substantially improving over joint training.
Problem
Shared LoRA adapters can suffer negative transfer because conflicting task gradients compete within the limited capacity of a low-rank update.
Method
Ortho-LoRA applies orthogonal gradient projection to the low-rank LoRA parameters A and B while keeping the pretrained backbone frozen.
Results
Ortho-LoRA achieves an average score of 89.6, only 0.3 points below the Single-Task upper bound, while nearly matching QQP performance at 87.9 versus 88.1.
Takeaways & Limitations
Ortho-LoRA offers a middle ground between the efficiency of joint training and the performance of single-task fine-tuning using one lightweight shared module.
Abstract
from arXiv · showhide
Multi-Task Learning (MTL) combined with Low-Rank Adaptation (LoRA) has emerged as a promising direction for parameter-efficient deployment of Large Language Models (LLMs). By sharing a single adapter across multiple tasks, one can significantly reduce storage overhead. However, this approach suffers from negative transfer, where conflicting gradient updates from distinct tasks degrade the performance of individual tasks compared to single-task fine-tuning. This problem is exacerbated in LoRA due to the low-rank constraint, which limits the optimization landscape's capacity to accommodate diverse task requirements. In this paper, we propose Ortho-LoRA, a gradient projection method specifically tailored for the bipartite structure of LoRA. Ortho-LoRA dynamically projects conflicting task gradients onto the orthogonal complement of each other within the intrinsic LoRA subspace. Extensive experiments on the GLUE benchmark demonstrate that Ortho-LoRA effectively mitigates task interference, outperforming standard joint training and recovering 95\% of the performance gap between multi-task and single-task baselines with negligible computational overhead.
1 Introduction
Multi-task LoRA reduces deployment overhead by sharing one adapter, but competing task gradients can cause negative transfer, especially under LoRA’s low-rank bottleneck. Ortho-LoRA addresses this by enforcing gradient orthogonality within the adapter space.
- Motivation: Sharing one LoRA adapter across tasks reduces parameter and storage requirements but creates competition among task updates.MTL can promote positive transfer, yet shared parameters may have limited capacity for diverse task requirements.
- Motivation: LoRA’s low-rank constraint increases gradient collisions, so improving one task can come at another task’s expense.The paper describes this as the “bottleneck conflict” problem.
- Method: Ortho-LoRA projects conflicting gradients onto orthogonal planes specifically for LoRA’s low-rank matrices A and B.This controls interference in the adapter parameters while leaving the frozen backbone untouched.
- Results: Ortho-LoRA provides GLUE evidence of performance comparable to independent single-task adapters while using only 1/N of the parameters.Here, N denotes the number of tasks.
2 Related Work
Prior work addresses multi-task LoRA interference through parameter-efficient adaptation, gradient optimization, or architectural routing. Ortho-LoRA instead targets the bipartite structure of LoRA with explicit gradient projection while retaining a single shared module.
- Parameter-Efficient Fine-Tuning: LoRA adapts pretrained models by representing weight changes as the low-rank update ΔW = BA while freezing the backbone.This design reduces the number of parameters updated during adaptation.
- Parameter-Efficient Fine-Tuning: Prior PEFT variants include bottleneck adapters, Prefix-Tuning, and dynamic-rank methods such as AdaLoRA.These methods differ in whether they modify intermediate layers, optimize continuous prompts, or allocate rank dynamically.
- Multi-Task Optimization: MTL optimization methods such as GradNorm and PCGrad address conflicting task gradients through weighting or projection.The related-work discussion motivates tailoring projection methods to LoRA’s specific structure.
- Architectural Solutions: MoE-LoRA approaches reduce interference through task-specific parameters or routing, but can add routers or complex routing logic.Examples include task-specific adapters and FlyLoRA’s implicit rank-wise mixture of experts.
- Architectural Solutions: Ortho-LoRA uses explicit gradient orthogonality to achieve task disentanglement without the architectural overhead of MoE systems.The method preserves the structural simplicity of a single shared LoRA module.
3 Methodology
Ortho-LoRA addresses gradient conflict in multi-task LoRA by projecting conflicting gradients independently within LoRA’s two matrix components. This structure-aware strategy preserves compatible transfer while reducing interference under the low-rank bottleneck.
- Gradient Conflict: Gradient conflict occurs when two task gradients have negative cosine similarity, causing destructive interference and potentially suboptimal individual-task solutions.Standard gradient descent combines these task directions in the shared update.
- Gradient Conflict: LoRA restricts adaptation to a rank-r manifold, increasing the likelihood that task-optimal gradient trajectories conflict.This reduced capacity amplifies negative transfer compared with optimization in a high-dimensional full-fine-tuning space.
- Orthogonal Projection: Ortho-LoRA detects conflicting task gradients and projects one gradient onto the normal plane of the other before updating the parameters.The projection removes the conflicting component and is applied when gi · gj < 0.
- Structure-Aware Decoupling: The method applies projection independently to LoRA matrices A and B because they encode distinct input-projection and output-reconstruction functions.This avoids treating the bipartite adapter structure as one undifferentiated vector.
- Structure-Aware Decoupling: Separate projections can preserve shared input representations when output requirements differ, or preserve output compatibility when feature extraction differs.This granular approach retains transfer information that a global projection might remove.
- Complexity Analysis: Ortho-LoRA computes task gradients separately, randomizes projection order, and operates only on the small LoRA parameter set.Gradient computation scales roughly as O(T), while projection overhead is negligible because LoRA parameters are less than 0.1% of the model for r = 8.
4 Experiments
Experiments evaluate Ortho-LoRA on selected GLUE tasks against independent and jointly trained LoRA baselines. Ortho-LoRA mitigates joint-training degradation, remains effective across ranks, and reaches peak performance earlier despite higher wall-clock cost.
- Experimental Setup: Experiments use MNLI, QQP, and SST-2 from GLUE with RoBERTa-base and LoRA modules in attention Query and Value projections.The LoRA configuration uses rank r = 8, α = 16, and 0.1 dropout.
- Baselines: The comparison includes independent Single-Task LoRA adapters, shared Joint-LoRA trained on summed losses, and shared Ortho-LoRA with orthogonal projection.Single-Task LoRA requires 3× parameters and serves as the performance upper bound.
- Main Results: 89.6 average score places Ortho-LoRA only 0.3 points behind the Single-Task upper bound, versus a 1.5-point average degradation for Joint-LoRA.On QQP, Ortho-LoRA scores 87.9 versus 88.1 for Single-Task LoRA.
- Rank Ablation: Ortho-LoRA provides consistent gains across ranks r ∈ {4, 8, 16, 32}, with the largest gain at r = 4.The strongest gain occurs where the low-rank bottleneck is most severe.
- Convergence: Ortho-LoRA takes 1.4x longer in wall-clock time than Joint-LoRA but reaches peak performance 2 epochs earlier.The method requires T backward passes per step and was observed to stabilize the optimization trajectory.
5 Conclusion
The paper addresses gradient conflict in LoRA-based multi-task learning by projecting gradients orthogonally within adapter parameters. On GLUE, Ortho-LoRA provides a middle ground between joint-training efficiency and single-task performance while retaining a lightweight shared module.
- Method: Ortho-LoRA mitigates negative transfer by applying orthogonal gradient projection within the low-rank adapter space.The method operates on adapter parameters rather than requiring separate task-specific modules.
- Conclusion: GLUE experiments show Ortho-LoRA as a middle ground between the efficiency of joint training and the performance of single-task fine-tuning.The conclusion characterizes the method as producing minimal performance degradation with one shared lightweight module.
- Future Work: Future work will extend the projection approach to other PEFT methods and combine it with dynamic task weighting.The proposed directions include Prefix-Tuning and dynamic task weighting.