PaperlayerView exact revision on arXiv

Source-linked AI summary

CRLoMTL: Conflict-Resolution through Low-rank matrices for Multi-Task Learning

Ahmed Agiza, Tasneem Shaffee, Rahil Badkul, Sherief Reda

arXiv:2610.36337v1cs.CV

TL;DR

Multi-task learning can suffer negative transfer when task updates interfere, and common remedies often require task-wise gradient manipulation or extra optimization steps. CRLoMTL learns task-specific low-rank updates and fuses them through a shared SVD-derived subspace. Across three benchmarks, it achieves a favorable accuracy–parameter trade-off without inference-time overhead.

  • Problem

    Negative transfer in MTL can arise when task updates interfere, while common mitigation methods often require task-wise gradient computation or additional optimization mechanisms.

    Source: §1 · p. 1

  • Method

    CRLoMTL alternates task-specific low-rank adaptation with global fusion through a shared subspace, producing one update that can be merged into the backbone.

    Source: §1 · p. 2

  • Results

    Across three MTL benchmarks, CRLoMTL achieves a favorable accuracy–parameter trade-off; on NYUD-v2 it improves the single-task baseline by 5.03% with 2.22M trainable parameters.

    Source: §4.4 · p. 10, §5 · p. 11

  • Takeaways & Limitations

    CRLoMTL provides a favorable accuracy–parameter trade-off across three MTL benchmarks without inference-time overhead.

    Source: §5 · p. 11

Abstract

from arXiv · show

Multi-task learning (MTL) improves model efficiency by sharing representations across related vision tasks, but it often suffers from negative transfer when task updates interfere with one another. Existing conflict-mitigation methods typically operate in gradient space, requiring explicit task-gradient manipulation and additional optimization steps. We propose Conflict-Resolution through Low-Rank Matrices for Multi-Task Learning (CRLoMTL), a parameter-efficient framework that mitigates task interference in the low-rank parameter-update space. CRLoMTL learns task-specific LoRA updates through task-specific passes, where each task updates its own low-rank matrices, and integrates these updates through global shared-subspace fusion. Specifically, we introduce a shared-subspace resolver based on singular value decomposition (SVD), referred to as the SVD resolver, which extracts a common low-dimensional orthonormal basis across task adapters and projects task updates into this shared space before merging them into a single deployment-time update. The resulting update can be folded into the backbone, preserving the inference structure of the original model without additional runtime branches. Experiments with Swin Transformer backbones on PASCAL-Context, NYUD-v2, and CelebA show that CRLoMTL achieves a favorable accuracy-parameter trade-off, improving over single-task baselines by up to 2.77% on PASCAL-Context, 5.03% on NYUD-v2, and 3.50% on CelebA, while training only a small fraction of the parameters used by single-task models.

1 Introduction

Multi-task learning can suffer negative transfer when task updates interfere, while common remedies often require task-wise gradient computation or additional optimization mechanisms. CRLoMTL instead learns task-specific low-rank updates and fuses them into a single deployment update.

Motivation

  • Negative transfer arises when optimizing one task degrades another, and existing mitigation often requires task-wise gradient computation, optimization subproblems, or extra training-time mechanisms.The cited approaches include loss re-weighting, gradient projection or surgery, and adaptive multi-objective optimization.Source: §1 · p. 1

Contributions

  • CRLoMTL alternates task-specific and global passes, learning separate low-rank adapters before fusing them through a common low-rank subspace.The fused update can be merged into the backbone for inference without task-specific adapter selection or runtime fusion.Source: §1 · p. 2
  • CRLoMTL improves overall multi-task performance while using substantially fewer trainable parameters than existing MTL baselines.The introduction also reports practical training-time and memory overhead.Source: §1 · p. 2

2 Background & Related Work

The background frames MTL as sharing a model across related tasks, while noting that negative transfer and gradient conflict complicate joint optimization. It reviews gradient-, architecture-, and parameter-efficient approaches before formalizing the shared-model objective.

2.1 Multi-task Learning & Challenges

  • MTL shares representations and parameters across tasks, with the stated benefits of cross-task knowledge transfer, lower separate-model training costs, and reduced memory use and FLOPs.The paper formulates a shared model that learns multiple tasks using task-specific losses.Source: §2.1 · p. 2, §2.5 · p. 4
  • Negative transfer can degrade one or more tasks; conflicting gradients can slow convergence, destabilize training, and degrade task performance.The background also notes that MTL performance depends on choices about which layers to share and where to add task-specific components.Source: §2.1 · p. 2

2.2 Gradient-Based Methodologies

  • Gradient-based methods address conflict or task imbalance through gradient modification, objective weighting, multi-objective optimization, or balancing mechanisms, often adding computational cost.The reviewed methods include PCGrad, CAGrad, MGDA, GradNorm, DWA, IMTL-G, FAMO, Nash-MTL, FairGrad, Go4Align, and ConsMTL.Source: §2.2 · p. 2

2.3 Architectural Approaches

  • Architectural MTL approaches enable cross-task interactions or adaptive sharing, but many use additional parameters, task-specific branches, or inference-time modules.Figure 1 depicts CRLoMTL alternating task-specific passes with global passes that fuse task-specific LoRA updates into a shared adapter.Source: §2.3 · p. 3, Figure 1 · p. 3

2.4 Parameter-Efficient Fine-Tuning (PEFT), Low-Rank Adaptation (LoRA), and Model Merging

  • Unlike post-hoc merging of independently fine-tuned models, CRLoMTL repeatedly fuses adapters during training for heterogeneous dense MTL, with the global loss backpropagating to task adapters.The section also introduces LoRA as low-rank matrices added to pretrained weights, with only those matrices updated during fine-tuning.Source: §2.4 · pp. 3–4

2.5 Problem Formulation

  • The problem formulation defines a shared model for T tasks, each with its own loss, and seeks parameters that perform well across tasks despite potentially conflicting gradients.The stated framework uses task-specific low-rank matrices to adapt parameters and find a shared representation.Source: §2.5 · p. 4

3 Methodology

CRLoMTL keeps the backbone frozen while learning low-rank task adaptations, then alternates independent task updates with global shared-subspace fusion. Its SVD resolver projects task updates into a common basis to form a fused update that can be folded into the backbone.

3 Methodology

  • CRLoMTL operates through low-rank parameter adaptations and alternates task-specific updates with global shared-representation learning to capture task-specific features and shared knowledge.The framework combines adaptive low-rank decomposition, task-specific passes, and global shared-subspace fusion.Source: §3 · p. 4

3.1 Low-Rank Task Adaptation

  • Task-specific low-rank matrices adapt selected frozen-backbone layers, reducing trainable parameters while retaining expressivity for task specialization.Each task has its own A and B matrices, with rank r much smaller than the layer dimensions.Source: §3.1 · p. 4

3.2 Multi-Pass Training with Shared-Subspace Fusion

  • Each mini-batch performs T independent task-specific adapter updates with gradients cleared between tasks, followed by one global fused update.The global loss backpropagates through the resolver to persistent task adapters, which are neither overwritten nor reset.Source: §3.2 · p. 5

3.2.1 Task-Specific Passes

  • During a task-specific pass, only that task’s LoRA parameters are updated; the backbone and other tasks’ adapters remain unchanged, reducing direct gradient interference.Other tasks’ adapters are not used in the current task’s forward pass, and gradients are cleared between passes.Source: §3.2.1 · p. 5

3.2.2 Global Passes

  • The global pass consolidates specialized directions through a common low-dimensional subspace, reducing the influence of residual task-specific or conflicting components.After training, the update is folded into the backbone so deployment uses one merged weight matrix without adapter branches or runtime fusion.Source: §3.2.2 · p. 5

3.3 SVD-Based Shared-Subspace Resolver

  • The SVD resolver extracts a shared subspace from task adapters and projects each update onto it using least squares before aggregation into one fused low-rank update.It uses the task-specific A matrices to define the shared input-side subspace while retaining task-dependent B mappings.Source: §3.3 · pp. 5–6

4 Experimental Results

The experiments evaluate CRLoMTL on dense-prediction and many-task classification benchmarks against representative MTL approaches. On PASCAL-Context, it improves over single-task baselines while training a small fraction of their parameters; analyses examine update alignment, SVD conflict suppression, and efficiency.

4.1 Setup

  • CRLoMTL is evaluated with Swin Transformer backbones on PASCAL-Context and NYUD-v2 dense-prediction tasks, and on CelebA as 40 binary classification tasks.The dense-prediction setup uses task-specific decoders; the reported LoRA rank is 8 with scaling α/r = 2.Source: §4.1 · pp. 6–7

4.3 Evaluation Results on PASCAL-Context

  • 2.77% improvement over the single-task baseline with 4% of its trainable parameters on Swin-Tiny, and 6.08% improvement with less than 6% on Swin-Small.The reported parameter counts refer to trainable parameters during training.Source: Table 1 · p. 7, Table 2 · p. 7, §4.3 · p. 8

4.2 Baseline Methods

  • CRLoMTL is compared with methods spanning architectural, parameter-efficient, loss-weighting, gradient-balancing, and parameter-space-merging approaches under the same backbone and datasets.Shared-LoRA and a KnOTS-style SVD merging control are included to isolate recurrent training-time fusion.Source: §4.2 · pp. 7–8

4.3.1 Analysis of Global Pass

  • The global pass increases both average and worst-case task-update alignment, while the no-global-pass ablation remains near zero or slightly negative.Alignment is measured using the mean and minimum off-diagonal entries of pairwise cosine similarities.Source: §4.3.1 · p. 8, Figure 2 · p. 9

4.3.2 Analysis of SVD Conflict Suppression

  • The retained SVD component has higher cross-task alignment and lower negative-pair rate than the original updates and residuals, and the fused update aligns more with the retained component.The residual contains most conflict-prone directions, while the fused update follows the shared SVD-retained subspace.Source: §4.3.2 · pp. 8–9, Figure 3 · p. 9

4.3.3 Efficiency Analysis

  • 14.35 GB memory and 2.79 min/epoch are reported for CRLoMTL; merging reduces parameters from 31.91M to 29.47M with comparable evaluation time.Most training overhead comes from the alternating multi-pass procedure; folded adapters add no inference-time branches or runtime fusion.Source: §4.3.3 · pp. 9–10, Figure 4 · p. 10, Table 3 · p. 10

4.4 Evaluation Results on NYUD-v2

CRLoMTL improves NYUD-v2 over the single-task baseline with a small trainable parameter count, and achieves the best overall CelebA result. Ablations report gains from shared-subspace fusion and limited sensitivity to the tested ranks and LoRA hyperparameters.

4.4 Evaluation Results on NYUD-v2

  • 5.03% improvement over the single-task baseline on NYUD-v2 uses 2.22M trainable parameters, 3.96% of the single-task and 7.79% of the standard MTL parameter counts.The evaluation uses mIoU for semantic segmentation and RMSE for depth estimation.Source: §4.4 · p. 10

4.5 Additional Evaluation on CelebA

  • Using 40 per-attribute adapters changes ∆m from +3.50% to +4.31%, while SVD takes 8.18% of epoch time, versus 45.43% for task-specific passes and 34.62% for the global backward.The 40-adapter setting increases task passes and resolver inputs.Source: §4.5 · p. 11
  • CRLoMTL achieves the best overall result on CelebA in Table 5.The table caption identifies its scope as CelebA results.Source: Table 5 · p. 10, §4.5 · p. 11

4.6 Ablation Study

  • Across tested shared ranks, ∆m varies by only 0.006 percentage points, supporting rs = r/2 as a compact default.The tested ranks are rs = r/4, r/2, and r.Source: §4.6 · p. 11
  • On PASCAL-Context, ∆m varies little across tested LoRA ranks and nonzero α values, indicating limited sensitivity to these hyperparameters.The tested settings use α = 2r for rank variation and fixed r = 8 for α variation.Source: §4.6 · p. 11
  • The SVD resolver reaches +2.77% ∆m, compared with +2.50% for average fusion and -0.60% for Shared-LoRA.The comparison isolates task-specific adaptation and shared-subspace fusion.Source: §4.6 · p. 11

5 Conclusion

CRLoMTL combines task-specific LoRA adaptation with shared-subspace fusion. Across three MTL benchmarks, it achieves a favorable accuracy–parameter trade-off without inference-time overhead.

5 Conclusion

  • Across three MTL benchmarks, CRLoMTL combines task-specific LoRA adaptation and shared-subspace fusion for a favorable accuracy–parameter trade-off without inference-time overhead.Future work will explore larger task settings, other PEFT modules, and shared-subspace dynamics.Source: §5 · p. 11

A Additional SVD Resolver Analysis

The retained SVD component captures the portion of a task update represented by the shared basis, not necessarily most of its energy. The residual can contain more energy while being less compatible across tasks.

A Additional SVD Resolver Analysis

  • The residual may hold more Frobenius energy than the retained component, yet be less compatible across tasks; the retained component captures the update represented by the shared SVD basis.Figure 5 shows retained and residual energy over training.Source: A Additional SVD Resolver Analysis · p. 15, Figure 5 · p. 15

B Additional Feature-Space Analysis

On PASCAL-Context, CRLoMTL and vanilla MTL both produce separable task features, but CRLoMTL's clusters are more compact. Its higher silhouette score supports this difference in cluster compactness and separation.

B Additional Feature-Space Analysis

  • The analysis uses t-SNE projections and silhouette scores to compare task-specific feature separation and intra-task cohesion beyond scalar metrics.The silhouette coefficient measures assignment to a cluster relative to other clusters.Source: B Additional Feature-Space Analysis · p. 15
  • After 175 epochs on PASCAL-Context, CRLoMTL produces more compact task clusters and a higher silhouette score than vanilla MTL.Both methods show separable task features, while MTL shows elongated clusters.Source: B Additional Feature-Space Analysis · p. 15

C Additional Ablation and Stability Analysis

Across three random seeds, CRLoMTL shows stable performance gains on PASCAL-Context, both overall and across the reported tasks.

C Additional Ablation and Stability Analysis

D CelebA Attribute Grouping

CelebA’s 40 attributes are manually grouped by semantic similarity, with the same seven-group assignment used across runs.

D CelebA Attribute Grouping

Related papers

Loading 2610.36337v1…