PaperlayerView exact revision on arXiv

Source-linked AI summary

SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, Ying Wei

arXiv:2501.13198v3cs.LG

TL;DR

Foundation-model continual learning needs to scale without retaining prior-task samples or expanding task-specific components. SD-LoRA decouples magnitude and direction learning in incrementally added LoRA components, and experiments report consistent effectiveness across benchmarks, task lengths, and backbones. The method also provides direct final-model inference and two more parameter-efficient variants.

  • Problem

    Existing prompt- and LoRA-based continual-learning methods often expand component pools or retain previous-task samples, creating scalability challenges as task numbers grow.

  • Method

    SD-LoRA incrementally adds LoRA components while separating magnitude and direction learning, with all parameters end-to-end optimized for continual-learning objectives.

  • Results

    SD-LoRA consistently outperforms existing methods across multiple continual-learning benchmarks and remains effective across varied task lengths and foundation-model backbones.

  • Takeaways & Limitations

    SD-LoRA supports rehearsal-free learning and efficient inference by evaluating the finally trained model directly without task-specific component selection.

  • Takeaways & Limitations

    The paper identifies extending SD-LoRA beyond ViTs and developing more theoretically grounded rank-reduction and distillation strategies as future research directions.

Abstract

from arXiv · show

Continual Learning (CL) with foundation models has recently emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. However, existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. To address these limitations, we propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends to follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting in an excellent stability-plasticity trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAs can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component selection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code is available at https://github.com/WuYichen-97/SD-Lora-CL.

1 INTRODUCTION

Foundation-model continual learning must retain prior knowledge while adapting to sequential tasks, but existing prompt- and LoRA-based approaches face scalability limits from growing component pools or rehearsal memory. SD-LoRA addresses these constraints with rehearsal-free, efficient, end-to-end adaptation.

  • Motivation: Continual learning trains on sequential non-stationary tasks while seeking to retain previously acquired knowledge.This setting differs from standard supervised learning, which assumes i.i.d. data, and introduces catastrophic forgetting as a central challenge.
  • Existing Approaches: Prompt-based methods incrementally learn prompt pools and selectively insert prompts to accommodate new tasks.Foundation models support knowledge transfer and resistance to catastrophic forgetting in continual learning.
  • Scalability Gap: Existing methods can require large-scale rehearsal memory, making them less scalable as task counts and deployment constraints grow.HiDe-Prompt stores many previous-task samples, while InfLoRA rehearses extensive samples during incremental LoRA optimization.
  • Design Requirements: An ideal foundation-model continual-learning method should be rehearsal-free, inference-efficient, and end-to-end optimized.These properties respectively avoid previous-task sample storage, additional inference computation, and segmented parameter optimization.
  • Proposed Method: SD-LoRA separates magnitude and direction learning while incrementally adding LoRA components, enabling direct final-model evaluation without task-specific component selection.Its analysis identifies a low-loss trajectory converging to an overlapping low-loss region across learned tasks, and two variants improve parameter efficiency through rank reduction and knowledge distillation.
  • Contributions: The paper presents SD-LoRA and two variants as rehearsal-free, inference-efficient, end-to-end optimized solutions, supported by empirical and theoretical analysis and evaluations across benchmarks and foundation models.The contributions also analyze why task-specific component selection can be eliminated.

2 RELATED WORK

Related continual-learning methods are commonly organized as rehearsal-based, regularization-based, or architecture-based approaches. Foundation-model continual learning increasingly combines knowledge transfer with prompts and parameter-efficient fine-tuning to reduce forgetting and resource demands.

  • Continual Learning: Rehearsal-based continual-learning methods retain and replay previous-task samples to alleviate catastrophic forgetting.This is one of three broad design philosophies for continual learning.
  • Continual Learning: Regularization-based methods constrain updates to parameters considered important for previously learned tasks through penalty terms.The constraint is imposed in the training objective.
  • Continual Learning: Architecture-based methods expand or adapt the model architecture to support continual learning.They represent the third broad design philosophy alongside rehearsal and regularization.
  • CL with Foundation Models: Foundation-model continual-learning methods use knowledge transfer and prompt tuning with ViTs to improve retention as new tasks are introduced.Examples include L2P, DualPrompt, and CODA-Prompt.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning is important because full fine-tuning for each task is prohibitive in computation and storage.Representative approaches include adapters, prompt-tuning, and prefix-tuning.

3 PROPOSED METHOD: SD-LORA

SD-LoRA incrementally decouples LoRA magnitude and direction learning for rehearsal-free, inference-efficient continual learning, preserving earlier directions while learning new ones. Its analysis links this design to low-loss trajectories, overlapping low-loss regions across tasks, and parameter-efficient variants.

  • Problem formulation: The class-incremental objective evaluates the current model across the current and all previously encountered tasks without access to previous-task training data.The formulation uses sequential tasks and an average test loss over tasks encountered so far.
  • SD-LoRA method: SD-LoRA incrementally separates LoRA magnitude and direction learning, fixing previously learned directions while optimizing the current component for sequential tasks.The layer output combines scaled LoRA components with the frozen original weight and previously learned directions.
  • Empirical motivation: Task-specific fine-tuned weights cluster more closely with one another than with the original foundation-model weights, supporting reuse of shared adaptation directions.This relationship is reported across five ImageNet-R tasks and remains consistent across benchmarks, backbones, and task lengths.
  • Empirical findings: Optimizing only LoRA magnitudes after the first task consistently surpasses vanilla LoRA in average accuracy, indicating that fixed directions can support continual learning.The experiment fixes the direction after the first task and continually optimizes magnitude.
  • Empirical findings: Earlier directions are increasingly reused through rising magnitudes, while later directions generally decline and provide minor task-specific refinements.The newly learned direction initially aligns strongly with earlier directions before diverging to add subtle variations.
  • Mechanism: SD-LoRA follows a low-loss path toward an overlapping low-loss region, improving later-task performance without sacrificing earlier-task accuracy.Weight interpolation compares this behavior with vanilla LoRA, where improving the new task comes at the expense of the earlier task.
  • Theory and variants: Theoretical analysis models SD-LoRA as matrix factorization and shows that small-initialization gradient descent sequentially approximates principal components of the optimal update.This explains the observed decline in later learned magnitudes and motivates the parameter-efficient variants SD-LoRA-RR and SD-LoRA-KD.

4 EXPERIMENTS

The experiments evaluate SD-LoRAs across multiple continual-learning benchmarks, task lengths, backbones, and efficiency criteria. SD-LoRA consistently outperforms competing methods, while its parameter-efficient variants retain nearly comparable performance.

  • Experimental setup: SD-LoRAs are evaluated on ImageNet-R, ImageNet-A, and DomainNet under varied sequential task partitions.ImageNet-R is split into 5, 10, and 20 tasks; ImageNet-A into 10 tasks; DomainNet into 5 tasks.
  • Experimental setup: The evaluation reports average accuracy and average anytime accuracy, measuring final overall performance and performance accumulated throughout training.Average accuracy averages performance across all tasks after continual learning; average anytime accuracy accumulates accuracy after each new task.
  • Main results: 7.68% in Acc and 4.62% in AAA are SD-LoRA’s margins over InfLoRA on ImageNet-R with 20 tasks.On ImageNet-A, SD-LoRA exceeds HiDe-Prompt by approximately 31.05% in Acc and 15.32% in AAA.
  • Main results: SD-LoRA remains superior with a self-supervised DINO ViT-B/16 backbone, while SD-LoRA-RR and SD-LoRA-KD show only marginal degradation relative to full SD-LoRA.The variants therefore preserve performance while improving parameter efficiency.
  • Main results: SD-LoRAs maintain superiority as ImageNet-R task counts increase, with performance margins growing across 5, 10, and 20 sequential tasks.This supports their suitability for resource-efficient continual learning under longer task sequences.
  • Ablation and efficiency analysis: Ablations indicate that satisfactory performance requires multiple decoupled LoRA components and magnitude rescaling, not decoupling alone.Fixing learned components without rescaling produces a noticeable performance decline, suggesting rescaling supports low-loss navigation.
  • Ablation and efficiency analysis: SD-LoRA and InfLoRA avoid task-specific prompt selection during inference, while SD-LoRA-RR further reduces LoRA parameters.The comparison considers inference computation, trainable parameters, and feature-storage requirements.

5 CONCLUSION AND DISCUSSION

The paper concludes that SD-LoRA addresses scalability in class-incremental learning through decoupled LoRA magnitude and direction learning. Its low-loss behavior, strong empirical results, and parameter-efficient variants support resource-constrained use, while broader generality remains future work.

  • Conclusion: SD-LoRA decouples LoRA magnitude and direction learning to provide rehearsal-free, inference-efficient, and end-to-end optimized continual learning.The method directly uses the finally trained model and does not require task-specific component selection.
  • Conclusion: SD-LoRA follows a low-loss trajectory toward an overlapping low-loss region across learned tasks, balancing stability and plasticity.The paper links this behavior to mitigation of catastrophic forgetting while retaining adaptability to new tasks.
  • Conclusion: SD-LoRA-RR and SD-LoRA-KD improve parameter efficiency and practicality for resource-constrained applications.These variants build on the proposed SD-LoRA framework.
  • Future directions: Future work includes testing beyond ViTs, combining SD-LoRA with other PEFT techniques, and developing more theoretically grounded rank-reduction and distillation strategies.These directions target broader generality and further parameter-efficiency improvements.

A.1 PROOF OF THEOREM 1

The proof analyzes sequential LoRA training as a matrix-approximation problem and studies convergence by separating dominant and error blocks. It establishes conditions under which the dominant component approaches the target while imbalance and error terms remain controlled.

  • Convergence conditions: Theorem 2 assumes a singular-value separation condition, σj+1 < σj, and constrains initialization and probability parameters for convergence.The theorem provides a high-probability guarantee under these conditions.
  • Proof strategy: The proof treats sequential LoRA training as a matrix-approximation problem for the optimal update matrix of each task.Gradients with respect to the LoRA factors are used to derive gradient-descent updates.
  • Block decomposition: The update is partitioned into dominant and error blocks, with convergence reduced to showing the dominant product approaches the target while error terms remain small.The dominant block is denoted UV, while J and K form the error terms.
  • Block decomposition: The proof introduces F and G through U = F + G and V^⊺ = F − G to analyze potential imbalance between the factor blocks.Subsequent updates for F and G support the convergence analysis.
  • Convergence conditions: The analysis uses a small-neighborhood condition in which task-optimal updates lie near one another in the loss landscape.This supports the existence of a shared low-loss region across sequential tasks.

A.2 ADDITIONAL RESULTS FOR SEC. 3.3

Additional experiments examine learned LoRA magnitudes over longer task sequences and across datasets. Both settings show a systematic decrease in magnitude coefficients, extending the trend observed in the main experiments.

  • Magnitude evolution: The learned LoRA magnitudes αk systematically decrease during training on ImageNet-R with 20 tasks and DomainNet with 5 tasks.These experiments test an extended task horizon and a more challenging multi-domain dataset.
  • Magnitude evolution: The decreasing αk trend is consistent across the extended ImageNet-R and DomainNet configurations.The result corroborates the descending behavior observed in the main experiments.
  • Additional analysis: Relative-distance analysis is additionally reported for DomainNet with 5 tasks and ImageNet-R with 5 and 10 tasks.These settings complement the magnitude-evolution experiments with cross-task distance measurements.

A.3 RESULTS ON OTHER CL BENCHMARKS

SD-LoRA is evaluated on CIFAR-100 and CUB-200 in ten-task class-incremental settings, with consistently outstanding performance reported on both datasets. Additional analyses across benchmarks, backbones, and task lengths show trends consistent with Finding 1.

  • CIFAR-100: CIFAR-100 contains 60,000 images across 100 classes and is split into ten tasks of ten classes each.Each class has 600 images.
  • CUB-200: CUB-200 contains 11,788 bird images across 200 species and is divided into ten tasks of 20 species each.The dataset is designed for fine-grained bird classification.
  • Performance: SD-LoRAs consistently deliver outstanding performance on CIFAR-100 and CUB-200.The comparison is reported in Table 6.
  • Additional analysis: Relative-distance analyses across benchmarks, backbones, and task lengths show trends consistent with Finding 1.These supplementary results are presented in Fig. 7.
Loading 2501.13198v3…