PaperlayerView exact revision on arXiv

Source-linked AI summary

KeepLoRA: Continual Learning with Residual Gradient Adaptation

Mao-Lin Luo, Zi-Hao Zhou, Yi-Lin Zhang, Yuanyu Wan, Tong Wei, Min-Ling Zhang

arXiv:2601.19659v1cs.CVcs.LG

TL;DR

Continual learning for pretrained vision-language models must preserve general knowledge and prior tasks while acquiring new knowledge. KeepLoRA analyzes parameter subspaces and projects gradient-based low-rank updates away from pretrained and prior-task directions. It achieves state-of-the-art performance while balancing plasticity, backward stability, and forward stability.

  • Problem

    Continual learning must balance plasticity, backward stability, and preservation of pretrained knowledge, while replaying pretraining data is often impractical.

  • Method

    KeepLoRA uses frozen-down-projection LoRA updates initialized from the current gradient and constrained orthogonally to the pretrained principal subspace and prior-task feature directions.

  • Results

    KeepLoRA achieves state-of-the-art Transfer, Average, and Last performance across evaluated CLIP and LLaVA continual-learning settings.

  • Takeaways & Limitations

    Residual-subspace updates preserve pretrained and previously learned knowledge while retaining learning capacity comparable to unconstrained LoRA on isolated tasks.

  • Takeaways & Limitations

    The unified subspace assumes that pretrained principal directions and dominant continual-learning feature directions represent compressed knowledge from their respective data distributions.

Abstract

from arXiv · show

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

1 INTRODUCTION

KeepLoRA addresses continual learning’s need to preserve general and prior-task knowledge while retaining plasticity. It identifies distinct parameter subspaces for general and domain-specific knowledge, then constrains low-rank updates to protect transferable capabilities.

  • Continual learning for vision-language models must balance plasticity, backward stability, and forward stability.
  • Reference-data regularization and architecture extension preserve stability but introduce data sensitivity, training costs, or added parameters.
  • Principal parameter components primarily encode general knowledge, whereas residual components capture domain-specific adaptations.Removing low-energy components leaves general-domain performance robust but sharply degrades performance on most specific-domain datasets.
  • KeepLoRA uses low-rank updates that are initialized from the first-step gradient and projected orthogonally to the pretrained principal subspace and prior task directions.The update is restricted to residual directions to reduce interference with transferable knowledge while improving adaptation.
  • Experiments and theory show that KeepLoRA balances plasticity, backward stability, and forward stability across CLIP and LLaVA benchmarks.The method is reported to achieve state-of-the-art results on benchmark datasets.

2 RELATED WORKS

Prior continual-learning methods preserve stability through reference data, architecture expansion, or gradient constraints. KeepLoRA belongs to the gradient-projection line while targeting both pretrained transferability and task retention.

  • Reference-Data Regularization: Reference-data regularization anchors representations to preserve pretrained zero-shot transfer capabilities.These methods rely on datasets such as ImageNet and Conceptual Captions for distillation.
  • Architecture Extension: Architecture-extension methods freeze the pretrained model and add task-specific parameters such as prompts or adapters.Examples include prompt pools, task-sharing prompts, and mixtures of adapters.
  • Gradient Projection: Gradient-projection methods constrain new updates to avoid interference with previously acquired knowledge.GPM uses a stored basis of principal gradient directions, while CoSo estimates task-specific subspaces.
  • Gradient Projection: Parameter-efficient methods extend gradient projection by enforcing orthogonality among LoRA subspaces or task-feature directions.O-LoRA uses previous-task LoRA subspaces, while InfLoRA constrains task features against prior dominant directions.

3 METHOD

KeepLoRA adapts new tasks with LoRA updates confined to a residual subspace that excludes principal pre-trained directions and dominant directions from previous tasks. Gradient-informed initialization preserves adaptation to the current task while the constrained updates support stability.

  • Stability: KeepLoRA unifies pre-trained and previous-task directions into a principal subspace, then confines new adaptations to its orthogonal residual subspace.The unified subspace is formed from Wp and Mt−1.
  • Stability: SVD decomposes backbone weights so large-singular-value components represent general knowledge, while residual components capture domain-specific knowledge.Specialized-dataset performance is described as sensitive to residual-subspace alterations.
  • Stability: Previous-task feature directions are stored as dominant singular vectors and accumulated in Mt−1, with the number retained determined by an energy threshold.After each task, additional dominant directions are appended to the subspace matrix.
  • Stability: KeepLoRA initializes A within the subspace orthogonal to Wp and Mt−1, freezes A, and optimizes only B to prevent interference with retained knowledge.This frozen-A LoRA update is equivalent to gradient descent constrained to span(A_t).
  • Plasticity: The first-step task gradient Gt is projected into the residual subspace, and its top-r singular vectors initialize the LoRA factors to improve plasticity.The initialization approximates the full-parameter tuning direction while maintaining the subspace constraints.
  • Initialization: KeepLoRA adjusts the frozen base parameterization so the initial forward behavior remains unchanged despite the non-zero initial LoRA product.The original parameter is replaced by W′ = W − α_rAB.

4 EXPERIMENTS

Experiments evaluate KeepLoRA across CLIP and LLaVA continual-learning benchmarks using Transfer, Average, and Last metrics, plus stability, plasticity, and ablation analyses. KeepLoRA achieves state-of-the-art performance across evaluated settings while reducing interference and improving plasticity.

  • Experimental setup: Experiments evaluate forward stability, backward stability, and plasticity using Transfer, Last, and Average metrics across continual-learning benchmarks.Transfer measures performance on future tasks, Last captures final continual-training performance, and Average summarizes accuracy across learned tasks.
  • Main results: KeepLoRA and KeepLoRA+ achieve state-of-the-art Transfer, Average, and Last performance across evaluated CLIP and LLaVA settings.Evaluations cover MTIL, MLLM-DCL, and UCIT, including different task orders, model architectures, and instruction formats.
  • Analysis of model stability: KeepLoRA reduces LoRA interference across tasks, with low overall backbone impact and minimal interference with unseen tasks supporting backward and forward stability.Figure 2 uses heatmaps for inter-task interference and a vertical bar for aggregate impact on the backbone.
  • Analysis of model plasticity: KeepLoRA shows consistently smaller performance drops than LoRA on newly learned tasks under matched 0.49 million and 0.98 million parameter budgets.Performance drop is measured against isolated-task accuracy after sequential learning and immediate testing.

5 CONCLUSION

KeepLoRA uses the distinction between principal and residual parameter subspaces to balance plasticity, backward stability, and forward stability. Theory and experiments support residual-subspace updates as effective for continual learning with limited interference.

  • KeepLoRA is a parameter-efficient fine-tuning method designed to balance plasticity, backward stability, and forward stability.
  • The method is motivated by assigning general knowledge to the principal parameter subspace and domain-specific adaptations to the residual subspace.
  • Theoretical analysis identifies residual-subspace-constrained updates as optimal for maximizing current-task plasticity while remaining orthogonal to general and previously learned knowledge.
  • Experiments find comparable isolated-task learning capacity to unconstrained LoRA but a significantly smaller performance drop in continual learning.

ETHICS STATEMENT

The paper presents KeepLoRA as an algorithmic contribution evaluated on public academic benchmarks without personally identifiable or sensitive information. The statement also acknowledges the broader societal consequences of machine-learning advances.

  • The paper presents KeepLoRA as an algorithmic contribution aimed at advancing continual learning.
  • Empirical validation uses publicly available academic benchmarks including CIFAR100 and Caltech101.
  • These benchmarks do not contain personally identifiable or sensitive information, while the paper acknowledges machine learning’s broad societal consequences.

REPRODUCIBILITY STATEMENT

The authors provide implementation materials and specify training settings for reproducibility. The proof material describes gradient updates when one LoRA factor is frozen.

  • Source code for KeepLoRA is provided in the supplementary material, with the core framework summarized in Algorithm 1.
  • The reproducibility materials specify hyperparameters for KeepLoRA and baselines, including learning rates, batch size, and preservation ratios ϵw and ϵf.
  • For a linear layer, the proof analyzes the gradient of Bt for task loss L when the layer output is y = xW.
  • When At is frozen and only Bt is updated, the parameter variation is represented through a negative-gradient step and the resulting change in W.

A.2 PROOFS OF PROPOSITION 3.2

The proof transforms the constrained optimization objective into a projected-gradient subspace problem. Applying the Eckart–Young–Mirsky Theorem shows that the optimal update subspace consists of the top-r left singular vectors of the projected gradient.

  • Step 1: Equivalent Transformation of the Objective Function: The proof begins by transforming the constrained objective using orthonormality and the Pythagorean theorem for the Frobenius norm.
  • Step 2: Substitute ˆGt and Simplify Using Constraints: Substituting the projected gradient ˆGt and applying the constraints simplifies the optimization problem to a subspace-selection objective.
  • Step 3: Optimal At via Eckart–Young–Mirsky Theorem: The Eckart–Young–Mirsky Theorem identifies the top-r left singular vectors of ˆGt as the optimal r-dimensional subspace.
  • Step 4: Verify Feasibility of the Optimal At: The proof verifies feasibility under the orthogonality constraints and concludes that this optimal solution matches KeepLoRA’s At initialization.

B.1 BENCHMARK

The benchmarks cover continual learning for image classification and visual question answering across diverse task sequences. Performance is evaluated with Transfer, Average, and Last metrics, with KeepLoRA+ extending KeepLoRA using class prototypes.

  • Benchmarks: MTIL contains 11 image-classification datasets, with each dataset treated as a separate task.The benchmark spans Aircraft, Caltech101, Cifar100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, and SUN397.
  • Benchmarks: MLLM-DCL contains eight downstream VQA datasets grouped into five specialized areas, each treated as a task.The areas are Remote Sensing, Medical, Driving, Finance, and Science.
  • Benchmarks: UCIT contains six VQA datasets, with each dataset treated as a task.The datasets include ArxivQA, CLEVR-Math, IconQA, ImageNet-R, VizWiz-Caption, and Flickr30k.
  • Evaluation metrics: Transfer, Average, and Last measure forward stability, mean accuracy across time steps, and final performance combining plasticity and backward stability.Transfer evaluates unseen-task performance after earlier task training, while Last evaluates performance after all training steps.
  • Method variant: KeepLoRA+ jointly optimizes class prototype vectors with KeepLoRA parameters and averages prototype-based logits during inference.Each prototype is initialized from mean vision-encoder features for the corresponding class samples.

B.4 ADDITIONAL IMPLEMENTATION DETAILS

Experiments cover CLIP and LLaVA settings with task-specific optimization configurations. On the random-order MTIL evaluation, KeepLoRA consistently outperforms prior methods across the reported metrics.

  • CLIP implementation: CLIP experiments use ViT-B/16, AdamW, learning rate 10^-3, batch size 64, and no more than 10 epochs per task.The primary settings use ϵw(vision) = 0.85, ϵw(text) = 0.2, and ϵf = 0.99.
  • LLaVA implementation: LLaVA-1.5-7b experiments use four NVIDIA H100 GPUs and AdamW for multimodal continual instruction tuning.MLLM-DCL uses learning rate 2 × 10^-5 for up to three epochs per task, while UCIT uses task-specific rates and one epoch per task.
  • Results: KeepLoRA consistently outperforms previous methods across Transfer, Average, and Last on random-order MTIL.The comparison covers StanfordCars, Food, MNIST, OxfordPet, Flowers, SUN397, Aircraft, Caltech101, DTD, EuroSAT, and CIFAR100.

C.2 HYPERPARAMETER ANALYSIS

The hyperparameter analysis examines how vision and text encoder constraints affect Transfer and Last, while accompanying tables report per-step and benchmark accuracy results.

  • Hyperparameter effects: For the image encoder, Transfer increases clearly between ϵw(vision) values 0.75 and 0.85, while Last changes only slightly.Last shows a minor decline when ϵw(vision) is larger.
  • Hyperparameter effects: Text-encoder performance is relatively insensitive to ϵw(text) when training uses only class names.The passage attributes this to the text encoder receiving much less training data than the image encoder.
  • Reported evaluations: The reported tables provide per-training-step accuracies for MTIL and additional benchmarks, alongside Transfer, Average, and Last metrics.The table captions cover KeepLoRA and KeepLoRA+ on MTIL and comparisons with LoRA-FT, O-LoRA, CL-MoE, and SEFE on MLLM-DCL and UCIT.
Loading 2601.19659v1…