PaperlayerView exact revision on arXiv

Source-linked AI summary

PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Yuxuan Li, Fanhu Zeng, Hao Tang

arXiv:2610.04226v1cs.LG

TL;DR

LoRA-based continual-learning methods constrain update directions, but the role and task-dependent strength of magnitude restrictions lack principled guidance. PaLoRA combines adaptive rank compression, nullspace projection, and rank-aware pacing; its main comparisons show consistent gains, including near-four-point gains over LoRA−DRS on 50-task ImageNet-R and ImageNet-A.

  • Problem

    Existing LoRA-based continual-learning methods focus on constraining update directions while largely overlooking update magnitude.

    Source: §1 · p. 1

  • Method

    PaLoRA compresses accumulated updates with adaptive SVD truncation, projects gradients onto the historical nullspace, and paces gradient scaling according to effective rank.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Across three benchmarks and evaluated task splits, PaLoRA achieves the highest final and average accuracy in the five-seed main comparisons; at 50 tasks, its final accuracy exceeds LoRA−DRS by 4.09 percentage points on ImageNet-R and 3.99 points on ImageNet-A.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The pacing law provides a rank-aware scaling principle for balancing stability and new-task progress under its stated model, while its deep-ViT use is approximate.

    Source: §1 · p. 2, §3.3 · p. 6

    The pacing law is derived under leakage and local-sensitivity assumptions, so its use for deep ViTs is an approximate scaling principle rather than a formal guarantee; evaluation is limited to image classification.

    Source: §5 · p. 10

Abstract

from arXiv · show

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law $s^*=\sqrt{R/c}$ that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where $R$ is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.

1 Introduction

LoRA-based continual-learning methods constrain update directions but lack principled guidance for how magnitude restrictions should change as tasks accumulate. PaLoRA combines rank-aware pacing with adaptive compression and nullspace projection, and its main comparisons show consistent gains, including on 50-task benchmarks.

1 Introduction

  • Existing LoRA-based methods constrain update directions but largely overlook update magnitude, whose restriction lacks theoretical guidance and principled adaptation as tasks accumulate.Small learning rates are commonly used as an additional safeguard, but fixed restriction strength may become suboptimal as historical knowledge grows.Source: §1 · pp. 1–2
  • As tasks accumulate, the effective rank of historical knowledge grows and the trade-off shifts toward stability, motivating stronger update restrictions.Restricted updates keep optimization closer to the joint solution than unrestricted movement toward a new task’s optimum.Source: §1 · p. 2
  • Under anisotropic leakage and local quadratic old-task sensitivity, the derived pacing law s*=sqrt(R/c) maximizes new-task progress subject to a forgetting budget.R is the effective rank of accumulated updates; for deep ViTs, the result motivates a scaling principle rather than a formal guarantee.Source: §1 · p. 2
  • PaLoRA compresses accumulated updates with adaptive SVD truncation, projects new-task gradients onto the historical nullspace, and applies rank-aware gradient scaling.An empirically motivated feature-space regularizer promotes current-task discrimination, but the pacing-law optimality claim does not cover the full regularized framework.Source: §1 · p. 2
  • At 50 tasks, PaLoRA’s final accuracy exceeds LoRA−DRS by 4.09 percentage points on ImageNet-R and 3.99 points on ImageNet-A.Across the five-seed main comparisons, PaLoRA achieves the highest final and average accuracy across all evaluated splits.Source: §1 · p. 2

2 Related Work and Preliminaries

Related work spans parameter-efficient adaptation and continual-learning methods, including approaches that isolate task directions but still face gradient leakage. The preliminaries frame example-free class-incremental learning and describe LoRA’s low-rank adaptation of frozen weights.

2.1 Related Work

  • LoRA reduces trainable parameters through low-rank weight decomposition and adds no inference overhead.PEFT methods adapt pretrained models with minimal trainable parameters while keeping pretrained weights frozen.Source: §2.1 · p. 3
  • Continual-learning approaches include rehearsal, output distillation with LwF, and gradient orthogonalization with GPM [41].The passage notes storage constraints and privacy concerns as reasons example-free approaches have drawn attention.Source: §2.1 · p. 3
  • LoRA-based continual-learning methods isolate task directions through orthogonality or disjoint parameter entries, yet gradient leakage still causes forgetting to increase with task count.O-LoRA uses orthogonality regularization, while BiLoRA assigns nearly disjoint entries to tasks.Source: §2.1 · p. 3

2.2 Preliminaries

  • Example-free class-incremental learning trains on current-task samples without past data, then predicts across all learned classes without task identity.Each task introduces a disjoint class set.Source: §2.2 · p. 3
  • LoRA represents task-specific adaptation as a low-rank update to a frozen pretrained weight matrix, restricting learning to a low-dimensional subspace.The factors A and B have rank r much smaller than the weight dimension d.Source: §2.2 · p. 3

3 Methodology

The section motivates strengthening update-magnitude restriction as tasks accumulate, derives an effective-rank pacing rule under a leakage model, and translates it into compression, nullspace projection, and scaled updates.

3.1 On the Role of Restricted Updates in Continual Learning

  • As tasks accumulate, the model’s previous-task solution better represents prior-task performance, so restriction should strengthen to balance stability and plasticity.The trajectory analysis places restricted updates between the new-task and joint optima.Source: §3.1 · p. 4

3.2 Theoretical Analysis: Rank-Aware Scaling for LoRA-Based Continual Learning

  • Nullspace projection preserves old-task forward passes under exact arithmetic, but finite-precision arithmetic and imperfect projection motivate modeling leakage into the historical subspace.The pacing trade-off is that increasing s suppresses forgetting quadratically while reducing new-task progress linearly.Source: §3.2 · pp. 4–5
  • Under the stated model, s*=sqrt(R/c) is the optimal pacing scale, with R the effective rank of accumulated updates and c a forgetting-budget tolerance.The analysis assumes anisotropic leakage and local quadratic old-task sensitivity; for deep ViTs, the result motivates scaling rather than providing a formal guarantee.Source: §3.2 · pp. 4–5

3.3 PaLoRA Framework

  • PaLoRA compresses accumulated updates by retaining singular components until their cumulative energy exceeds threshold τ, then uses the resulting effective rank and nullspace projection for paced updates.It projects the At gradient and scales the Bt gradient by sqrt(R/C), where C is a budget constant.Source: §3.3 · pp. 5–6
  • An easy-positive triplet-inspired regularizer compensates for reduced plasticity from nullspace projection by encouraging a margin against the nearest negative sample.The objective combines cross-entropy and regularization, balanced by λ.Source: §3.3 · p. 6
  • The pacing law is derived for a single-layer linear model; its use in deep ViTs is an approximate scaling principle, and the Pareto claim does not cover the triplet-regularized system.The triplet loss is an empirically motivated compensator, not a consequence of the pacing-law derivation.Source: §3.3 · p. 6

4 Experiments

Across three image benchmarks and multiple task splits, PaLoRA leads accuracy comparisons, including long task sequences. Ablations find that projection and rank-aware pacing improve performance, with α = 0.5 performing best among tested exponents.

4.1 Experimental Setup

  • Experiments cover CIFAR-100, ImageNet-R, and ImageNet-A, each split into task sequences of 10, 20, 25, or 50 tasks.The main accuracy tables report means and standard deviations over five random seeds.Source: §4.1 · pp. 6–7

4.2 Main Results

  • PaLoRA achieves the highest mean final and average accuracy across settings; at 50 tasks, its ImageNet-A final-accuracy gain over LoRA−DRS is 3.99 points.The main accuracy tables report five-seed results, and the gain is also stated in Table 3.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 Comparison with SD-LoRA

  • At 50 tasks, PaLoRA's final-accuracy gains over SD-LoRA are 6.72, 21.16, and 29.10 points on ImageNet-R, CIFAR-100, and ImageNet-A, respectively.PaLoRA performs better in every reported setting, with larger final-accuracy gaps at 50 tasks than at 10 tasks.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 Ablation Study

  • Projection mitigates forgetting, rank-aware pacing further improves accuracy trajectories, and triplet regularization adds a smaller gain within PaLoRA.On 50-task CIFAR-100, triplet loss adds 0.63 points to PaLoRA; adding it to Sequence yields 80.00% final accuracy.Source: §4.4 · p. 9, Figure 4 · p. 9, Table 6 · p. 10
  • Among tested pacing exponents, α = 0.5 performs best; larger exponents reduce plasticity, while smaller ones reduce stability and increase forgetting.The tested values were 0.125, 0.25, 0.5, 1.0, and 2.0.Source: §4.4 · pp. 9–10, Figure 5 · p. 10

5 Limitations and Future Work

The pacing law is derived under a single-layer linear model and specific leakage and sensitivity assumptions, while its use for deep ViTs is an approximate scaling principle. Evaluation is limited to image classification, with broader settings left for future study.

5 Limitations and Future Work

  • The pacing law assumes a single-layer linear setting with anisotropic leakage and local quadratic old-task sensitivity; its validity in deep ViTs is not established.For deep ViTs, the law motivates an approximate scaling principle rather than providing a formal guarantee.Source: §5 · p. 10
  • The Pareto-optimality claim applies to budget-constrained pacing, while the auxiliary triplet loss is an empirically motivated plasticity compensator.Source: §5 · p. 10
  • Evaluation covers image classification; generative models remain future work, and the pacing constant and SVD-threshold behavior in broader continual-learning settings remains to be studied.Source: §5 · p. 10

6 Conclusion

PaLoRA addresses anisotropic leakage and fixed gradient scaling with rank-aware pacing, and reports gains in long-horizon image-classification settings. The appendix frames its theoretical trade-off in a linear model, where projected updates preserve old-task passes under exact arithmetic but residual leakage is modeled under stated assumptions.

6 Conclusion

  • LoRA continual learning exhibits anisotropic leakage into previously acquired subspaces, and fixed gradient-scaling heuristics are suboptimal; PaLoRA uses the rank-aware pacing law s*=sqrt(R/c).The law is described as Pareto optimal under the stability–plasticity trade-off.Source: §6 · p. 10
  • Up to 4% gains are reported on the challenging 50-task ImageNet-A and ImageNet-R continual-learning settings.Source: §6 · p. 10

A.1 Setup and Notation

  • The analysis sets a frozen-pretrained-weight linear layer with accumulated LoRA updates of effective rank R, then constrains new increments to the historical nullspace.Under exact arithmetic, the constraint preserves old-task forward passes.Source: §A.1 · p. 15

A.2 Leakage and Sensitivity Assumptions

  • The leakage analysis models finite-precision and projector-imperfection residuals as anisotropic, with direction-dependent quadratic sensitivity of old-task loss.The assumptions describe residual leakage along historical directions, not the full displacement between task optima, and their deep-ViT validity is unestablished.Source: §A.2 · p. 15

A.3 The Forgetting–Progress Trade-off

  • Under the stated assumptions, halving the step cuts forgetting fourfold but halves new-task progress, motivating a small linear price for quadratic forgetting reduction.Source: §A.3 · p. 16

A.4 Proofs of Main Lemmas

The appendix formulates pacing as maximizing first-order new-task progress under a per-step forgetting budget. The optimal scale lies at the least restrictive feasible boundary, and the resulting rank-aware scale meets the prescribed budget across historical subspaces.

A.4 Proofs of Main Lemmas

  • The proof frames optimal scaling as maximizing new-task progress subject to a per-step forgetting budget.For projected gradient g with squared norm G², progress has magnitude ηG²/s.Source: §A.4 · p. 16
  • Because progress decreases with s, the unique optimal scale is the smallest value allowed by the forgetting constraint.Source: §A.4 · p. 16

A.5 Properties of the Scaling Law

  • With s*=sqrt(R/c), per-step forgetting equals c times the average per-direction noise floor regardless of the directions in the rank-R subspace.This is the unique Pareto-optimal point for budget-constrained pacing, not for the full system with auxiliary regularization.Source: §A.5 · p. 17, §A.4 · p. 17

B.1 Implementation Details

The implementation inserts LoRA into ViT attention key and value projections and fixes the compression, pacing, and regularization settings across datasets and task settings. The paper also specifies the evaluation seeds and training configuration.

B.1 Implementation Details

  • LoRA modules are inserted into ViT attention key and value projections, with SVD threshold τ=0.99, pacing constant C=4, and regularization weight 0.5 with margin 2.0.Source: §B.1 · p. 17
  • Main accuracy tables report means and standard deviations over seeds 0–4, while additional analyses report point estimates unless otherwise stated.Source: §B.1 · p. 17
  • Experiments use a ViT-B/16-IN21K backbone, Adam at learning rate 0.0005, batch size 128, and one NVIDIA GeForce RTX 4080 SUPER GPU.Source: §B.1 · p. 17
  • Training runs for 50 epochs on ImageNet-R and ImageNet-A and 20 epochs on CIFAR-100.Source: §B.1 · p. 17

B.2 Additional Accuracy Curves

Figure 6 extends the main-text accuracy trajectories across CIFAR-100, ImageNet-A, and ImageNet-R and 10-, 20-, 25-, and 50-task splits; PaLoRA maintains strong performance throughout.

B.2 Additional Accuracy Curves

  • Figure 6 shows PaLoRA maintaining strong performance throughout the learning sequences across datasets and task splits.The main text presents representative trajectories; Figure 6 provides the full results.Source: §B.2 · p. 17, Figure 6 · p. 18
  • The accuracy trajectories cover CIFAR-100, ImageNet-A, and ImageNet-R under 10-, 20-, 25-, and 50-task splits.Source: Figure 6 · p. 18
  • At τ = 0.99, SVD compression makes effective rank grow more slowly than the uncompressed R = rt reference, with growth varying by task sequence.Source: Figure 7 · p. 18

B.3 Effectiveness of SVD Compression

Across datasets and task settings, SVD compression adapts effective rank to task difficulty; on ImageNet-A, the adaptive choice retains fewer directions while keeping accuracy comparable to the uncompressed reference.

B.3 Effectiveness of SVD Compression

  • At τ = 0.99, effective rank adjusts dynamically with task difficulty instead of growing linearly with the number of tasks.The analysis evaluates the SVD compression module across three datasets and different task settings.Source: §B.3 · p. 18
  • On ImageNet-A, adaptive SVD retains fewer directions than k = r while maintaining comparable accuracy.The figure compares rank growth and accuracy for 10-, 20-, and 50-task settings.Source: Figure 8 · p. 19
  • Performance varies little across the tested pacing-constant range on CIFAR-100 with 50 tasks and τ = 0.99.Source: Table 7 · p. 19
  • The adaptive method's accuracy curves stay close to the k = r upper bound while compressing according to task difficulty.The fixed-rank comparison retains the top t×k components; k = 1 is a lower bound and k = r is the no-compression upper bound.Source: §B.3 · pp. 18–19

B.4 Hyperparameter Sensitivity

These appendix parts document PaLoRA’s hyperparameter and backbone checks, compatibility, training-time overhead, and research checklist information, alongside a declaration about LLM use in the core method.

B.4 Hyperparameter Sensitivity

  • On 50-task CIFAR-100, varying C from 2.25 to 6.25 yields 88.70%–88.92% final accuracy, supporting the default C=4 and τ=0.99.At C=4, τ=0.95 gives average final rank 39.6 and final accuracy 84.16%; increasing τ from 0.99 to 0.999 raises rank from 159.8 to 390.8 with little accuracy change.Source: Table 8 · p. 19, §B.4 · p. 19

B.5 Computational Overhead

  • PaLoRA averages 3.01 minutes per task, versus 2.67 for Sequence and 4.36 for LoRA−DRS; without triplet loss, it takes 2.81 minutes.Compression, projection, and pacing together account for the difference without triplet loss, while triplet loss adds 0.20 minutes; SVD compression takes approximately 0.08 seconds per task, or about 4 seconds over 50 tasks.Source: §B.5 · pp. 19–20, Table 9 · p. 20

B.6 Compatibility with Other LoRA Variants

  • On CIFAR-100 with 50 tasks, PaLoRA raises PiSSA final accuracy from 71.85% to 89.30%, slightly above the 88.86% with standard LoRA.The pacing formulation uses the aggregated low-rank task update rather than a specific factor initialization.Source: Table 10 · p. 20, §B.6 · p. 20

B.7 Different Pretrained Backbones

  • PaLoRA achieves the highest final and average accuracy in all four 50-task ImageNet-R and ImageNet-A settings with DINOv1 and iBOT backbones.This extends the comparison beyond the default ViT-B/16-IN21K backbone.Source: Table 11 · p. 20, §B.7 · p. 20

NeurIPS Paper Checklist

  • The checklist reports that limitations are discussed, proofs and assumptions are provided in the appendix, and code, logs, and implementation details are supplied in the supplements or appendix.Main accuracy tables report means and standard deviations over five random seeds (0–4); other tables and accuracy curves use point estimates.Source: NeurIPS Paper Checklist · pp. 21–23
  • The paper marks LLM-use disclosure N/A because LLMs were not an important, original, or non-standard component of its core method.The checklist directs authors to the NeurIPS handbook for guidance on what LLM use should be described.Source: NeurIPS Paper Checklist · p. 27

Related papers

Loading 2610.04226v1…