PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dynamical Theory of LoRA in Continual Learning

Théo Marchetta, Filippo Alessandroni, Alessandro Breccia, Alessandro Ingrosso, Federica Gerace

arXiv:2609.39367v1stat.MLcond-mat.stat-mechcs.LGmath.PRmath.STstat.TH

TL;DR

The paper addresses how low-rank adaptation changes representations learned on earlier tasks and how that affects transfer and forgetting. It derives a high-dimensional dynamical theory of sequential learning, finding reduced interference but slower early adaptation with LoRA. State-dependent masking and rank shape the stability–plasticity trade-off, while the analysis is limited to a solvable two-layer, two-task online-learning model.

  • Problem

    How low-rank updates dynamically reorganize prior-task representations and jointly shape transfer to new tasks and forgetting of old ones remains an open continual-learning question.

    Source: §1 · pp. 1–2

  • Method

    The paper derives a high-dimensional dynamical theory for sequential Task 1 feature learning followed by Task 2 LoRA adaptation, tracking macroscopic overlaps that determine both tasks’ generalization errors.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    LoRA reduces interference with prior-task features but slows early adaptation; state-dependent masking reduces forgetting, while transfer saturates with rank even as forgetting continues to grow.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The orientation of the adaptation subspace relative to learned features matters: low rank alone does not prevent forgetting, and structured masking mitigates the stability–plasticity trade-off.

    Source: §6 · p. 10

    The analysis is restricted to a solvable two-layer, two-task online-learning model, so its quantitative predictions should not be transferred directly to large deep networks.

    Source: §6 · p. 10

Abstract

from arXiv · show

Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.

1 Introduction

The introduction frames a gap in understanding how LoRA reorganizes prior-task representations and jointly shapes transfer and forgetting. It presents a high-dimensional dynamical theory and highlights reduced interference, slower early adaptation, and the roles of masking, rank, and task similarity.

1 Introduction

  • Continual learning requires acquiring new information without catastrophically forgetting prior tasks, and LoRA updates can still interfere with useful earlier features.The introduction motivates analyzing how the adapter interacts dynamically with the representation learned before the task switch.Source: §1 · pp. 1–2
  • Existing dynamical analyses leave open how low-rank updates reorganize prior-task representations and jointly shape transfer and forgetting.The paper positions its question as complementary to work conditioning on a fixed pretrained state or not resolving time dependence across stages.Source: §1 · p. 2
  • The paper derives an asymptotically exact high-dimensional ODE theory for sequential Task 1 feature learning and Task 2 LoRA adaptation.The macroscopic dynamics jointly determine transfer, forgetting, and representation geometry.Source: §1 · p. 2
  • LoRA freezes pretrained overlaps but introduces adapter overlaps, revealing reduced interference alongside slower adaptation from initialization.The proposed state-dependent masking strongly reduces forgetting while preserving Task 2 performance, with qualitatively similar behavior on MNIST.Source: §1 · p. 2
  • Transfer saturates once adapter rank can represent the target feature space, while forgetting increases with rank.Interference peaks at intermediate task similarity and is strongly suppressed by structured masking.Source: §1 · p. 2

2 Related Work

The related-work section places the paper at the intersection of high-dimensional continual-learning dynamics and theories of LoRA. It distinguishes its time-resolved study of a dynamically learned pretrained state from prior work on constrained adaptation and fine-tuning dynamics.

High-dimensional learning dynamics

  • High-dimensional online-learning theory reduces stochastic updates to deterministic macroscopic dynamics, and Lee et al. [2021] used it to study how task similarity shapes continual-learning transfer and forgetting.Later work considered feature reuse and optimal control [Lee et al., 2022, Mori et al., 2025].Source: §2 · pp. 2–3

Continual learning with LoRA

Theoretical analyses of LoRA

  • Prior LoRA theory studies expressivity, optimization, convergence, and generalization, while Nwemadji et al. [2026] analyze fine-tuning dynamics conditional on a pretrained state.The paper instead generates that state through Task 1 and tracks both Task 2 acquisition and Task 1 forgetting.Source: §2 · p. 3
  • Duranthon et al. [2026] provide a high-dimensional theory of pre-training and LoRA fine-tuning in an attention model but do not resolve time-dependent training dynamics across the two stages.The paper describes its framework as connecting dynamical continual-learning theory with time-resolved low-rank adaptation.Source: §2 · p. 3

3 Continual Learning with LoRA: Problem Setting

The problem setting is a two-task teacher–student model trained sequentially with online learning. Task 1 trains the shared representation; after the switch, Task 2 adaptation uses a low-rank update while the learned representation is frozen.

3 Continual Learning with LoRA: Problem Setting

  • The student is a two-layer fully connected network with D hidden units, first-layer weights J, readout weights v, and activation g.Inputs are independently sampled from a standard Gaussian distribution.Source: §3 · p. 3
  • Two fixed teacher networks generate the tasks, and parameter c controls their feature similarity, from asymptotically orthogonal features at c = 0 to identical first-layer representations at c = 1.The teacher construction also specifies uniform readout weights with opposite signs.Source: §3 · p. 3
  • Task 1 trains the student’s shared first layer and Task 1 readout by online SGD; Task identity is known at evaluation, and the default width K = 2M avoids an intrinsic hidden-layer capacity bottleneck.The measured forgetting is attributed to changes in the shared first layer because the Task 1 readout is frozen after the switch.Source: §3 · p. 4
  • At the switch, Task 2 examples replace Task 1 examples and training continues for τ ∈ [0, α], with α setting the number of steps per task as T = αN.The first-layer weights at the switch are denoted Js; during Task 2, Js and the Task 1 readout remain frozen.Source: §3 · p. 4
  • Task 2 adaptation trains a factorized LoRA perturbation and its readout, with the adapter initialized so that its update is zero at the switch.The rank is L, and γ controls the update scale.Source: §3 · p. 4
  • In the limit N → ∞ with K, M, and L = O(1), stochastic training concentrates on a closed deterministic system for finitely many macroscopic order parameters.The order parameters track both tasks’ generalization errors over training time.Source: §3 · p. 4

4 A Dynamical Theory of LoRA in Continual Learning

The theory compares LoRA with full fine-tuning across Task 1 retention and Task 2 learning, then explains their differing representation dynamics. It derives a macroscopic closure for the frozen representation and adapter, with results qualified by the task-switch regime and supported by simulations.

Deterministic high-dimensional dynamics

  • The ODE predictions and finite-dimensional simulations are compared across Task 1 and Task 2 errors and student–teacher overlaps after the task switch.Figure 1 marks the switch and distinguishes theoretical trajectories from simulation markers.Source: Figure 1 · p. 5, §4 · p. 7

Macroscopic order parameters

  • The macroscopic theory tracks student and teacher overlaps plus LoRA-direction geometry and alignment with the frozen representation and both teachers.These order parameters determine the generalization errors during Task 2 training.Source: §4 · p. 6

How LoRA changes the dynamical closure

  • After the switch, LoRA freezes the Task 1 student overlaps and reconstructs the effective representation from those overlaps and adapter variables.The resulting quantities determine the covariance of adapted preactivations and the population errors on both tasks.Source: §4 · pp. 4, 6, 7

LoRA versus full fine-tuning

  • LoRA preserves substantially more Task 1 representation than full fine-tuning while reaching a comparable asymptotic Task 2 error.Theoretical trajectories closely match finite-dimensional simulations.Source: Figure 1 · p. 5, §4 · p. 7
  • Under LoRA, the pretrained overlaps stay fixed and adapter-mediated changes increase Task 2 alignment while preserving more Task 1 alignment than full fine-tuning.Under full fine-tuning, the shared representation moves toward Task 2 and modifies Task 1 features.Source: §4 · pp. 6–7

Task 1 feature learning

  • The reported task switch occurs before full specialization to the Task 1 teacher, a regime the authors describe as practically relevant.They note that full specialization for K > M requires substantially longer training.Source: §4 · p. 7

Task 2 adaptation dynamics

  • Zero-initialized LoRA adapts more slowly early on because the Task 2 signal must first build adapter overlaps through the rank-L bottleneck, while the up-projection evolves on a slower readout timescale.This transient delays growth of Task 2 overlap relative to full fine-tuning.Source: §4 · p. 7

State-dependent masking

  • In sequential MNIST, state-dependent masking improves Task 1 retention while preserving comparable Task 2 performance.Task 1 separates digits below 5 from those at least 5; Task 2 separates even from odd digits.Source: Figure 2 · p. 8

5 Stability and Plasticity through the Lens of the Theory

State-dependent masking protects Task 1-relevant units while adapting in the complementary subspace, reducing forgetting while preserving Task 2 performance. The section also shows how adapter rank and task similarity shape the stability–plasticity trade-off.

5.1 State-Dependent Masking Mitigates Forgetting

  • SDGM substantially improves Task 1 retention over vanilla LoRA while preserving similar asymptotic Task 2 performance, and the gain depends on protecting important rather than merely fewer units.Source: §5.1 · pp. 8–9
  • Applying the mask to full fine-tuning also reduces forgetting, while masked LoRA yields the strongest Task 1 retention at comparable Task 2 performance in the considered regime.Source: §5.1 · p. 9
  • On sequential MNIST, SDGM plus LoRA again improves Task 1 retention while preserving competitive Task 2 performance.Source: §5.1 · p. 9

5.2 Adapter Rank and the Stability–Plasticity Trade-off

  • Vanilla LoRA's Task 2 transfer initially improves with rank, but Task 1 forgetting also rises; SDGM keeps forgetting substantially lower.Source: §5.2 · p. 9
  • Transfer gains saturate as rank approaches teacher width M; beyond that, added rank brings little Task 2 representation benefit while Task 1 forgetting grows.Source: §5.2 · p. 9

5.3 Task similarity and interference

  • Forgetting under full fine-tuning and vanilla LoRA peaks at intermediate task similarity: tasks overlap in shared directions but remain different enough to distort Task 1 representations.Source: §5.3 · p. 9
  • SDGM suppresses intermediate-similarity interference while preserving comparable transfer; rank restriction alone does not control update orientation relative to Task 1 features.Source: §5.3 · p. 9

6 Discussion and Conclusions

The discussion attributes LoRA’s reduced interference and slower early adaptation to changes in learning geometry and timescale, while arguing that state-dependent masking reduces forgetting. The appendix supplies Gaussian-integral expressions used to derive closed-form quantities.

6 Discussion and Conclusions

  • LoRA reduces interference by freezing the pretrained representation, but its zero-initialized rank-restricted adapter first builds alignment and therefore slows early adaptation.The discussion adds that low rank alone does not prevent forgetting; the orientation of the adaptation subspace relative to previously learned features also matters.Source: §6 · pp. 9–10
  • Figure 3 shows vanilla LoRA gaining transfer at the cost of increased forgetting, while LoRA+SDGM maintains greater stability and transfer gains saturate near teacher width M.The figure varies task similarity in panels a–b and LoRA rank in panel c.Source: Figure 3 · p. 10
  • The analysis is restricted to a solvable two-layer, two-task online-learning model, so its quantitative predictions should not be transferred directly to large deep networks.The discussion frames the model as a way to isolate mechanisms and notes qualitative agreement on sequential MNIST.Source: §6 · p. 10

A Integrals computation

  • The appendix reports Gaussian-integral expressions for deriving closed-form quantities, with the subsequent computations specified for g(x) = erf(x).It includes two-, three-, and four-dimensional Gaussian averages.Source: A Integrals computation · p. 13

B Standard sequential training

This appendix presents the standard sequential-training results and the order-parameter calculations used to obtain errors and continuous-time dynamics. The Task 1 results match earlier work, while the Task 2 equations provide a comparison for the LoRA theory.

B Standard sequential training

  • Task 1 training results are identical to those in Lee et al. [2021, 2022], and the appendix also reports standard Task 2 ODEs for comparison with this work.The Task 1 results apply to the first phase of the LoRA framework.Source: B Standard sequential training · p. 14

List of order parameters

  • The appendix defines teacher and student preactivation fields and distinguishes time-dependent order parameters from static quantities set by teacher sampling.The supplied passages introduce these definitions as the setup for the error calculations.Source: List of order parameters · p. 14

Generalization errors

  • In the N→∞ limit, the preactivation fields are jointly Gaussian, and Gaussian integrals yield closed-form generalization errors from their covariance.The calculations focus on the joint covariance of the preactivation fields.Source: Generalization errors · p. 15

Gradient updates

  • The standard-training derivation specifies online-SGD prediction errors and gradients for the first-layer rows and readout weights.It proceeds from task-specific student outputs and per-example prediction errors.Source: Gradient updates · p. 15

Differential equations

  • With τ = μ/N, discrete order-parameter updates converge to continuous-time differential equations as N→∞; teacher delays depend on the active task.The delays are needed because the relevant teacher preactivations depend on which task is being solved.Source: Differential equations · p. 16

C Low-rank Adaptation on Task 2

For Task 2, the appendix derives LoRA-specific dynamics after Task 1 training, keeping pretrained quantities fixed while tracking new adapter-related order parameters. It expresses the task errors and adapter updates through Gaussian averages and high-dimensional continuous-time equations.

C Low-rank Adaptation on Task 2

List of order parameters

  • After the switch, the frozen student preactivation x_k is constant, while the theory tracks additional time-dependent order parameters for LoRA adaptation.The order-parameter setup includes the teacher, student, and adapter-direction preactivation fields.Source: List of order parameters · p. 18

Generalization errors

  • The Task 2 generalization error is expressed through preactivations and Gaussian integrals, which close the error equation using their covariance.The appendix sets up a higher-dimensional Gaussian vector containing student, teacher, and adapter-related preactivations.Source: Generalization errors · pp. 18–20

Gradient Updates

  • The appendix derives LoRA-factor and readout updates, then takes the thermodynamic limit to obtain deterministic continuous-time dynamics.An extra 1/N factor in the B update is used to obtain a well-defined, non-trivial thermodynamic limit.Source: Gradient Updates · pp. 20–21

D.1 SDGM Implementation Details

SDGM protects the Task 1 features with the largest readout magnitudes by freezing their hidden units and adapting the complementary subspace. The inverse-selection ablation and ReLU experiment support targeted feature protection rather than capacity restriction or activation saturation alone.

D.1 SDGM Implementation Details

  • SDGM freezes the κ hidden units with the largest Task 1 readout magnitudes and leaves the complementary units plastic for Task 2 adaptation.The partition is determined from the network state after Task 1; the LoRA implementation fixes a sparse up-projection matrix on the plastic units.Source: §D.1 · p. 24

D.2 Applying the inverse SDGM

  • Inverse SDGM freezes the smallest Task 1 readout magnitudes and yields higher Task 1 generalization error than standard SDGM, supporting protection of task-relevant features.The comparison is designed to distinguish targeted feature protection from architectural capacity restriction alone.Source: §D.2 · p. 25, Figure 4 · p. 25

D.3 Validation under an unbounded activation function

  • With ReLU activations, SDGM preserves Task 1 performance throughout Task 2 adaptation while achieving similar Task 2 transfer.The experiment-only result supports the selection rule independently of activation saturation.Source: Figure 5 · p. 25, §D.3 · p. 26

E Results in the specialized regime

In the specialized regime, LoRA-based methods adapt more slowly on Task 2, while SDGM preserves partial Task 1 alignment. The section also describes initialization choices intended to balance retention and plasticity.

E Results in the specialized regime

  • The reported results use the symmetric regime; specialization is harder when K > M, and its subspace-escape time constant increases linearly with K.The authors also state that their readout initialization makes specialization more difficult.Source: E Results in the specialized regime · p. 26
  • In the unspecialized case, SDGM produces no forgetting across the full range of task similarity; its initialization biases dynamics toward self-pruning and leaves directions free for Task 2.As similarity increases, readout magnitude on new directions decreases; at c = 1, no new directions are learned.Source: E Results in the specialized regime · pp. 26–27
  • With α increased to 2000, LoRA and LoRA+SDGM still adapt more slowly on Task 2; standard+SDGM adapts faster, with greater transfer and comparable forgetting.The prolonged symmetric plateau delays Task 2 learning, limiting both its acquisition and subsequent interference with Task 1.Source: E Results in the specialized regime · p. 27, Figure 7 · p. 27

F Additional details on LoRA

  • The alternative LoRA initialization sets A = 0 and B = O(1), so early learning proceeds through A rather than through a static random projection from A.The passage contrasts this with classical LoRA initialization, where the initial gradient with respect to A vanishes and early updates are driven by B.Source: F Additional details on LoRA · p. 28
  • Deterministically initializing B is proposed to reduce run-to-run variability in the LoRA trajectory while leaving both low-rank adapters trainable.Randomly selecting B's rows makes the optimization trajectory and final A configuration sensitive to initialization.Source: F Additional details on LoRA · p. 28

G Trying the various procedures on a real dataset

The real-data evaluation uses two binary classifications built from MNIST and compares continual-learning procedures on sequential tasks. The reported qualitative pattern matches the theory: SDGM+LoRA retains Task 1 better, while LoRA initially slows Task 2 adaptation.

G Trying the various procedures on a real dataset

  • Task 1 classifies digits below 5 versus digits 5 and above, while Task 2 classifies even versus odd digits.Both tasks are binary classifications constructed from MNIST.Source: G Trying the various procedures on a… · p. 28
  • The online-learning experiment uses 3 × 10^4 examples per task, flattened 28 × 28 images, and generalization curves averaged over 10 runs.Variability comes from both the data split and student-module initialization.Source: G Trying the various procedures on a… · p. 28
  • SDGM+LoRA shows the least Task 1 forgetting, while the standard procedure shows the most; LoRA also slows adaptation at the beginning of Task 2.The paper reports the same qualitative behavior in the theoretical setting and the real-data experiments.Source: G Trying the various procedures on a… · p. 28

H Hyper-parameters for numerical experiments

The numerical simulations use a default LoRA rank and task time horizon, with a longer horizon for the specialized case, and specify Euler integration and network initializations.

H Hyper-parameters for numerical experiments

Related papers

Loading 2609.39367v1…