PaperlayerView exact revision on arXiv

Source-linked AI summary

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Junkang Liu

arXiv:2610.02283v1cs.LG

TL;DR

Continual LoRA adaptation can concentrate update energy in a few modes, leaving nominal low-rank capacity underused, while balancing factors alone does not ensure balanced weight updates. MuLoRA selects directions with strong current-task response relative to historical response and balances updates within that subspace. Across five benchmarks and eight incremental settings, it achieves the highest mean accuracy in 15 of 16 reported metrics.

  • Problem

    Continual LoRA updates can concentrate energy in a few singular modes, leaving effective rank below nominal rank, and balancing both factors does not necessarily balance the induced weight update.

    Source: §1 · pp. 1–2

  • Method

    MuLoRA selects a task-responsive subspace using historical whitening, fixes a row-orthonormal basis, and applies approximate polar balancing to momentum updates of the trainable factor.

    Source: §1 · p. 2

  • Results

    Across five benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Effective adaptation requires both selecting task-responsive directions and controlling how updates use them.

    Source: §6 · p. 10

    The work provides no formal privacy, fairness, or robustness guarantees, and aggregate accuracy does not establish reliable performance across demographic groups or deployment conditions.

    Source: ETHICS STATEMENT · p. 14

Abstract

from arXiv · show

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify \emph{spectral plasticity collapse}: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose \texttt{MuLoRA}, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, \texttt{MuLoRA} achieves the highest mean accuracy in 15 of 16 reported metrics.

1 INTRODUCTION

Continual LoRA updates can leave much of their nominal rank underused when energy concentrates in a few singular modes, and balancing factor updates alone need not balance the resulting weight update. MuLoRA selects task-responsive directions relative to historical response and controls update spectra within that fixed subspace.

1 INTRODUCTION

  • Spectral plasticity collapse concentrates update energy in a few singular modes, lowering effective and stable rank without necessarily reducing algebraic rank.Thus, a fixed nominal LoRA rank does not guarantee effective use of adaptation capacity.Source: §1 · p. 1
  • When both LoRA factors are trained, individually balanced factor steps can still produce an uneven weight-step spectrum through factor interactions.The induced change includes δW = δBA + BδA + δBδA.Source: §1 · pp. 1–2
  • MuLoRA fixes a row-orthonormal, task-responsive input basis and applies Muon-style updates to the trainable factor, transferring its nonzero singular values exactly to weight steps.The basis is selected by current-task response relative to historical exposure.Source: §1 · p. 2
  • Exact spectral selection maximizes worst-direction relative response among subspaces of the same dimension, while the orthonormal basis connects subspace selection to stepwise spectral control.RRS chooses where adaptation occurs; SSB controls how update energy is distributed within each weight step.Source: §1 · p. 2
  • Cumulative spectral guarantees require controlled cross-step anisotropy because balanced individual steps need not yield a balanced accumulated update.The analysis derives cumulative condition-number and spectral-participation bounds under that condition.Source: §1 · p. 3
  • Across five benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.Ablations support the complementary benefits of selecting task-responsive subspaces and balancing updates within them.Source: §1 · p. 3

2 RELATED WORKS

Prior continual-learning work uses lightweight prompts, adapters, and structured low-rank spaces to accommodate sequential tasks or mitigate interference. MuLoRA instead selects task-responsive subspaces and balances momentum steps within them.

2 RELATED WORKS

  • PEFT continual-learning approaches include prompt selection, projection constraints, lightweight adapters, and low-rank methods that structure adaptation to balance stability and plasticity.Examples include restricting overlap with historical subspaces and exploring subspace decomposition, orthogonality, or allocation.Source: §2 · p. 3
  • The comparison distinguishes independent adapters, historical-direction isolation, and MuLoRA’s fixed task-responsive orthonormal basis with Muon updates to the trainable factor.MuLoRA transfers spectral balancing to weight steps.Source: Figure 3 · p. 4
  • MuLoRA addresses the stability–plasticity trade-off by selecting directions with strong current-task response relative to historical exposure and balancing momentum steps within them.This contrasts with methods focused on avoiding overlap with historical subspaces.Source: §2 · pp. 3–4

3 METHODOLOGY

The method uses a fixed, row-orthonormal basis to connect task-responsive subspace selection with spectral control of LoRA weight updates. It evaluates sequential, task-agnostic class-incremental learning with non-overlapping task classes.

3 METHODOLOGY

  • The setting has sequential tasks with non-overlapping classes, and task identity is unavailable at inference.The model is evaluated on all classes seen so far.Source: §3 · p. 4
  • For each task, MuLoRA freezes the backbone and row-orthonormal basis A_t, trains B_t, and merges B_tA_t into the backbone afterward.The fixed basis preserves the nonzero singular values from each factor update to its induced weight update.Source: §3 · p. 4

3.1 RELATIVE-RESPONSE SPECTRAL ALLOCATION

  • RRS allocates a rank-r subspace using current-task feature response relative to accumulated historical response, allowing shared directions when sufficiently responsive.Historical whitening converts the selection into a spectral problem; its leading eigenvectors identify directions with the strongest relative response.Source: §3.1 · pp. 4–5
  • Mapping the selected directions back and orthonormalizing them preserves the selected subspace while supplying the basis needed for exact spectral transfer.For the first task, without historical statistics, RRS selects the dominant current-task feature subspace.Source: §3.1 · p. 5

3.2 SUBSPACE SPECTRAL BALANCING

  • SSB applies approximate polar orthogonalization to momentum updates, distributing step energy across active singular modes without creating modes absent from a rank-deficient update.Because the basis is orthonormal, factor-step and weight-step spectra match; finite polynomial iterations approximate balancing, while accumulated steps need not remain balanced.Source: §3.2 · pp. 5–6
  • Newton–Schulz polynomial iterations approximate polar orthogonalization, and cumulative spectral behavior is characterized under controlled cross-step anisotropy.The small LoRA dimension makes the iterative computation efficient; individual-step balancing alone does not guarantee a balanced sum.Source: §3.2 · p. 6

4 THEORETICAL ANALYSIS

RRS selects a subspace that is max–min optimal for relative current-task response, while cumulative spectral guarantees depend on controlled cross-step anisotropy. Together, these results connect task-responsive capacity allocation with spectral control of accumulated updates.

4 THEORETICAL ANALYSIS

  • With a row-orthonormal basis, the factor update and induced weight update have identical nonzero singular values, linking RRS subspace allocation to SSB spectral control.The QR step changes the basis, not the selected subspace.Source: §4 · pp. 6–7
  • No other r-dimensional subspace achieves a higher worst-direction relative response than the RRS-selected subspace, whose least-responsive direction reaches at least λr.Euclidean re-orthogonalization preserves the selected subspace and its guarantee.Source: §4 · p. 6
  • Figure 6 tracks effective rank, κ+, stable rank, and test accuracy across tasks; SSB maintains high ranks and a well-conditioned nonzero spectrum where accuracy is better preserved.The passage describes this geometric advantage as associated with retained adaptation capacity.Source: Figure 6 · p. 7

4.1 CUMULATIVE SPECTRAL BALANCE

  • For cumulative updates, the condition-number, stable-rank, and effective-rank guarantees require bounds on cross-step anisotropy.When the cumulative anisotropy term is zero, the singular values are equal and condition number is 1.Source: §4.1 · p. 7
  • The robust bound separates cross-step anisotropy ρ from relative accumulated orthogonalization error τK; ρ + τK < 1 ensures a positive lower bound on the smallest squared singular value.When τK = 0, the bounds reduce to those of Theorem 4.2.Source: §4.1 · pp. 7–8

5 EXPERIMENTS

Across five benchmarks and eight incremental settings, MuLoRA leads in 15 of 16 reported metrics. Ablations report gains from both RRS and SSB, with their combination achieving the highest mean accuracy across the eight component-comparison metrics.

5 EXPERIMENTS

  • The evaluation covers ImageNetR, ImageNetA, CIFAR-100, CUB, and DomainNet, reporting ALast after the final session and AAvg across incremental sessions.The main comparison averages results over three random seeds.Source: §5 · p. 8

5.1 COMPARISON WITH BASELINE METHODS

  • MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics across five benchmarks and eight incremental settings.On 20S-ImageNetA, it reaches 55.86% ALast and 65.61% AAvg, exceeding LoRA-P&M by 3.59 and 3.46 points, respectively.Source: §5.1 · pp. 8–9
  • On 20S-ImageNetA and 20S-ImageNetR, MuLoRA better maintains overall performance while preserving strong adaptation to newly introduced classes.The comparison tracks overall and new-task accuracy across incremental tasks.Source: Figure 7 · p. 9

5.2 ABLATION STUDIES

  • Combining RRS and SSB achieves the highest mean accuracy on all eight ablation metrics; adding RRS to SSB further raises ALast by 0.37–1.54 points and AAvg by 0.77–1.08 points.RRS alone raises ALast by 1.65–3.30 points over baseline, while SSB alone raises it by 2.29–5.70 points.Source: §5.2 · p. 9, Table 3 · p. 9

6 CONCLUSION

MuLoRA couples history-aware subspace selection with spectral balancing through a fixed row-orthonormal basis. The paper concludes that effective adaptation requires both task-responsive directions and control over how updates use them.

6 CONCLUSION

  • MuLoRA combines history-aware subspace selection and Muon-style spectral balancing through a fixed row-orthonormal basis, with conditional spectral guarantees for accumulated updates.Experiments across five benchmarks and eight incremental settings support the conclusion that effective adaptation requires selecting task-responsive directions and controlling update utilization.Source: §6 · pp. 9–10

ETHICS STATEMENT

The work evaluates MuLoRA on existing image-classification benchmarks and provides no formal privacy, fairness, or robustness guarantees.

ETHICS STATEMENT

  • MuLoRA is evaluated on ImageNet-R, ImageNet-A, CIFAR-100, CUB, and DomainNet, without formal privacy, fairness, or robustness guarantees.Aggregate classification accuracy does not establish reliable performance across demographic groups or deployment conditions; sensitive applications require safeguards and application-specific evaluation.Source: ETHICS STATEMENT · p. 14

REPRODUCIBILITY STATEMENT

The paper specifies its experimental setup and reports that the main comparison averages results across three random seeds.

REPRODUCIBILITY STATEMENT

  • Experiments use a pretrained ViT-B/16, specified adapter and optimization settings, one RTX 4090 GPU, and results averaged over three random seeds.The implementation and experiment scripts are to be publicly released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 14

AI USE STATEMENT

The statement limits language-model assistance to editing, while the appendix states assumptions for its proofs and clarifies the spectral quantities and basis-preservation result.

AI USE STATEMENT

  • Language models assisted with rephrasing, grammar, and readability, but not research ideation, methods, experiments, derivations, implementation, data processing, or result interpretation.The authors reviewed the text and take responsibility for the manuscript.Source: AI USE STATEMENT · p. 14

A PROOFS OF THE THEORETICAL RESULTS

  • The cumulative analysis retains all r singular values, including zeros, and studies within-task optimization with the input basis fixed.The appendix assumes real matrices and natural logarithms.Source: A PROOFS OF THE THEORETICAL RESULTS · p. 16

A.1 TWO SPECTRAL PRELIMINARIES

  • For a nonzero matrix, entropy-based effective rank and energy participation are distinct from algebraic rank, while a row-orthonormal basis preserves nonzero singular values and associated spectral statistics.When positive singular values are equal, the spectral quantities in equation 7 equal their count.Source: §A.1 · p. 16

A.2 EXACT RELATIVE-RESPONSE SUBSPACE SELECTION

The section characterizes relative-response subspace selection and gives conditional cumulative spectral guarantees, with explicit limits on approximate selection, historical-response interpretation, and cross-task scope.

A.2 EXACT RELATIVE-RESPONSE SUBSPACE SELECTION

  • Exact leading eigenvectors select the r-dimensional subspace maximizing worst-direction relative response, and QR preserves that subspace while providing a Euclidean-orthonormal basis.Repeated eigenvalues may make the optimum nonunique; randomized SVD does not automatically retain global max–min optimality.Source: §A.2 · pp. 17–18

A.3 CUMULATIVE SPECTRAL BALANCE WITH EXACT STEPS

  • Under the stated positive-mean and bounded-anisotropy condition, exact cumulative-step analysis bounds singular values and stable rank, with the bounds transferring to the weight update under a row-orthonormal basis.The argument bounds entropy effective rank through energy participation rather than equating the two quantities.Source: §A.3 · pp. 18–19

A.4 CUMULATIVE SPECTRAL BALANCE WITH APPROXIMATE STEPS

  • For approximate steps, the cumulative bounds require the stated strict positivity condition for condition-number and rank results, and the resulting bounds transfer to the weight update.The strict condition is δK < aK(1 − ρ); the approximate-step analysis also accounts for accumulated orthogonalization error.Source: §A.4 · pp. 19–20

B.1 INTERPRETING HISTORICAL RESPONSE

  • Historical response uses unnormalized second-moment statistics collected at earlier backbone states, making it a surrogate rather than an exact recomputation of past features or historical loss.The method retains aggregate statistics rather than past examples or individual historical features.Source: §B.1 · p. 20

B.2 A COMPUTABLE CHARACTERIZATION OF AN APPROXIMATE RRS SUBSPACE

  • For an approximate RRS subspace, its worst-direction relative response is characterized by θr, and a spectral residual supplies a sufficient, potentially conservative certificate against the exact optimum.A small invariant-subspace residual alone does not establish that the computed subspace is leading.Source: §B.2 · p. 21

B.3 INTERPRETATION AND SCOPE OF THE CUMULATIVE GUARANTEES

  • The cumulative guarantee is conditional on cross-step anisotropy and applies within one task with a fixed basis, not directly to sums across tasks with changing bases.The results concern update geometry and do not independently bound task loss, classification error, or forgetting.Source: §B.3 · p. 21

B.4 CHECKING THE CUMULATIVE CONDITIONS WITHOUT PAIRWISE STEP STORAGE

  • The cumulative conditions can be checked from actual factor steps without storing every step pair, but a failed sufficient certificate does not establish poor accuracy or rank deficiency.The conditions must include all step scaling and any additional implementation terms.Source: §B.4 · p. 21

C.1 DEFINITIONS OF THE COMPARED VARIANTS

  • The comparison variants separate input-subspace choices from spectral-control choices, while MuLoRA combines relative-response selection with fixed-basis updates to B.The listed alternatives include trainable factors, random or current-task PCA bases, historical isolation, and EBLoRA’s adapter-spectrum constraint.Source: §C.1 · p. 22

C.2 RESULTS AND INTERPRETATION

MuLoRA has the highest mean final and average accuracy in the mechanism comparison. Relative-response selection outperforms the listed alternatives, while comparisons with other designs do not isolate the effect of any single component.

C.2 RESULTS AND INTERPRETATION

  • MuLoRA attains the highest mean final and average accuracy in the mechanism comparison.The comparisons do not establish that fixed orthonormal parameterization is necessary or isolate a causal effect of spectral balancing.Source: §C.2 · pp. 22–23
  • Relative to a fixed random orthonormal basis, RRS improves ALast and AAvg by 1.54 and 1.08 percentage points, and gains 3.41 and 2.05 points over current-task PCA.The PCA comparison suggests that strong current-task feature response alone is insufficient in this setting.Source: §C.2 · p. 22
  • Relative to historical isolation, RRS improves ALast and AAvg by 1.19 and 0.85 points, supporting graded relative-response selection over excluding retained historical directions.Source: §C.2 · p. 22
  • MuLoRA exceeds jointly trainable factors with Muon and EBLoRA, but these comparisons do not isolate spectral balancing or fixed parameterization as the cause.Mean differences are not claims of statistical significance, and overall and average accuracy do not distinguish new-task learning from reduced forgetting.Source: §C.2 · p. 23

D.1 REPORTED EXPERIMENTAL CONFIGURATION

The reported experiments use a specified ViT-B/16 configuration and three-seed averages. The accounting distinguishes persistent historical state and task-initialization costs, while the theoretical and empirical conclusions have stated scope boundaries.

D.1 REPORTED EXPERIMENTAL CONFIGURATION

  • The main configuration uses an ImageNet-21K-pretrained ViT-B/16 with rank 32, a cosine classifier, and one RTX 4090 GPU; results average three seeds.Tasks use 20 epochs, batch size 48, and a cosine learning-rate schedule; the initial rate is 10^-4 on ImageNet-R and 2 × 10^-4 on other benchmarks, including ImageNet-A.Source: §D.1 · p. 23

D.2 MEMORY AND COMPUTATIONAL ACCOUNTING

  • A dense historical second-moment matrix costs O(m^2) persistent storage, while task initialization includes feature collection, O(m^3) Cholesky factorization, and other construction costs.The trainable factor has nr parameters and the fixed basis stores mr entries; one orthogonalization iteration costs O(nr^2 + r^3), with r ≤ n.Source: §D.2 · p. 23
  • These operation counts do not establish a wall-clock advantage, which depends on feature collection and subspace construction as well as optimizer steps.Source: §D.2 · p. 23

D.3 SCOPE OF EMPIRICAL AND THEORETICAL CONCLUSIONS

  • The exact spectral-transfer identity assumes a row-orthonormal basis; cumulative guarantees rely on justified cross-step conditions and do not bound forgetting, classification error, or generalization.The mechanism comparison covers one 20-session benchmark setting; broader accuracy results do not replace matched mechanism comparisons across additional settings.Source: §D.3 · p. 23

Related papers

Loading 2610.02283v1…