PaperlayerView exact revision on arXiv

Source-linked AI summary

SPACE-LoRA: Allocating Activation-Subspace Protection for Continual Learning

Seunghyun Yoo, Kiseok Kim, Hyeontae Joo, Junyeop Bang, Hwangnam Kim

arXiv:2609.34453v1cs.LGcs.AI

TL;DR

Sequential LoRA learning can forget old tasks because a new residual branch may respond to historically important inputs even when prior parameters are frozen. SPACE-LoRA projects new branches away from selected historical activation directions and sizes protection per module using Fisher coverage. The paper reports reduced forgetting with competitive performance across vision and language benchmarks, while noting that Fisher-coverage sizing does not consistently outperform uniform sizing.

  • Problem

    Existing parameter or subspace constraints do not fully address additive interference: a new LoRA branch can respond to old-task activations and alter previous predictions.

    Source: §1 · pp. 1–2

  • Method

    SPACE-LoRA hard-projects each new branch’s input factor away from selected historical activation directions and uses module-wise Fisher coverage to choose protection size.

    Source: §1 · p. 2

  • Results

    Across vision and language benchmarks, SPACE-LoRA is reported to alleviate forgetting while preserving incoming-task adaptation and improving the stability-plasticity trade-off.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The conclusion reports that protecting important historical activation directions is effective for continual low-rank adaptation, although Fisher-coverage sizing does not consistently outperform uniform sizing.

    Source: §6 · p. 10

Abstract

from arXiv · show

This study addresses the catastrophic forgetting problem that occurs when sequentially learning successive tasks using Low-Rank Adaptation (LoRA) from a lifelong learning perspective. While existing approaches have primarily constrained parameter updates or learning subspaces to reduce interference with past knowledge, they have not fully considered additive interference. This occurs when a newly added residual adapter on top of a fixed past model generates non-zero responses along input directions important for old tasks, thereby altering previous predictions. To this end, we propose Subspace Protection with Allocated Capacity for Efficient Continual Adaptation (SPACE-LoRA). SPACE-LoRA directly suppresses the responses of the new residual branch along input activation directions that are important for old tasks and adaptively determines the protection coverage for each module based on past-task sensitivity estimated via a common Fisher sensitivity-based coverage target. Under a fixed LoRA rank, this approach adaptively adjusts module-specific protection coverage while suppressing interference along input directions sensitive to old tasks. We assess the effectiveness of activation-subspace protection in mitigating catastrophic forgetting and examine the role of sensitivity-guided protection in continual learning across diverse tasks. Code is available at https://anonymous.4open.science/r/SPACE-LoRA-7864.

1 INTRODUCTION

The introduction identifies additive interference from new LoRA branches as a source of forgetting that parameter freezing and existing subspace constraints do not fully address. SPACE-LoRA protects selected historical activation directions, allocates protection by module sensitivity, and is reported to perform competitively across vision and language benchmarks.

1 INTRODUCTION

  • New LoRA branches can alter old-task predictions through non-zero responses to historical activations, even when previously learned parameters remain frozen.Fixed protection sizes may underprotect sensitive modules or unnecessarily constrain adaptation in less-sensitive ones.Source: §1 · pp. 1–2
  • SPACE-LoRA projects each new branch’s input factor away from selected historical activation directions, guaranteeing zero residual response there while preserving the complementary space for adaptation.Both LoRA factors remain trainable, and the projection is applied after optimizer steps.Source: §1 · p. 2
  • A Fisher-coverage scheme adapts protection size to each module’s sensitivity, aiming to balance knowledge retention and new-task adaptation across the network.The protection size is chosen independently for each module.Source: §1 · p. 2
  • Across vision and language benchmarks, SPACE-LoRA is reported to alleviate forgetting while preserving adaptation to incoming tasks and improving the stability-plasticity trade-off.The evaluation compares diverse benchmarks and competitive baselines.Source: §1 · p. 2

2 RELATED WORK

Related work covers continual learning and parameter-efficient adaptation, then reviews approaches that limit interference through parameter, gradient, or subspace constraints. SPACE-LoRA instead combines a hard activation-response constraint with module-wise Fisher-coverage sizing.

2 RELATED WORK

  • Continual learning seeks to retain knowledge across sequential tasks; PEFT freezes a pretrained model and trains fewer parameters, with LoRA using two low-rank factors.CL settings include task-, class-, and domain-incremental learning, and methods are commonly grouped as regularization-, memory-, or expansion-based.Source: §2 · pp. 2–3
  • LoRA-based continual-learning methods separate or constrain task updates using orthogonal subspaces, fixed low-interference factors, sensitivity-based selection, or gradient rectification.Examples include O-LoRA, InfLoRA, PLAN, LB-CL, SplitLoRA, and JANUS-LoRA.Source: §2 · p. 3
  • Related projection methods protect old-task subspaces using representation energy or input-feature covariance, while EWC uses Fisher importance to constrain parameter changes.AdaLoRA and OA-Adapter instead allocate adaptation capacity selectively across components.Source: §2 · p. 3
  • SPACE-LoRA projects each new branch’s input factor onto the complement of selected old-task activation directions, producing zero response on that protected subspace.It orders directions by activation eigenvalue and selects the minimum prefix reaching the target Fisher coverage.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe task-specific LoRA adapters accumulated over a frozen pretrained model. They distinguish parameter freezing from behavioral preservation: a new adapter can still change module outputs on old-task inputs and contribute to forgetting.

3.1 LORA-BASED CONTINUAL LEARNING

  • LoRA continual learning accumulates task-specific adapters while updating only the current task’s factors and freezing the pretrained weights and prior adapters.The update uses a low-rank factorization with rank r and scaling hyperparameter α.Source: §3.1 · p. 3

3.2 ADAPTER INTERFERENCE AND CATASTROPHIC FORGETTING

  • A newly learned adapter can change a module’s output on a previous-task activation despite the earlier adapters remaining frozen.Changes in preceding modules may also alter the activation reaching a later module.Source: §3.2 · p. 4
  • Task-wise forgetting measures the decline in task j’s accuracy after subsequent adaptation, relative to its accuracy immediately after learning.The supplied definition identifies the compared accuracies and the direction of decline.Source: §3.2 · p. 4

4 SPACE-LORA

SPACE-LoRA constrains new residual branches on selected historical activation directions while leaving their orthogonal complement available for adaptation. Module-wise Fisher coverage determines how many directions to protect, and the procedure updates historical statistics between tasks.

4 SPACE-LORA

  • SPACE-LoRA suppresses new-branch responses along selected past activation directions and sets each module’s protection size using a common Fisher-sensitivity coverage target.Under fixed LoRA rank, the complementary input subspace remains available for learning the new task.Source: §4 · p. 4

4.1 RESIDUAL RESPONSE NULLIFICATION

  • Historical activation directions are ordered by eigenvalue, and the new branch’s input factor is projected onto the orthogonal complement after each optimizer update.The hard constraint gives zero residual response on protected directions while retaining the complementary space for adaptation.Source: §4.1 · p. 5

4.2 MODULE-WISE FISHER-COVERAGE PROTECTION

  • Each module protects the minimum number of activation-ordered directions needed to capture its target fraction of historical loss sensitivity.The activation basis supplies direction order, while Fisher sensitivity determines coverage; protection is capped to leave at least r dimensions for the rank-r adapter.Source: §4.2 · pp. 5–6
  • Historical Fisher sensitivity is computed from training-example gradients with respect to each module’s effective weight, using the task-end classifier and observed labels.These statistics are accumulated for use before subsequent tasks.Source: §4.2 · p. 5

4.3 UPDATE PROCEDURE

  • Before each new task, the procedure constructs protected bases and selects module sizes; after training, it adds endpoint activation and Fisher statistics to historical records.The first task is trained without protection because historical statistics are unavailable.Source: §4.3 · p. 6

5 EMPIRICAL STUDY

The evaluation spans vision and language continual-learning benchmarks, with results varying across task settings and benchmarks. Coverage and allocation ablations show trade-offs between retention and adaptation, while Fisher-guided allocation improves final accuracy over uniform allocation in both tested settings.

5.1 EXPERIMENTAL SETUP

  • Evaluation spans ImageNet-R, DomainNet, VTAB, and VDD in vision and 15 NLP classification datasets, using class-incremental evaluation for the first three named vision benchmarks and task-incremental evaluation for VDD.The language benchmarks use sequences of 5, 10, or 15 tasks; experiments use pretrained ViT-B/16 and T5-large backbones.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • SPACE-LoRA’s vision results were mixed on ImageNet-R but competitive across DomainNet, VTAB, and VDD, reaching 86.05% accuracy on VTAB and 86.26% on VDD with ρ = 0.90.On ImageNet-R, outcomes vary by task count: both configurations slightly exceed InfLoRA in mean final accuracy and forgetting in the 5-task setting, but both have lower final accuracy than InfLoRA and Janus-LoRA in the 10- and 20-task settings.Source: §5.2 · pp. 7–8, Table 1 · p. 7, Table 2 · p. 8
  • SPACE-LoRA achieves the highest final accuracy in the 5- and 10-task language settings, at 71.88% and 73.95%, respectively.In the 15-task setting, it reaches 69.79% accuracy, 2.25 points above InfLoRA, with 2.53 versus 3.91 points of forgetting; it does not have the lowest forgetting in any setting.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 EFFECT OF PROTECTION COVERAGE

  • As ρ rises from 0.75 to 0.99, average protected dimension grows from 90 to 517 and forgetting falls from 7.41 to 3.50 percentage points, while accuracy peaks at 79.09% for ρ = 0.90.The sweep is on 5-task ImageNet-R; accuracy does not improve monotonically with stronger protection.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 ABLATION ON PROTECTION ALLOCATION

  • Fisher-guided allocation yields 2.12 percentage points higher ImageNet-R accuracy and 1.34 points higher language accuracy than uniform allocation.ImageNet-R forgetting is slightly higher with Fisher-guided protection (4.82 versus 4.40 points); the ablation uses ρ = 0.90 for Fisher-based protection.Source: §5.4 · p. 9, Table 5 · p. 9

6 CONCLUSION

SPACE-LoRA suppresses new LoRA branches’ residual responses on selected historical activation directions, with module-wise protection sizing based on Fisher sensitivity. Experiments report reduced forgetting while remaining competitive, though Fisher-coverage sizing does not consistently outperform uniform sizing.

6 CONCLUSION

  • SPACE-LoRA projects each new branch’s input factor onto the orthogonal complement of selected historical activation directions, while a module-wise Fisher-coverage criterion sets protection size.The criterion uses historical loss sensitivity along activation-ordered directions.Source: §6 · p. 9
  • Experiments on vision and language benchmarks show that SPACE-LoRA reduces forgetting while remaining competitive.The coverage analysis shows a trade-off between protecting previous tasks and learning new ones.Source: §6 · pp. 9–10
  • Fisher-coverage sizing does not consistently outperform uniform sizing, while protecting important historical activation directions is effective for continual low-rank adaptation.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools to assist with manuscript drafting and polishing, with the authors reviewing and revising all AI-assisted text and taking responsibility for the final content.

AI USE STATEMENT

A FORMAL PROPERTIES OF THE HARD PROTECTION CONSTRAINT

The appendix characterizes SPACE-LoRA’s hard input-space projection and the historical statistics used to select and size protected directions. Its guarantees concern module-level responses; they do not alone guarantee zero network-level forgetting.

A.1 EXACT NULLIFICATION ON THE PROTECTED SUBSPACE

  • The projected input factor makes the new branch produce exactly zero response for inputs in the selected protected subspace, irrespective of the output factor.This follows from parameter projection rather than a soft penalty, first-order loss approximation, or approximately orthogonal optimization gradient.Source: §A.1 · p. 13
  • This nullification is local to a module and does not guarantee zero network-level forgetting or unchanged downstream activations.Old-task activations need not lie entirely in the protected subspace, and earlier modules can alter later inputs.Source: §A.1 · p. 13

A.2 INTERFERENCE OUTSIDE THE PROTECTED SUBSPACE

  • Outside the protected subspace, the new adapter responds only to the input’s unprotected component, with perturbation magnitude also depending on learned adapter norms.For fixed adapter norms, the module-output perturbation is controlled by how much of the input remains outside the protected subspace.Source: §A.2 · p. 13
  • For a fixed protection dimension, leading eigenvectors of the historical activation Gram matrix minimize past-activation energy left outside the protected subspace.Fisher-based allocation determines how many activation-ordered directions each module retains.Source: §A.2 · pp. 13–14

B CONSTRUCTION OF HISTORICAL STATISTICS

B.1 HISTORICAL ACTIVATION STATISTICS

  • The historical activation Gram matrix provides eigenvalue-ordered candidate directions; Fisher sensitivity sets how far the protected activation-ordered prefix extends rather than reordering it.The prefix’s directions are determined by the activation statistic, while Fisher determines its length for each module.Source: §B.1 · p. 14

B.2 HISTORICAL FISHER SENSITIVITY

  • The input-space empirical Fisher measures historical loss sensitivity along activation directions, and cumulative sensitivity across the prefix underlies the Fisher-coverage ratio.The statistic uses supervised-loss gradients with the current task classifier and observed ground-truth labels at each task endpoint.Source: §B.2 · p. 15

C.1 VISION EXPERIMENTAL SETUP

The vision experiments cover several class-incremental and domain-incremental settings, with task counts, class sets, and splits defined by benchmark. The supplied Table 6 caption refers to language datasets, not vision datasets.

C.1 VISION EXPERIMENTAL SETUP

  • ImageNet-R compares 5-, 10-, and 20-task settings with the same image split and class ordering, isolating changes in sequential learning stages.The overall class set remains fixed across settings.Source: §C.1 · p. 15
  • DomainNet uses five domains and 345 classes split into five disjoint 69-class tasks, with all retained domains represented in every task.Quickdraw is excluded.Source: §C.1 · p. 15
  • VTAB comprises five dataset-specific tasks with 10 classes each, using official data and a 10% per-class training holdout for validation.The resulting split contains 1,615 training, 181 validation, and 8,619 test images.Source: §C.1 · p. 15
  • Table 6 lists datasets used in the language continual-learning experiments in their training order.It is not a vision-experiment table.Source: Table 6 · p. 16
  • VDD uses nine Visual Domain Decathlon datasets, excluding ImageNet, with each task introducing a new domain and label set.The benchmark has 2,128 classes; because official test labels are unavailable, the official validation split is used for testing.Source: §C.1 · p. 16

C.2 LANGUAGE EXPERIMENTAL SETUP

The language experiments use ordered sequences of classification and NLU datasets, with longer task settings extending earlier sequences. Training, data splits, and hyperparameter selection are specified across the experiments.

C.2 LANGUAGE EXPERIMENTAL SETUP

  • Language adaptation uses an ordered sequence of 15 text-classification and NLU datasets; the 5- and 10-task settings use the first five and ten datasets.The 15-task setting uses the complete sequence, so longer settings extend earlier ones without changing prior task order.Source: §C.2 · p. 16
  • For most tasks, datasets contain 1,000 training samples, 200 validation samples, and up to 500 test samples, with non-overlapping validation and test sets when possible.For GLUE and SuperGLUE datasets without public test labels, labeled validation data supplies evaluation splits; RTE and WiC test sets contain 77 and 438 samples.Source: §C.2 · p. 16
  • T5 large experiments use AdamW, constant learning rate, batch size 8, three epochs per task, and rank-8 LoRA on attention query and value projections.The scaling parameter is 16, each task receives a new adapter, and maximum input length is 128 tokens.Source: §C.2 · p. 16
  • The learning rate is selected once on the five-task setting with seed 0 and then reused across methods and sequence lengths without method-specific tuning.Compared methods share data, optimizer, and training schedule; method-specific hyperparameters use official implementation defaults.Source: §C.2 · p. 16

C.3 EVALUATION PROTOCOL

Methods are evaluated after each task under a shared protocol, with final average accuracy and backward transfer reported. SPACE-LoRA freezes prior components, selects protected activation directions, and projects new-branch inputs away from them.

C.3 EVALUATION PROTOCOL

  • All methods use the same task ordering, dataset splits, training schedule, and evaluation protocol; evaluation follows each task and reports final average accuracy and backward transfer.Average accuracy measures final predictive performance, while backward transfer measures changes in performance on previously learned tasks.Source: §C.3 · pp. 16–17
  • For each task, SPACE-LoRA freezes the base model and earlier adapters, then introduces a new rank-r LoRA branch for each module.It later collects endpoint activations on the current task and computes task statistics.Source: §C.3 · p. 18
  • For each module, the algorithm orders historical activation eigenvectors by eigenvalue and selects the smallest protection size meeting the past-sensitivity coverage target, subject to the rank constraint.The selected directions form a projection matrix; when historical Fisher trace is zero, the protection size is set to zero.Source: §C.3 · p. 18
  • During optimization, each new LoRA input factor is projected away from the module's protected subspace; after the task, historical activation and Fisher statistics are updated.The procedure computes G(t) using the current-task classifier and ground-truth labels, updates historical activation statistic H, and accumulates the example count.Source: §C.3 · p. 18

Related papers

Loading 2609.34453v1…