PaperlayerView exact revision on arXiv

Source-linked AI summary

Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning

Yibo Feng

arXiv:2608.26960v1cs.CV

TL;DR

LoRA-based rehearsal-free CIL lacks explicit geometric control over evolving shared and task-specific subspaces, motivating Geo-LoRA. Geo-LoRA adds SPP, ACSA, and MCBO to regulate these subspaces, and it achieves state-of-the-art results across multiple datasets and task lengths.

  • Problem

    Existing LoRA-based CIL methods provide limited explicit control over geometric subspace evolution, allowing shared drift and repeated reuse of task-specific directions.

  • Method

    Geo-LoRA integrates SPP, ACSA, and MCBO into Dual-LoRA to regulate shared and task-specific subspaces through unified geometric objectives.

  • Results

    Geo-LoRA achieves state-of-the-art results across multiple datasets and tasks, including 2.19% and 1.25% accuracy improvements on ImageNet-A with 10 tasks and ImageNet-R with 20 tasks, respectively, using the 1K pretrained model.

  • Takeaways & Limitations

    Geometry-aware regulation provides a principled framework for continual low-rank adaptation without additional adapter types, replay, or additional model capacity.

Abstract

from arXiv · show

Rehearsal-free class-incremental learning (CIL) with LoRA adapters remains challenging because the low-rank subspaces updated across tasks evolve without geometric control, causing unstable shared representations and repetitive collapse of task-specific updates into previously occupied directions. We introduce Geo-LoRA, a geometry-aware framework that explicitly regulates how low-rank subspaces, both shared and task-specific, evolve during continual learning. For the shared branch, Subspace Projection Preservation (SPP) constrains consecutive updates to follow smooth trajectories on the Grassmann manifold, and Adaptive Core-Slack Alignment (ACSA) decomposes transitions into principal and residual components, aligning the former while modulating the latter to balance stability and plasticity. For the task-specific branch, Median-Calibrated Block Overlap (MCBO) imposes a statistical constraint via normalized projection overlap, penalizing excessive reuse to mitigate subspace crowding. These constraints jointly regulate the evolution of all LoRA subspaces across layers and tasks without introducing additional adapter types beyond standard LoRA. Geo-LoRA provides a principled geometric formulation for continual low-rank adaptation and consistently achieves state-of-the-art performance across multiple benchmark datasets and different task lengths.

1 Introduction

Geo-LoRA addresses uncontrolled geometric evolution in LoRA subspaces by adding geometry-aware objectives to a Dual-LoRA continual-learning framework. It reports state-of-the-art performance across datasets and tasks while retaining a small trainable-parameter footprint.

  • Motivation: LoRA-based CIL methods provide limited explicit control over subspace geometry, allowing shared drift and repeated reuse of task-specific directions.These effects undermine the stability–plasticity trade-off through misaligned representations and subspace crowding.
  • Framework: Geo-LoRA uses Dual-LoRA to separate shallow shared adaptation from deep task-specific specialization.The shared branch accumulates transferable structure, while deeper layers support semantic specialization.
  • Contribution: Geo-LoRA introduces geometric objectives rather than a new LoRA architecture or additional adapter types.The framework is designed to stabilize and structure learning dynamics within the existing architecture.
  • Geometric objectives: SPP, ACSA, and MCBO regulate shared and task-specific subspace evolution through projection-based geometric objectives.They respectively reduce shared drift, align principal directions while moderating residuals, and penalize statistically excessive reuse.
  • Results: Geo-LoRA achieves state-of-the-art performance while maintaining a small number of trainable parameters.The introduction presents this result through the ImageNet-R comparison in Figure 1.
  • Contributions: The paper frames geometric subspace evolution as a principled perspective for continual low-rank adaptation.It integrates SPP, ACSA, and MCBO under a unified projection-based principle for exemplar-free continual learning.

2 Related work

Related work positions LoRA as a low-dimensional subspace adaptation method whose task-dependent directions can interfere in continual settings. Existing remedies regulate parameters, overlap, sparsity, rank, or module composition rather than directly controlling subspace evolution.

  • PEFT for CIL: PEFT methods support continual adaptation with frozen backbones, limiting full-parameter interference across tasks.The related-work discussion includes adapters, prompt tuning, and LoRA among these approaches.
  • LoRA subspaces: LoRA represents adaptation through low-rank matrices interpreted as new directions in task-dependent feature subspaces.In continual or multi-task settings, these directions may rotate, collapse, drift, or interfere.
  • Existing remedies: Prior methods address these effects with orthogonality, sparsity, adaptive rank allocation, or shared/task-specific branch designs.The passage contrasts these approaches with direct regulation of subspace evolution.

3 Preliminaries

The preliminaries describe rehearsal-free CIL with frozen pre-trained ViTs and lightweight LoRA adapters, then reinterpret Dual-LoRA branches as evolving subspaces regulated by Geo-LoRA’s three constraints.

  • CIL setting: Rehearsal-free CIL learns disjoint-label tasks sequentially without storing past samples, while the pre-trained ViT backbone remains frozen.Lightweight adapters inserted into Transformer blocks provide continual adaptation.
  • LoRA formulation: LoRA implements each adapter as a rank-r update to a frozen weight matrix.The displayed formulation expresses the adapted transformation as W plus AB, with r much smaller than the matrix dimensions.
  • Limitation: Allocating task-specific LoRA modules for every task causes linear parameter growth and limited knowledge sharing.This motivates a structure that distinguishes shared and task-specific adaptation.
  • Dual-LoRA structure: Geo-LoRA uses a shared LoRA module in the first l blocks and task-specific modules in deeper blocks.The shallow subspace should evolve smoothly, while deep subspaces should expand or reuse prior directions in a controlled way.
  • Geometric regulation: SPP, ACSA, and MCBO collectively regulate the evolution of shared and task-specific LoRA subspaces.The framework applies these constraints across tasks without adding inference-time overhead.

4 Method

Geo-LoRA treats shared and task-specific LoRA updates as evolving subspaces and regulates each branch with geometry-aware constraints. SPP and ACSA control shared-subspace stability and plasticity, while MCBO limits excessive reuse in deep task-specific updates.

  • Architecture: Geo-LoRA uses a dual-LoRA architecture with shallow shared modules for cross-task preservation and deep task-specific modules for semantic specialization.The backbone structure separates transferable shared representations from localized task-level updates.
  • Shared-subspace regulation: SPP represents concatenated shallow-layer updates through basis-invariant orthogonal projectors and minimizes consecutive projector distance to limit shared-subspace drift.The projector distance is the squared chordal distance on the Grassmann manifold and is independent of basis choices.
  • Shared-subspace regulation: ACSA separates shared-subspace transitions into principal core directions and residual slack directions, aligning the core while adaptively modulating slack to balance stability and plasticity.Its weighting emphasizes core preservation when alignment is strong and shifts toward controlled expansion when alignment weakens.
  • Task-specific regulation: MCBO measures normalized overlap between each current deep-layer update and the accumulated historical task-specific subspace, then calibrates excessive reuse with a task-wise median.Only blocks exceeding their typical overlap level are softly penalized, accommodating distinct reuse levels across deep blocks.
  • Optimization: The unified objective combines current-task cross-entropy with three training-time geometric regularizers and adds no inference-time overhead or adapter types beyond standard LoRA.The geometric terms regulate shared and task-specific subspace evolution during optimization.

5 Experiments

Geo-LoRA is evaluated across diverse rehearsal-free CIL benchmarks, task lengths, backbones, computational settings, and geometric diagnostics. It consistently outperforms baselines while its components jointly regulate subspace drift, alignment, and overlap.

  • Experimental Setup: Geo-LoRA is evaluated on six rehearsal-free CIL benchmarks spanning natural, shifted, fine-grained, and heterogeneous visual domains.The evaluation includes CIFAR-100, ImageNet-R, ImageNet-A, CUB200, VTAB, and OmniBenchmark.
  • Experimental Setup: The comparison covers representative prompt-tuning, adapter, and LoRA-based rehearsal-free CIL methods.Baselines include L2P, DualPrompt, CODA-Prompt, InfLoRA, SD-LoRA, ACMap, SEMA, BiLoRA, and CL-LoRA.
  • Experimental Results: Geo-LoRA consistently surpasses baselines across task numbers and both 1K- and 21K-pretrained ViT backbones.It remains the top performer in short-horizon 5-task and long-horizon 40-task ImageNet-A/R experiments.
  • Experimental Results: 2.19% and 1.25% accuracy improvements are reported on ImageNet-A with 10 tasks and ImageNet-R with 20 tasks, respectively, using the 1K-pretrained model.The reported gains occur under distribution shifts involving artistic transformations and adversarial perturbations.
  • Efficiency: Geo-LoRA incurs only a slight increase in training time while inference time remains nearly unchanged relative to baselines.The comparison uses ImageNet-A with the 21K-pretrained backbone under 10-task and 20-task settings.
  • Representation Dynamics Analysis: SPP suppresses shared-subspace drift, ACSA improves and maintains core alignment, and MCBO reduces excessive deep task-specific overlap.These diagnostics directly evaluate the geometric dynamics targeted by the method.
  • Ablation Study: Ablations show that SPP, ACSA, and MCBO provide complementary benefits, and removing any one causes a clear performance drop.The results support their joint interaction rather than reliance on a single objective.
  • Ablation Study: The Grassmann projector normalization consistently yields higher accuracy, indicating greater geometric stability and robustness under low-rank updates.This comparison is reported in the SPP normalization study on ImageNet-R with 20 tasks.

6 Conclusion

Geo-LoRA reframes continual LoRA adaptation as geometry-aware evolution of shared and task-specific subspaces. Its complementary constraints deliver strong accuracy and forgetting reductions without additional model capacity or replay.

  • Conclusion: Geo-LoRA regulates shared and task-specific low-rank structures through SPP, ACSA, and MCBO.The framework preserves global consistency in shallow layers while managing localized plasticity in deep layers.
  • Conclusion: The method delivers strong accuracy and forgetting reductions over prior adapter-based CIL methods without additional model capacity or replay.The conclusion presents continual adapter tuning as management of subspace trajectories on evolving low-rank manifolds.
Loading 2608.26960v1…