PaperlayerView exact revision on arXiv

Source-linked AI summary

CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

Yunxiang Fu, Meng Lou, Zicheng Liao, Yizhou Yu

arXiv:2609.17026v1cs.LGcs.CV

TL;DR

Continual learning must scale to long task sequences while balancing new-task plasticity against retention of earlier knowledge. CLARE learns sparse task-critical masks, fine-tunes only selected parameters, and accumulates updates in one shared adapter. It reaches 66.88% final accuracy after 100 OmniBenchmark-1k tasks and outperforms strong baselines, but its finite adapter capacity limits indefinite scaling.

  • Problem

    Pretrained-model continual-learning methods often lose performance on long task sequences because inter-task interference and reduced plasticity make retaining earlier knowledge difficult.

  • Method

    CLARE learns a sparse task-critical parameter mask with a sparsity-inducing objective, then performs mask-constrained fine-tuning and accumulates updates in a shared adapter.

  • Results

    CLARE achieves 66.88% final accuracy after 100 tasks on OmniBenchmark-1k, surpassing the strongest adapter-based baseline by 4.64%.

  • Takeaways & Limitations

    CLARE provides a shared-adapter continual-learning approach that scales across long and standard task sequences without task-specific routing at inference.

  • Takeaways & Limitations

    The finite shared-adapter pool cannot scale indefinitely to very large task streams such as 10k tasks because unused coordinates eventually run out.

Abstract

from arXiv · show

Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.

1. Introduction

CLARE addresses the difficulty of scaling pretrained-model class-incremental learning by allocating sparse, task-critical parameter updates within a shared adapter. Its two-stage mask learning and constrained fine-tuning achieve strong long-sequence and standard benchmark results.

  • Long task sequences expose a trade-off in pretrained-model continual learning: new-task updates can overwrite parameters important for earlier knowledge, reducing performance as interference accumulates.
  • Sparse fine-tuning is motivated by long-tailed update magnitudes and near-full-fine-tuning performance, with only 8.5K parameters receiving updates of at least 0.05 in the cited analysis.
  • CLARE first learns a binary mask for task-critical parameters with a sparsity-inducing objective, then fine-tunes only the selected parameters while freezing the remainder.
  • 66.88% final accuracy after 100 tasks on OmniBenchmark-1k, surpassing the strongest adapter-based baseline by 4.64%.CLARE also surpasses SD-LoRA by over 38% on the same benchmark.
  • CLARE achieves the highest final accuracy across all six evaluated 10- and 20-task settings, with gains of 5.40% on ImageNet-A and 8.40% on ObjectNet over the strongest baseline.
  • The framework accumulates all tasks in a shared adapter without task-specific routing or task identity at inference.

2. Related work

Prior pretrained-model continual-learning methods use prompts, task-specific adapters, merged updates, or sparse parameter subsets to manage interference. CLARE instead uses sparsity during training to allocate updates, then keeps one shared adapter without per-task masks or routing at inference.

  • Existing parameter-efficient methods combine task-specific updates into one inference model using constraints, merging rules, or related mechanisms to reduce inter-task interference.
  • Sparse continual-learning methods commonly select, prune, or allocate different parameter subsets across tasks, including binary-mask and progressive-pruning approaches.
  • CLARE’s core novelty is using sparsity as a training-time allocation mechanism rather than an inference-time selection mechanism.
  • Unlike methods that retrieve prompts or route task-specific modules, CLARE keeps no per-task mask at inference and uses the same shared adapter for every input.
  • After each task, CLARE merges the sparse update into the shared adapter, while earlier coordinates remain active so prior knowledge is reused rather than gated.

3. Method

CLARE performs exemplar-free class-incremental learning with a shared adapter by allocating sparse, capacity-aware parameter subsets to successive tasks. It discovers task-critical coordinates under L1 regularization, then trains only the selected coordinates before accumulating each sparse update in the shared adapter.

  • 3.1. Problem Definition: CLARE uses a shared adapter while keeping the pretrained ViT backbone frozen, updating only lightweight adapters and the classifier.The model receives disjoint new classes sequentially without replaying previous-task samples and must classify all seen classes.
  • 3.3. Two-Stage Sparse Update Optimization: Stage 2 resets the selected coordinates to their original values and optimizes only them without the L1 term, while nonselected coordinates remain fixed.Separating mask discovery from task learning prevents the sparsity objective from weakening classification learning.
  • 3.2. Capacity-Aware Sparse Adapter Learning: CLARE reserves previously selected adapter coordinates and restricts each new task to a sparse subset of the remaining free coordinates.This capacity-aware allocation reduces direct overwriting of parameters assigned to earlier tasks and preserves capacity for future tasks.
  • 3.2. Capacity-Aware Sparse Adapter Learning: For ρ = 0.95 and 100 tasks, the geometric schedule uses approximately 99.4% of adapter capacity rather than repeatedly assigning the same coordinates.The schedule allocates 5% of the coordinates still available to each task, so capacity consumption slows over the sequence.
  • 3.4. Sequential Sparse Update and Inference: After each task, CLARE adds the sparse task update to the shared adapter, freezes the updated coordinates, and starts the next task from the accumulated adapter state.This sequential accumulation combines a single adapter with reduced direct overlap among task updates.
  • 3.3. Two-Stage Sparse Update Optimization: Stage 1 applies an L1-regularized objective to free coordinates, then scores coordinates by the magnitude of their movement and selects the task budget by global top-k ranking.The sparsity penalty keeps most updates near zero, while larger movements identify coordinates treated as more useful for the current task.

4. Experiments

CLARE is evaluated in exemplar-free CIL across long and standard task sequences, where it generally achieves the strongest final accuracy and remains robust to design choices, sparsity ratios, seeds, and adapter width.

  • Standard CIL Evaluation: Across six standard CIL settings, CLARE achieves the highest final accuracy, with gains of 5.40 points on ImageNet-A and 8.40 points on ObjectNet over the strongest baselines.The compared settings use 10- and 20-task protocols across ImageNet-R, CIFAR-100, ImageNet-A, and ObjectNet.
  • Ablation Studies: 13.71 percentage points are lost on 100 tasks when each task receives a constant predefined parameter portion, showing the importance of capacity-aware sparsity.The ablation compares CLARE with a fixed sparsity allocation on ImageNet-R with 10 tasks and OmniBenchmark-1k with 100 tasks.
  • Ablation Studies: A sparsity ratio of 90% can improve 10-task performance but substantially harms 100-task performance because too few parameters remain for new-task learning.Ratios from 95% to 97.5% are comparatively robust, with declines no larger than 0.38% for 10 tasks and 0.41% for 100 tasks.
  • Ablation Studies: Removing sparse masking causes near-zero final accuracy, while random masks reduce accuracy by 0.51% on ImageNet-R and 5.65% on OmniBenchmark-1k.Independent tuning also drops final accuracy to 32.33% and 11.03%, indicating that shared knowledge accumulation and sparse masking jointly matter.
  • Robustness and Efficiency: CLARE is robust to random class orders and slightly faster than SD-LoRA on 50-task ImageNet-A, taking 4.37 versus 4.41 hours on one NVIDIA L40 GPU.It uses one shared adapter without task identity or routing at inference, while Stage 2 updates only selected coordinates.
  • Ablation Studies: Across an eight-fold adapter-width range, final-accuracy spread is only 0.27 points; d = 16 changes AT by −0.11 points and d = 128 by +0.13 points.The result suggests that assigning sparse subsets, rather than simply enlarging the adapter, drives the long-sequence gains.

5. Limitations

CLARE’s capacity grows with task count but is finite, so the current adapter cannot support arbitrarily long sequences without expansion. The reported ablations also show that performance is relatively insensitive to adapter width across a wide range.

  • Capacity limitation: After 100 tasks, the default adapter is about 99.4% full, and the finite coordinate pool limits CLARE’s support for substantially longer sequences without expansion.The authors propose freezing a saturated adapter and starting a fresh one or increasing the bottleneck, leaving such expansion to future work.
  • Capacity limitation: The geometric capacity schedule slows coordinate consumption, but remaining free coordinates eventually run out as the number of tasks increases.The method is designed without knowing the sequence length in advance, yet its current capacity remains finite.
  • Scope: CLARE’s robustness to longer sequences is demonstrated through 100 tasks, whereas prior SD-LoRA evaluations mostly focus on 10 to 20 tasks.This establishes a broader demonstrated range, not unlimited scalability.

6. Conclusion

The paper concludes that CLARE scales class-incremental learning by accumulating sparse, non-overlapping task updates in one adapter. Experiments across long-sequence and standard benchmarks consistently show higher final accuracy than representative continual-learning methods, while the capacity schedule remains a key boundary.

  • Conclusion: CLARE accumulates sparse updates selected from previously unassigned coordinates, avoiding task-specific routing or stored task-specific parameters at inference.This design targets long task sequences where storing per-task modules becomes impractical.
  • Conclusion: CLARE consistently outperforms representative continual-learning methods on both long-sequence and standard benchmarks.The conclusion summarizes results across the paper’s evaluated settings without reducing them to a single benchmark.
  • Limitations: The geometric capacity schedule eventually saturates the adapter, limiting the maximum sequence length without expanding adapter capacity.Dynamic sparsity budgets and extensions beyond vision tasks with a ViT backbone remain open directions.
Loading 2609.17026v1…