PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Continual Learning to 300+ Tasks with Bi-Level Routing Mixture-of-Experts

Meng Lou, Yunxiang Fu, Yizhou Yu

arXiv:2602.03473v2cs.LGcs.CV

TL;DR

Continual learners have mainly been evaluated on short task sequences, while learning discriminative, comprehensive representations across hundreds of tasks remains difficult. CaRE addresses this with bi-level routing and shows effectiveness across diverse datasets, particularly for sequences of 100–301 tasks.

  • Problem

    Existing continual-learning methods have mostly been validated on limited task sequences, leaving long-sequence performance and fine-grained cross-task discrimination unclear.

  • Method

    CaRE uses bi-level routing to dynamically select task-specific routers and aggregate task-specific and shared experts at each hidden layer.

  • Results

    Experiments across diverse datasets demonstrate CaRE’s effectiveness, particularly on long-sequence continual learning with 100–301 tasks.

  • Takeaways & Limitations

    CaRE provides a continual-learning approach evaluated at a scale reaching more than 300 non-overlapping tasks.

  • Takeaways & Limitations

    Model complexity grows linearly with the number of tasks because new efficient modules are appended, while longer or unbounded sequences are outside the study’s computational scope.

Abstract

from arXiv · show

Continual learning, especially class-incremental learning (CIL), on the basis of a pre-trained model (PTM) has garnered substantial research interest in recent years. However, how to effectively learn both discriminative and comprehensive feature representations while maintaining stability and plasticity over very long task sequences remains an open problem. We propose CaRE, a scalable {C}ontinual Le{a}rner with efficient Bi-Level {R}outing Mixture-of-{E}xperts (BR-MoE). The core idea of BR-MoE is a bi-level routing mechanism: a router selection stage that dynamically activates relevant task-specific routers, followed by an expert routing phase that dynamically activates and aggregates experts, aiming to inject discriminative and comprehensive representations into every intermediate network layer. On the other hand, we introduce a challenging dataset, OmniBenchmark-1K, for CIL performance evaluation on very long task sequences with hundreds of tasks. Extensive experiments show that CaRE demonstrates leading performance across a variety of datasets and task settings, including commonly used CIL datasets with classical CIL settings (e.g., 5-20 tasks). To the best of our knowledge, CaRE is the first continual learner that scales to very long task sequences (ranging from 100 to over 300 non-overlapping tasks), while outperforming all baselines by a large margin on such task sequences. We hope that this work will inspire further research into continual learning over extremely long task sequences. Code and dataset are publicly released at https://github.com/LMMMEng/CaRE.

1. Introduction

CaRE addresses catastrophic forgetting and representation limitations in class-incremental learning by combining bi-level routing with task-specific components and introducing OmniBenchmark-1K for long task sequences. Experiments report strong gains over PTM-based CIL baselines across evaluations spanning 100 to 301 tasks.

  • Continual learning targets streaming, non-stationary environments where conventional systems suffer catastrophic forgetting as new information overwrites historical knowledge.
  • Class-incremental learning requires continuously learning unseen classes while preserving knowledge of previously seen classes, increasingly using pre-trained models instead of training from scratch.
  • Long task sequences challenge existing methods because task-specific representations may lack discrimination across semantically related classes, while intermediate layers require local decisions to retrieve complementary historical knowledge.
  • CaRE introduces BR-MoE, which learns a class perceptron, router network, and adapter at each incremental step and uses two-stage routing to activate relevant experts.
  • OmniBenchmark-1K contains 1,000 classes, around 190,000 images, and 21 visual realms for comprehensive long-sequence CIL evaluation.
  • At 100 tasks, CaRE surpasses TUNA by 8.23% in last accuracy (AB), while at 151 tasks it outperforms MIN by 8.68% in AB.

2. Related Work

Related work in class-incremental learning spans regularization-, replay-, and prompt-based approaches, with pre-trained models emerging as a promising foundation. Mixture-of-Experts methods extend this direction by dynamically combining specialized experts through learned routing.

  • Class-Incremental Learning: CIL methods are organized into regularization-based, replay-based, and other major lines, while PTM-based CIL uses embedded prior knowledge to mitigate catastrophic forgetting and improve performance.The passage identifies PTM-based CIL as a prospective direction because pre-trained models provide powerful prior knowledge.
  • Class-Incremental Learning: Prompt-based PTM methods progressively improve task-specific adaptation through learnable prompt retrieval and enhanced prompt-tuning designs.L2P introduces a learnable prompt pool; DualPrompt, DAP, and CODA-Prompt further enhance prompt tuning, while APER uses a shared adapter with prototype-based classification.
  • Mixture-of-Experts: MoE combines multiple specialized experts with dynamic gating, motivating continual-learning methods that use task-specific routers and expert sets on top of pre-trained models.MoE-Adapter trains a dedicated router and experts for each task, and MoE-Adapter++ further enhances this design.

3. Method

CaRE builds on a pre-trained ViT with Bi-Level Routing Mixture-of-Experts modules in every building block, updating only newly introduced components while preserving earlier task parameters. BR-MoE selects relevant task routers using entropy and then routes features through complementary experts, including a shared expert, with intermediate-layer supervision supporting local knowledge retrieval.

  • Framework: CaRE integrates BR-MoE into every pre-trained ViT building block and updates only BR-MoE components during incremental training.Parameters learned from previous tasks remain frozen while new tasks are learned.
  • BR-MoE Components: Each BR-MoE module contains task-specific class perceptrons, routers, and experts, plus one shared expert for cross-task knowledge.The shared expert is implemented as a momentum-based adapter, fully trained on the initial task and updated by EMA thereafter.
  • Bi-Level Routing: BR-MoE first selects Top-M routers with the smallest entropy, then each selected router activates Top-K experts with the largest gating scores.During training, the latest-task router is always active and the remaining M−1 routers are selected dynamically; default settings are M=2 and K=3.
  • Training Objectives: Each intermediate-layer class perceptron receives direct supervision alongside final classification, enabling layer-specific decision-making and customized knowledge retrieval.The auxiliary class-perceptron loss is aggregated across layers and scaled by λ, set to 1 by default.
  • Router Selection: Entropy ranks task-specific prediction distributions by uncertainty, prioritizing routers associated with the most likely task.Lower entropy indicates greater confidence that the input belongs to a corresponding task.

4. A Benchmark for Long Task Sequence Class-Incremental Learning

OmniBenchmark-1K is a 1,000-class benchmark designed to evaluate class-incremental learning over long task sequences. It balances classes across diverse visual realms and provides sufficient per-class training data to support scalable, challenging evaluation.

  • Benchmark construction: OmniBenchmark-1K curates 1,000 classes from OmniBenchmark-V2 across its thematic realms, excluding categories overlapping potential pre-training datasets.Classes are sampled roughly evenly across realms using a fixed random seed of 1993.
  • Dataset statistics: The test set contains 19,849 images, averaging 19 images per class, with each class represented by 17 to 20 images.Test samples are directly extracted for the selected classes from the original test portion.
  • Dataset statistics: The training set contains 168,718 images across 1,000 classes, averaging 169 images per class and ranging from 100 to 403 samples per class.The complete dataset contains 188,569 images, including 19,849 test images.
  • Benchmark motivation: The benchmark combines broad visual diversity, many classes, and sufficient training samples to mitigate overfitting in long-sequence CIL evaluations.The authors position OmniBenchmark-1K as a challenging and scalable testbed for long-sequence CIL.

5. Experiments

CaRE achieves leading performance across very long, moderately long, and classical short CIL task sequences, including scaling to over 300 non-overlapping tasks. Ablation and efficiency analyses support the effectiveness of its entropy-based routing and bi-level MoE design.

  • Very long task sequences: 4.67% and 4% are CaRE’s AB gains over MIN and MOS in the 100-task setting, while its gain over TUNA reaches 8.32% at 200 tasks.CaRE also improves AB by 6.02% over APER-Adapter in the 151-task setting and retains an advantage at 301 tasks.
  • Moderately long task sequences: CaRE improves AB over EASE by 12.59% on 60-task OmniBenchmark-V1 and 10.68% on 50-task ObjectNet, and over SSIAT by 2.51% and 10.8% on ImageNet-R and ImageNet-A.These results support robust performance on established benchmarks with moderately long task sequences.
  • Classical short task sequences: CaRE maintains leading short-sequence performance, including AB gains of 1.97% over MoAL on 10-task CIFAR-100 and 5.24% over SLCA on 20-task ObjectNet.It also improves over SD-LoRA by 3.19% and 8.82% on 10-task ImageNet-R and ImageNet-A, respectively, and remains superior on 5-task VTAB.
  • Ablation studies: All alternative router-selection strategies cause notable performance degradation, validating entropy-based dynamic selection of layer-wise task identities.The alternatives include prototype-based selection, an autoencoder, a cosine head, and a softmax-maximum strategy.
  • Efficiency analysis: CaRE improves AB over MOS by 4% with approximately 80% fewer average trainable parameters and 95% lower inference latency.Compared with MIN, it improves AB by 4.67% with comparable inference latency and fewer trainable parameters.

6. Conclusion

The paper proposes CaRE, a PTM-based continual learner with efficient BR-MoE for dynamic knowledge retrieval and aggregation, and introduces OmniBenchmark-1K for scalable CIL evaluation across hundreds of tasks.

  • Contributions: CaRE combines a pre-trained-model-based continual learner with efficient BR-MoE and bi-level routing at each hidden layer.The routing mechanism enables dynamic knowledge retrieval and aggregation.
  • Contributions: OmniBenchmark-1K is a challenging long-sequence CIL benchmark designed for scalable evaluation with hundreds of tasks.

A. Appendix · A.1. More Experimental Comparisons

The appendix evaluates CaRE’s robustness under varied task orders and pre-trained weights on OmniBenchmark-1K. CaRE remains competitive across these settings, surpassing baselines under the ViT-B/16-IN1K 100-task evaluation.

  • A.1. More Experimental Comparisons: CaRE’s robustness is evaluated under different task orders on OmniBenchmark-1K for long-sequence continual learning.The experiments use 100-task results as a basis for additional task-sequence evaluations.
  • A.1. More Experimental Comparisons: Three distinct task sequences are generated using random seeds 1990, 1996, and 1999.Performance is evaluated across these additional task-order runs.
  • A.1. More Experimental Comparisons: Mean and standard deviation are reported across the three random-seed task-order runs.This summarizes performance variability under different task sequences.
  • A.1. More Experimental Comparisons: The task-order comparison includes PTM-based CIL methods such as MOS and TUNA.These methods are examined for robustness in long-sequence evaluations.
  • A.1. More Experimental Comparisons: All methods are additionally evaluated with the ViT-B/16-IN1K pre-trained model under the 100-task B0 Inc10 setting.The evaluation uses OmniBenchmark-1K and complements experiments with the ViT-B/16-IN21K backbone.
  • A.1. More Experimental Comparisons: CaRE consistently outperforms all baselines by a clear margin under the alternative pre-trained weights.This demonstrates robustness across different pre-trained models.
  • A.1. More Experimental Comparisons: 6.99%: CaRE surpasses the strong MIN baseline by 6.99% in AB.This result is reported for the ViT-B/16-IN1K, 100-task B0 Inc10 evaluation on OmniBenchmark-1K.

A.2. More Ablation Studies

Additional ablations show that BR-MoE performs best with layer-specific routing decisions, multiple activated routers and experts, and a stable shared-expert update. Performance is relatively insensitive to the class-perceptron loss weight, while adapter configuration affects task-specific and cross-task knowledge representation.

  • Effect of the local decision scope: Performance degrades steadily as Scope increases, indicating that each BR-MoE layer benefits from customized knowledge retrieval.Scope = 1 selects routers independently, whereas larger scopes reuse selections across successive layers.
  • Effect of the number of activated router networks: The study evaluates M=1, 2, 3, and 4 router networks per forward pass, including a single-router model with twice as many adapters.The default setting is M=2.
  • Effect of the number of activated experts: Performance improves significantly when increasing K from 1 to 2, while larger K values can restrict early tasks because fewer experts existed during their training.The default setting is K = 3, with each router activating its Top-3 experts; alternatives include K ∈{1, 2, 6, 18}.
  • Analysis of the class perceptron: Removing the class-perceptron auxiliary loss causes a slight performance drop, while varying its scaling factor produces minimal performance variation.The tested scaling factors are 0.5, 1.5, and 2.0.
  • Effect of different configurations of adapter: A task-specific expert with 16 channels yields the highest ¯ A and AB, while µ = 0.999 provides the best trade-off for stable cross-task knowledge accumulation.The task-specific bottleneck sizes tested are {8, 16, 32, 64}; µ controls shared-expert updating, with larger values retaining more earlier-task knowledge.

A.3. More Analytical Experiments

Visualization analyses of CaRE’s bi-level routing show that it combines task-specific and complementary knowledge while making layer-dependent routing decisions. Early layers reuse broadly shared components, whereas deeper layers increasingly specialize by task, supporting discriminative and comprehensive representations and test-time integration across tasks.

  • Visualization analysis of bi-level routing: For a Corgi image from Task 96, bi-level routing combines task-specific facial cues with complementary ear-shape and texture cues for a more accurate representation.Router 96 emphasizes discriminative facial characteristics, while router 53 captures shared details; their feature maps aggregate both cue types.
  • Visualization of router and expert activation patterns: Across all 100 tasks, activation patterns reveal a hierarchical structure in which routing and expert usage vary across network layers.The analysis uses the model trained under the 100-task evaluation protocol and inference over the entire validation set.
  • Visualization of router and expert activation patterns: Early layers activate a small subset of routers and experts frequently across many tasks, reflecting shared low-level visual commonalities such as edges and textures.Layers 3 and 6 show broadly reused components that act as robust feature extractors for shared elements.
  • Visualization of router and expert activation patterns: Deeper layers exhibit sparser, more task-specific activation, with components associated with the ground-truth task activated at substantially higher intensity.This pattern appears in layers 9 and 12 and indicates specialization for high-level semantic representations.
  • Visualization of router and expert activation patterns: The model integrates knowledge from different tasks at every layer, producing discriminative and comprehensive features rather than relying on a single router or expert.The cited ablation study reports notable performance degradation when routing is restricted to a single router or expert.
  • Visualization of router and expert activation patterns: Bi-level routing enables layer-specific decisions and incorporates later-task knowledge during inference on earlier tasks, regardless of task order.These observations support dynamic layer-wise local decision-making and test-time flexibility as contributors to CaRE’s superior long- and short-sequence CIL performance.

A.4. Limitations

CaRE’s task-specific module expansion causes model complexity to grow linearly with the number of tasks, limiting scalability to even longer or unbounded sequences. The authors propose architectural simplification and extending long-sequence evaluation to vision-language models as future directions.

  • Scalability limitation: Model complexity grows linearly with the number of tasks because new efficient modules are appended for each task.This limitation is shared with existing popular PEFT-based approaches.
  • Future directions: Further architectural simplification without sacrificing performance is identified as a promising research direction.The authors frame their method and dataset as a foundation for future research on extremely long task sequences.
  • Future directions: Extending the long-sequence evaluation protocol to vision-language models is another proposed direction.The cited examples include Radford et al. (2021) and Yang et al. (2025).
Loading 2602.03473v2…