PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Hongwei Zhao, Rui Liu, Yansong Liu

arXiv:2609.39839v1cs.LG

TL;DR

CIL must learn new classes while preserving knowledge of earlier ones, but PEFT approaches still face interference and unreliable module–sample alignment. DLEPEM combines task-specific LoRA-Experts with prototype-ensemble matching, and reports leading performance across evaluated CIL benchmarks. Its expert bank and prototype dictionary grow with tasks and classes.

  • Problem

    CIL must continually recognize new classes while retaining prior knowledge, while PEFT-based methods still face cumulative interference and suboptimal module–sample alignment.

    Source: §1 · p. 2

  • Method

    DLEPEM trains a dedicated LoRA-Expert for each incremental task and uses frozen-PTM and task-adaptive expert prototypes for task-level matching.

    Source: §1 · p. 2, §4 · p. 5, §4.1 · p. 7

  • Results

    DLEPEM achieves leading performance among evaluated methods on six challenging CIL benchmarks under the evaluated protocols.

    Source: §1 · p. 3, §6 · p. 19

  • Takeaways & Limitations

    Under evaluated protocols, the dual design supports a more favorable empirical stability-plasticity trade-off.

    Source: §6 · p. 19

    The expert bank grows with tasks and the prototype dictionary with seen classes; stricter edge deployment or much longer streams may require compression or pruning.

    Source: §5.5 · p. 18, §6 · p. 20

Abstract

from arXiv · show

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuning with pre-trained models reduces parameter overhead but can suffer from cumulative interference and suboptimal alignment between inference samples and specialized modules. We propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a two-stage rehearsal-free framework. DLEPEM allocates a task-specific LoRA-Expert for each incremental task to reduce cross-task interference, then combines frozen pre-trained-model prototypes with task-adaptive LoRA-Expert prototypes for reliable task-level discrimination. Experiments on standard CIL and Few-Shot CIL benchmarks demonstrate strong performance under the evaluated protocols.

1. Introduction

CIL must balance learning new classes with retaining prior knowledge [3,4], yet PEFT-based approaches still face interference and module–sample mismatches. DLEPEM addresses these challenges with task-specific LoRA-Experts and prototype-ensemble matching.

1. Introduction

  • Existing approaches face cumulative interference: shared prompts can overwrite earlier knowledge, constrained LoRA can limit new-task plasticity, and fusion can degrade old and new knowledge [4] [7] [8] [11].Fixed PTM selection can also yield suboptimal activations and degraded predictions under substantial domain shifts [7] [12].Source: §1 · p. 2
  • The paper asks whether stability-plasticity dynamics and module–sample matches can be improved together to mitigate catastrophic forgetting in CIL.CIL arises as novel categories arrive continuously, and learning new classes can disrupt existing representations.Source: §1 · pp. 1–2
  • DLEPEM allocates one task-specific LoRA-Expert per incremental task, trains only the current expert, freezes prior experts, and matches samples using frozen-PTM and domain-specific prototypes.Prototype-ensemble matching is intended to improve expert retrieval when PTM-only matching is unreliable under downstream domain shift.Source: §1 · pp. 2–3
  • DLEPEM achieves leading performance among evaluated methods on six challenging CIL benchmarks under the evaluated protocols.The paper also reports that DLEPEM-MLP retains competitive results while exploring alternative expert integration strategies.Source: §1 · p. 3

2. Related Work

Related work spans CIL strategies, PEFT modules, and expert-routing designs. DLEPEM distinguishes itself through task-wise expert isolation and prototype-based task-level retrieval.

2.1. Class-Incremental Learning

  • Regularization, rehearsal, and dynamic networks address CIL differently, with trade-offs involving domain shifts, buffer constraints, memory overhead, or reliance on old data [14] [16] [17].CIL requires recognizing new classes while retaining discriminability for previously learned classes.Source: §2.1 · p. 3

2.2. PEFT-Based CIL

  • PEFT-based CIL includes prompt retrieval [7] [8] [12] [18] and adapter- or LoRA-based methods [4] [10] [11], whose approaches trade retrieval reliability, stability, and plasticity.Prompt retrieval depends on separating tasks in PTM feature space, while constraint-based LoRA strategies may restrict adaptation to new classes.Source: §2.2 · p. 3

2.3. Mixture-of-Experts and Expert Retrieval

  • MoE methods commonly route inputs among a fixed or predefined expert set, while open-ended CIL makes expert life cycle and retrieval central design choices [13,19–21].A fixed pool may not fit unpredictable task streams, and a learned soft gate may show task-recency bias when trained only on the current task.Source: §2.3 · pp. 3–4

2.4. Our Approach

  • DLEPEM allocates and trains one LoRA-Expert per task while freezing historical experts, then uses top-1 retrieval from an append-only dictionary of PTM and router-domain prototypes.This replaces learned soft gating with prototype matching and separates old-task preservation through parameter isolation from module-identity inference.Source: §2.4 · p. 4

3. Preliminaries

The preliminaries formulate rehearsal-free CIL as sequential task learning with disjoint task label spaces. They also introduce the MoE and LoRA components used by the method.

3. Preliminaries

  • CIL presents sequential tasks with mutually exclusive label spaces; under the rehearsal-free setting, training uses only current-task data and evaluation covers all classes seen so far.The training objective minimizes empirical risk on the current task’s training set.Source: §3 · p. 4
  • A typical MoE layer has expert networks and an input-dependent router that produces a soft selection over experts and weights their outputs.In Transformer architectures, MoE layers can replace standard FFN blocks to route representations.Source: §3 · pp. 4–5
  • LoRA fine-tunes a pre-trained weight matrix through an additive low-rank update, reducing trainable parameters while maintaining expressiveness [22].The low-rank factors have dimensions din × r and r × dout, with r much smaller than both input and output dimensions.Source: §3 · p. 5

4. The Proposed Method

DLEPEM targets within-task prediction and module-identity inference with dedicated task-specific LoRA-Experts and prototype-ensemble matching.

4. The Proposed Method

  • DLEPEM addresses within-task prediction and module-identity inference through dedicated task experts and prototype-ensemble matching.The two components target the two prediction stages identified by the paper.Source: §4 · pp. 5–6

4.1. Dynamic LoRA-Experts

  • Each task trains a new LoRA-Expert while earlier experts stay frozen; experts can be inserted in an MLP branch or the attention projections Wq and Wv.The pre-trained model weights remain fixed, and new experts are initialized from the preceding expert after the first task.Source: §4.1 · pp. 6–7

4.2. Prototype-Ensemble Matching Mechanism

  • The router uses plasticity distillation toward current-task class prototypes and stability distillation toward the previous router's feature distribution.The router is updated across tasks, and the combined loss weights its plasticity and stability terms.Source: §4.2 · p. 8
  • Each class key concatenates frozen-PTM and router-domain prototypes; nearest-key retrieval selects an expert, then classification is restricted to that expert's classes.The dictionary retains old keys without recomputing them and stores no raw samples or exemplar sets.Source: §4 · p. 6, §4.2 · p. 9

4.3. Optimization Objective and Training Procedure

  • Training first learns task experts, then optimizes the router and builds the prototype ensemble; a separate procedure describes task-agnostic inference.The training algorithm retains old dictionary entries and appends keys for newly introduced classes.Source: §4.3 · pp. 9–10

5. Experiments

Across five CIL and three FSCIL benchmarks, DLEPEM is evaluated against representative methods using shared pre-trained backbones and data splits. The experiments report leading benchmark performance, ablation and resource analyses, parameter sensitivity, and behavior across sequential tasks.

5.1. Experimental Settings

  • Experiments use a shared ViT-B/16-IN21K backbone and matched data splits across five CIL and three FSCIL benchmarks, with established protocols and three-run reporting.CIL covers VTAB, CIFAR100, CUB200, ImageNet-R, and OmniBenchmark; FSCIL covers CUB200, CIFAR100, and miniImageNet.Source: §5.1 · pp. 11–12, Table 3 · p. 12

5.2. Benchmark Comparison

  • DLEPEM achieves the best average accuracy on all five CIL benchmarks and leads on CUB200 and CIFAR100 FSCIL, while remaining competitive on miniImageNet.Against each dataset’s strongest CIL baseline, average accuracy improves by 0.80% on CIFAR100, 2.11% on CUB200, 0.53% on ImageNet-R, 1.54% on OmniBenchmark, and 1.39% on VTAB; on FSCIL, CUB200 average accuracy reaches 88.77%, 5.31% above ASP, and CIFAR100 reaches 90.50%, 1.96% above ASP.Source: Table 3 · p. 12, §5.2 · pp. 12–13, Table 4 · p. 14

5.3. Ablation Study

  • Task-specific experts outperform a shared expert under domain shift, while ensemble keys outperform single-source matching; removing the prototype-ensemble router also degrades performance.On ImageNet-R, replacing the shared expert with task-specific experts raises AL/average accuracy for both MLP-Expert and QV-Expert variants; frozen-prototype keys also yield lower AL/average accuracy than prototype-ensemble retrieval.Source: §5.3 · pp. 14–16
  • DLEPEM improves more with larger backbones and beats SimpleCIL among similarly sized ViT-B/16 variants despite architectural and pretraining differences.The evaluated alternatives include ViT-B/16-IN1K, ViT-L/16, ViT-B/16-DINO, and ViT-B/16-SAM.Source: §5.3 · p. 16

5.4. Analysis of Trainable Parameters and Accuracy

  • DLEPEM trades additional parameters for accuracy, with expert-bank storage scaling by task count and prototype-dictionary storage by seen classes.In the default 10-task setting, the bank stores 1.84M LoRA parameters for MLP-Experts and 3.69M for QV-Experts; the dictionary requires about 0.59–1.76 MiB for 100–300 classes.Source: §5.4 · pp. 16–17, Figure 6 · p. 17

5.5. Training and Inference Time Comparison

  • 8.39 ms/image on CIFAR100 and 8.47 ms/image on ImageNet-R exceed SD-LoRA and InfLoRA latency, despite competitive training time.The authors characterize the added inference overhead as non-negligible but practical.Source: §5.5 · p. 18, Table 7 · p. 18

5.6. Parameter Sensitivity Analysis

  • Performance is stable across tested LoRA ranks and insertion ranges on CUB200; α values in (0, 0.1] perform consistently well, with 0.04 as default.The rank tests span 1, 2, 4, 8, 10, and 16; insertion ranges span the first 2, 4, 8, or 12 Transformer layers.Source: §5.6 · p. 18, Figure 7 · p. 19

5.7. Performance of LoRA-based methods across sequential tasks

  • DLEPEM outperforms LAE on old tasks while freezing prior experts, and its unconstrained current experts show stronger new-task adaptation than InfLoRA and SD-LoRA.The paper describes the comparison as shifting stability and plasticity into separate factors: stationary old experts and a newly trainable expert.Source: §5.7 · p. 19

6. Conclusion

DLEPEM combines task-specific Dynamic LoRA-Experts with Prototype-Ensemble Matching, reporting a favorable stability-plasticity trade-off and strong performance across six evaluated benchmarks.

6. Conclusion

  • DLEPEM achieves strong, competitive performance on six benchmarks under the evaluated protocols.The authors describe the combined design as supporting a more favorable empirical stability-plasticity trade-off.Source: §6 · p. 19
  • DLEPEM combines task-specific Dynamic LoRA-Experts for adaptation with Prototype-Ensemble Matching for module selection.Source: §6 · p. 19
  • The expert bank grows linearly with task count, while the prototype dictionary grows with the number of seen classes.The authors identify expert compression, rank sharing, pruning, and merging or distillation as future-work directions.Source: §6 · pp. 19–20

Abbreviations

The manuscript lists abbreviations and their expansions for CIL/FSCIL, PEFT/PTM/LoRA/MoE, ViT/MHA/FFN, and WTP/MII.

Abbreviations

  • CIL means Class-Incremental Learning, and FSCIL means Few-Shot Class-Incremental Learning.Source: Abbreviations · p. 21
  • PEFT means Parameter-Efficient Fine-Tuning, PTM means Pre-Trained Model, LoRA means Low-Rank Adaptation, and MoE means Mixture-of-Experts.Source: Abbreviations · p. 21
  • ViT means Vision Transformer, MHA means Multi-Head Self-Attention, FFN means Feed-Forward Network, WTP means Within-Task Prediction, and MII means Module Identity Inference.Source: Abbreviations · p. 21
Loading 2609.39839v1…