Source-linked AI summary
DER: Dynamically Expandable Representation for Class Incremental Learning
Shipeng Yan, Jiangwei Xie, Xuming He
TL;DR
Class incremental learning under limited memory requires balancing retention of old knowledge with adaptation to novel classes. DER freezes previously learned representations, adds new feature extractors, and dynamically prunes channels with auxiliary supervision. Across three benchmarks, it consistently outperforms prior methods and reports positive backward and forward transfer.
Problem
Class incremental learning under limited memory must incorporate novel concepts without forgetting existing knowledge, balancing stability against plasticity.
Method
DER uses two-stage learning with frozen prior representations, newly added feature dimensions, channel-level mask-based pruning, and an auxiliary loss for novel discriminative features.
Results
DER consistently outperforms prior state-of-the-art approaches across CIFAR-100, ImageNet-100, and ImageNet-1000, with positive backward and forward transfer.
Takeaways & Limitations
The dynamically expandable representation achieves a better stability-plasticity trade-off while maintaining compact features for novel concepts.
Takeaways & Limitations
The method models new parameters with a uniform prior while approximating old parameters with a Dirac prior to preserve learned information.
Abstract
from arXiv · showhide
We address the problem of class incremental learning, which is a core step towards achieving adaptive vision intelligence. In particular, we consider the task setting of incremental learning with limited memory and aim to achieve better stability-plasticity trade-off. To this end, we propose a novel two-stage learning approach that utilizes a dynamically expandable representation for more effective incremental concept modeling. Specifically, at each incremental step, we freeze the previously learned representation and augment it with additional feature dimensions from a new learnable feature extractor. This enables us to integrate new visual concepts with retaining learned knowledge. We dynamically expand the representation according to the complexity of novel concepts by introducing a channel-level mask-based pruning strategy. Moreover, we introduce an auxiliary loss to encourage the model to learn diverse and discriminate features for novel concepts. We conduct extensive experiments on the three class incremental learning benchmarks and our method consistently outperforms other methods with a large margin.
1. Introduction
Class incremental learning must incorporate novel concepts while retaining old knowledge under limited memory. DER addresses this stability-plasticity trade-off with a dynamically expandable representation, two-stage learning, pruning, and auxiliary supervision.
- Problem: Class incremental learning seeks sequential learning of novel concepts while performing well on all observed classes.Limited memory and catastrophic forgetting make this challenging for modern visual recognition systems.
- Problem: Limited rehearsal memory creates a stability-plasticity dilemma: excessive plasticity forgets old categories, while excessive stability impedes adaptation to novel concepts.Existing approaches may sacrifice plasticity or remain susceptible to forgetting through feature degradation.
- Method: DER freezes prior representations and augments them with feature dimensions from a new extractor at each incremental step.The resulting super-feature increases dimensionality for new classes while retaining existing knowledge.
- Method: The method uses a two-stage strategy, auxiliary loss, and channel-level mask-based pruning to learn discriminative novel features and compact representations.The representation expands according to novel-concept complexity, while the auxiliary loss encourages diverse and discriminative features.
2. Related Work
Prior class-incremental methods address representation and classifier learning through regularization, distillation, structure allocation, and classifier correction. DER differs from related structure-based methods by freezing complete prior representations and adding new feature extractors.
- Overview: Rehearsal-based class incremental learning is commonly analyzed through representation learning and classifier learning under limited data memory.Classifier learning is especially challenged by class imbalance.
- Representation Learning: Regularization methods constrain changes to important old weights, while distillation methods preserve network outputs, features, spatial structure, or topology.Examples include EWC-style regularization, iCaRL, EE2L, UCIR, PODNet, and TPCIL.
- Representation Learning: Structure-based methods keep parameters for previous classes fixed while allocating unused parameters or additional networks for novel knowledge.RPSNet selects paths progressively, while CCGN uses task-specific gating modules and a task predictor.
- Classifier Learning: Classifier-learning methods address class imbalance through joint training, classifier finetuning, bias correction, or weight alignment.EEIL finetunes on balanced data, BiC adds a bias-correction layer, and WA aligns new and old classifier-weight norms.
- Discussion: DER preserves old-concept structure by freezing the previous representation and augmenting it with a new feature extractor, unlike RPSNet and CCGN.This lets the final classifier reuse old structure while mitigating forgetting.
3. Methods
DER uses a two-stage class-incremental procedure that freezes prior representations, adds a new feature extractor, and retrains the classifier. Channel masks, auxiliary supervision, and sparsity control regulate the added representation and its size.
- Method overview: DER decouples representation learning from classifier learning while using rehearsal data from the memory and incoming classes.The representation stage expands features, followed by classifier retraining with currently available data and balanced finetuning.
- Expandable representation: At step t, the super-feature extractor concatenates the reused previous extractors with a newly created extractor F_t.Previous extractor parameters remain frozen, while the new extractor is trained to capture novel aspects of incoming classes.
- Auxiliary supervision: The auxiliary classifier distinguishes each new class from a single category containing all old concepts.Its label space has |Y_t|+1 categories, and λ_a controls the auxiliary loss contribution; λ_a=0 when t=1.
- Dynamic expansion: A differentiable channel-level mask prunes the new extractor according to novel-concept complexity to reduce redundancy and maintain a compact representation.Masks modulate layer feature maps, are learned jointly with the representation, and are binarized for inference to obtain the pruned network.
- Dynamic expansion: The mask sharpness is increased through an annealing schedule, progressively changing training from uniformly activated channels toward binarized masks.The schedule uses batch index b, epoch batch count B, and s_max to control the progression; a sparsity loss penalizes the ratio of used weights.
4. Experiments
Experiments on CIFAR-100, ImageNet-100, and ImageNet-1000 evaluate DER across incremental protocols, model sizes, and ablations. DER consistently outperforms prior methods, including difficult long-step settings, while exhibiting positive backward and forward transfer.
- Experiment setup: DER is evaluated on CIFAR-100, ImageNet-100, and ImageNet-1000 under multiple class-incremental benchmark protocols.CIFAR-100 uses B0 and B50 protocols; ImageNet-100 uses B0 and B50, with fixed memory settings described for the protocols.
- CIFAR-100 evaluation: 72.05% average incremental accuracy is achieved versus 64.32% under CIFAR100-B0 with 50 incremental steps, a +7.73% improvement using fewer parameters.The reported pruning-induced degradation is negligible in this setting.
- CIFAR-100 evaluation: 58.66% last-step accuracy is achieved versus 42.75% under CIFAR100-B0 with 50 steps, a +15.91% improvement.The method surpasses competing methods at every step, and the gap increases as novel classes are added.
- CIFAR-100 evaluation: 65.58% last-step accuracy is achieved versus 52.56% under CIFAR100-B50 with 10 steps, a +13.02% improvement, while average incremental accuracy exceeds PODNet by 8.41%.DER performs better than other methods at each step across the reported CIFAR100-B50 splits.
- ImageNet evaluation: DER consistently surpasses competing methods across ImageNet-100 and ImageNet-1000 splits, including +5.15% on ImageNet100-B50 last-step top-1 accuracy and +3.02% on ImageNet1000-B0 final-step top-1 accuracy.The reported ImageNet100-B0 average top-5 accuracy improvement is about 1.79%.
- Ablation and transfer analysis: Representation expansion raises CIFAR100-B0 average accuracy from 61.84% to 73.26%, with auxiliary loss providing a further 2.10% gain.The ablation uses the 10-step CIFAR100-B0 setting.
- Ablation and transfer analysis: DER achieves positive backward transfer of +1.36% and positive forward transfer of +1.49%, whereas other methods generally show forgetting or negative forward transfer.The backward-transfer analysis tracks representation quality across increments, while forward transfer measures influence on subsequent concepts.
5. Conclusion
The paper proposes dynamically expandable representation for class incremental learning, combining frozen prior features with novel parameterized features. Experiments on three major benchmarks show consistent improvement and positive backward and forward transfer.
- The method freezes previously learned representations and augments them with novel parameterized features at each incremental step.
- Channel-level mask-based pruning expands the representation according to the difficulty of novel concepts.
- An auxiliary loss encourages learning better novel discriminative features.
- Experiments on three major incremental classification benchmarks show consistent performance gains over other methods with a sizable margin.
- The method achieves positive backward and forward transfer.
Appendices
The appendices specify separate optimization settings for representation learning and classifier learning, including dataset-specific schedules and loss temperatures.
- Representation learning uses SGD with weight decay 0.0005, dataset-dependent batch sizes, warmup, and learning-rate decay schedules.CIFAR-100 uses batch size 128 and 160 post-warmup epochs; ImageNet100 and ImageNet1000 use batch size 256 and 120 epochs.
- The auxiliary-loss coefficient λa is fixed at 1, while λs is tuned for comparable parameter counts and held constant across steps within each experiment.
- Classifier learning updates only the classifier for 30 epochs with SGD, using weight decay 0.0005 and a learning rate of 0.1.The learning rate decays at 15 epochs, and the cross-entropy temperature is δ = 5 for CIFAR-100 and δ = 1 for ImageNet-100 and ImageNet-1000.
B. Sensitive Study of Hyper-parameters
The sensitivity study evaluates the auxiliary-loss coefficient λa on CIFAR100-B0 with 10 steps and reports robustness, while also examining λs.
- The method is robust to different values of λa in the CIFAR100-B0 10-step sensitivity study.
- Experiments also evaluate different values of λs, with results shown in Figure 1 in the main body.
- Table 6 presents the sensitive study on the effects of λa.
C. The Quality of Decision Boundary
The appendix evaluates whether the second-stage classifier can learn a high-quality decision boundary using memory, comparing the method with an all-data upper-bound strategy. It also reports comparable inference latency against baselines.
- The second-stage classifier is evaluated against an ideal strategy that uses all previous data to train the classifier.
- 76.14 ± 0.80% is achieved by the all-data upper bound on CIFAR100-B0 with 10 steps, versus 74.64 ± 0.28% for the proposed method.
- Similar results on other benchmarks support the efficacy of the second-stage learning.
- 1.07ms/image is the proposed method’s ImageNet inference latency on GTX 1080Ti, compared with 0.99ms/image for BiC and WA.
E. Results for modified 32-layer ResNet
On CIFAR100-B0 and CIFAR100-B50, the method outperforms competing approaches even with a modified 32-layer ResNet.
- Our method outperforms other methods on both CIFAR100-B0 and CIFAR100-B50 using a modified 32-layer ResNet.The comparison uses the same small-network setting as other methods.
F. More detailed results on CIFAR100
Figure 5 reports performance across incremental steps on CIFAR100-B0 under three step settings and illustrates the superiority of the proposed method.
- Figure 5 evaluates performance across incremental steps on CIFAR100-B0 with 5, 10, and 2 incremental steps.
- The figure includes CIFAR100-B0 results for the 5-step and 10-step settings.
- The figure also includes CIFAR100-B0 results with 2 incremental steps.
G. Detailed results on ImageNet
Performance curves demonstrate the method’s effectiveness across ImageNet datasets with different class-incremental configurations.
- The method’s performance is evaluated across ImageNet100-B0, ImageNet100-B50, and ImageNet1000-B0, each with 10 incremental steps.Figure 6 reports performance with respect to incremental steps on these three datasets.
- The ImageNet curves are presented as evidence of the method’s effectiveness on complex datasets.
- Tables 7 and 8 report average inference parameters and average accuracy for CIFAR100-B0 and CIFAR100-B50, including an unpruned variant.# Paras is measured in millions, and Avg is average accuracy over steps.
- Figure 5 reports CIFAR100 performance for 5-step and 10-step CIFAR100-B0 settings and a 2-step CIFAR100-B50 setting.