Source-linked AI summary
Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning
Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, Zheng-Jun Zha
TL;DR
Non-exemplar class-incremental learning must recognize old and new classes when old samples cannot be stored, while representation optimization and retention rely on new-class supervision. The paper proposes self-sustaining representation expansion with dynamic structure reorganization, main-branch distillation, and prototype selection. Experiments show superiority and adaptability over state-of-the-art methods, especially across multiple phases.
Problem
Non-exemplar class-incremental learning lacks stored old samples, making it difficult to preserve old representations and discriminate old from new classes during new-class optimization.
Method
The method combines dynamic structure reorganization, main-branch distillation, and prototype selection to preserve old features and separate old and new classes.
Results
The method is superior in performance and adaptability to state-of-the-art methods, especially in multi-phase processes.
Takeaways & Limitations
Self-sustaining representation expansion supports NECIL by integrating newly optimized features with preserved old structure and invariant knowledge.
Takeaways & Limitations
Existing NECIL transfers can suffer performance degradation and parameter explosion, while similar old and new classes remain difficult to distinguish.
Abstract
from arXiv · showhide
Non-exemplar class-incremental learning is to recognize both the old and new classes when old class samples cannot be saved. It is a challenging task since representation optimization and feature retention can only be achieved under supervision from new classes. To address this problem, we propose a novel self-sustaining representation expansion scheme. Our scheme consists of a structure reorganization strategy that fuses main-branch expansion and side-branch updating to maintain the old features, and a main-branch distillation scheme to transfer the invariant knowledge. Furthermore, a prototype selection mechanism is proposed to enhance the discrimination between the old and new classes by selectively incorporating new samples into the distillation process. Extensive experiments on three benchmarks demonstrate significant incremental performance, outperforming the state-of-the-art methods by a margin of 3%, 3% and 6%, respectively.
1. Introduction
Non-exemplar class-incremental learning must learn new classes without retaining old samples, making feature retention and class discrimination difficult. The paper addresses these problems with self-sustaining representation expansion, distillation, and prototype selection.
- Class-incremental learning aims to recognize new classes without forgetting previously learned classes.
- Without exemplars, fine-tuning on new classes biases representations and classifiers toward new classes, causing catastrophic forgetting of old classes.
- Existing rehearsal-based methods suffer unreliable distillation without old samples, while structure-based methods incur parameter explosion and confusion between similar old and new classes.
- The proposed self-sustaining representation expansion uses dynamic structure reorganization to optimize expanded directions while preserving old feature structure, then integrates the overall structure after each phase.
- Prototype selection uses samples similar to old classes for distillation and dissimilar samples for new-class training, reducing confusion between similar classes.
- Experiments on CIFAR-100, TinyImageNet, and ImageNet-Subset report superiority over state-of-the-art methods.
2. Related Work
Related work covers regularization-, rehearsal-, and structure-based continual-learning methods, along with recent non-exemplar approaches. The paper distinguishes its focus on joint representation learning and distillation when past data cannot be stored.
- Continual-learning methods include regularization-based parameter constraints, rehearsal-based exemplar storage with distillation, and structure-based subnetwork expansion.
- Recent NECIL methods address settings where past data cannot be stored because of equipment limitations or privacy concerns.
- Prior NECIL work estimates semantic drift, augments prototypes, or learns transferable features for future tasks.
- This paper instead emphasizes joint representation learning and the distillation process in the absence of exemplars, while residual adapters provide related architectural context.
3. Problem Description
NECIL trains on sequential, disjoint class sets while retaining no old samples, yet evaluates each phase on all classes seen so far.
- Each incremental phase provides sufficient current samples, but no old-class samples remain available in memory.
- Incremental classes are disjoint, with Y_i ∩ Y_j = ∅ for i ≠ j.
- At phase i, performance is measured by classification accuracy on Z_i containing classes from all label sets seen through phase i.
4. Methodology
The methodology combines standard NECIL training analysis with dynamic structure reorganization, distillation, prototype-based classifier calibration, and prototype selection to preserve old representations while learning new classes.
- Standard NECIL Paradigm: The baseline analyzes representation learning and classifier learning separately, using cross-entropy for new classes and distillation to retain information from the previous model.The classifier maps representations to the label space, while Euclidean-distance distillation measures similarity to the previous representation.
- Standard NECIL Paradigm: Prototype rehearsal oversamples one deep-feature prototype per class to the batch size for classifier calibration without exemplars.This follows a long-tail recognition strategy rather than augmenting prototypes with Gaussian noise.
- Standard NECIL Paradigm: Without exemplars, joint optimization collapses toward features relevant only to incremental classes, undermining old-class representation retention.
- Self-Sustaining Representation Expansion: Dynamic structure reorganization adds residual adapters to fixed convolution blocks, allowing optimization through expanding side branches while maintaining old features.After training, structural reparameterization fuses side-branch parameters into the main branch through zero-padding and linear transformation, removing the adapters and avoiding parameter growth at phase end.
- Self-Sustaining Representation Expansion: Prototype selection routes dissimilar new samples to residual-adapter updates and similar samples to distillation, based on similarity with old prototypes.Normalized cosine scores are computed in the learned embedding space; threshold masks select cross-entropy or distillation contributions, which are combined with prototype-balance loss.
5. Experiments
Experiments on CIFAR-100, TinyImageNet, and ImageNet-Subset evaluate ablations, expanding structures, representation effects, and comparisons with non-exemplar and exemplar-based methods. The proposed scheme improves incremental accuracy, reduces forgetting, balances old and new class performance, and remains robust across datasets and phase counts.
- Ablation Study: 4.3% and 4.8% overall-performance improvements result from dynamic representation and main-branch distillation, respectively.The former contributes more with 5 and 10 phases, while the latter contributes more with 20 phases.
- Ablation Study: A 1×1 convolution structure performs similarly to the 1×1 convolution plus BatchNorm combination, while 3×3 convolution is one point lower.The results suggest 1×1 convolution is sufficient for learning new-class representation in the residual adapter.
- Ablation Study: At a threshold of 0.8, all prototype-selection curves peak before gradually declining as the threshold increases.The analysis links this behavior to similarity differences and large intra-class fluctuations among incremental samples.
- Comparison with SOTA: The method performs similarly on old and new classes without favoring either through overfitting, and its advantage expands as incremental phases increase.Accuracy curves show superiority at almost all phases and robustness on both smaller CIFAR-100 and larger ImageNet-Subset.
- Analysis: DSR largely preserves old-class discrimination and relationships, while MBD improves separation between novel clusters and similar old classes.These effects are illustrated using t-SNE visualizations of representations after multi-phase optimization.
- Comparison with SOTA: The method achieves average improvements of 3, 3, and 6 points over non-exemplar state-of-the-art methods on CIFAR-100, TinyImageNet, and ImageNet-Subset, respectively.Its performance is comparable to classical exemplar-based methods with E=20, and it achieves much lower average forgetting.
6. Conclusion and Discussion
The paper presents a self-sustaining representation expansion scheme for NECIL that combines dynamic structure reorganization, main-branch distillation, and prototype selection. Experiments show superior performance and adaptability to state-of-the-art methods, especially across multiple phases.
- The proposed NECIL scheme optimizes new features in a side branch while preserving old feature distributions through structural expansion and main-branch distillation.Prototype selection is integrated into joint training to enhance distinction between old and new classes.