Source-linked AI summary
FOSTER: Feature Boosting and Compression for Class-Incremental Learning
Fu-Yun Wang, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan
TL;DR
Class-incremental learning must learn emerging categories without catastrophically forgetting earlier ones, while existing solutions can face stability-plasticity or overhead challenges. FOSTER combines residual-fitting feature boosting with distillation-based compression, and reports state-of-the-art performance across three incremental-learning benchmarks.
Problem
Class-incremental learning must continually acquire new categories while preserving discrimination for old classes, despite catastrophic forgetting and limitations involving plasticity, computation, or storage.
Method
FOSTER freezes the old model, expands modules to fit residuals between targets and prior outputs, then distills the expanded model into a single backbone by removing redundant parameters and feature dimensions.
Results
FOSTER achieves state-of-the-art performance across CIFAR-100, ImageNet-100, and ImageNet-1000 under the evaluated incremental-learning settings.
Takeaways & Limitations
The reported paradigm adaptively learns new categories while compressing the expanded model into a single backbone with negligible performance loss after distillation.
Takeaways & Limitations
The method is evaluated with the imbalanced current-stage dataset and is designed to achieve performance without requiring auxiliary data.
Abstract
from arXiv · showhide
The ability to learn new concepts continually is necessary in this ever-changing world. However, deep neural networks suffer from catastrophic forgetting when learning new categories. Many works have been proposed to alleviate this phenomenon, whereas most of them either fall into the stability-plasticity dilemma or take too much computation or storage overhead. Inspired by the gradient boosting algorithm to gradually fit the residuals between the target model and the previous ensemble model, we propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively. Specifically, we first dynamically expand new modules to fit the residuals between the target and the output of the original model. Next, we remove redundant parameters and feature dimensions through an effective distillation strategy to maintain the single backbone model. We validate our method FOSTER on CIFAR-100 and ImageNet-100/1000 under different settings. Experimental results show that our method achieves state-of-the-art performance. Code is available at: https://github.com/G-U-N/ECCV22-FOSTER.
1 Introduction
Class-incremental learning must accommodate new categories while preserving old-class discrimination, but existing approaches face plasticity, feature degradation, and growing overhead. FOSTER addresses these issues through gradient-inspired feature boosting followed by compression.
- Class-incremental learning seeks continual learning of novel categories while maintaining discrimination for previously learned classes.
- Knowledge-distillation methods preserve old outputs but may lack plasticity, while dynamic architectures increase plasticity at the cost of expanding modules and overhead.
- FOSTER applies an additive gradient-boosting framework in which new modules fit residuals between targets and prior model outputs.
- Feature compression uses distillation to remove redundant parameters and inconsistent feature dimensions while retaining a single backbone with negligible performance loss.
- Feature boosting retains the frozen old model, expands a trainable extractor, concatenates features, and maps the resulting super feature to logits.
- FOSTER achieves state-of-the-art performance on CIFAR-100 and ImageNet-100/1000 across the evaluated settings.
2 Related Work
Related work addresses catastrophic forgetting through knowledge distillation, rehearsal, and dynamic architectures, each using different mechanisms to retain or recover prior knowledge.
- Knowledge distillation transfers teacher information by encouraging a student model to approximate the teacher’s outputs for old categories.
- Rehearsal methods preserve access to prior information through exemplars, low-dimensional features, or generated instances replayed during later tasks.
- Dynamic architectures preserve selected frozen modules and expand trainable modules to maintain old knowledge while increasing plasticity for new categories.
3 Preliminary
Gradient boosting builds an additive predictor by repeatedly adding functions that fit residual-related objectives. Class-incremental learning instead receives new-category data stage by stage, with optional limited rehearsal data and evaluation over all seen categories.
- 3.1 Gradient Boosting: Gradient boosting minimizes empirical risk by incrementally adding weighted weak functions from specified function spaces.
- 3.1 Gradient Boosting: After m iterations, the next function is selected to optimize the loss of the additive model formed by the prior predictor and the new function.
- 3.1 Gradient Boosting: Because direct optimization is typically infeasible, gradient boosting uses a steepest-descent update whose new function approximates the negative loss gradient.
- 3.1 Gradient Boosting: Under mean-squared error, the residual objective is expressed through the difference between the target y and the current prediction F_m(x).
- 3.2 Class-Incremental Learning Setup: In class-incremental learning, each stage supplies a batch of new training data and expands the label space with disjoint categories.
- 3.2 Class-Incremental Learning Setup: The training set may combine current data with a limited subset of saved old data, while the model must perform well on all seen categories.
4 Method
FOSTER applies gradient boosting to class-incremental learning by expanding a module that fits residuals, then compresses the expanded representation into a single backbone. Calibration strategies address old–new class imbalance, while knowledge distillation removes redundant parameters and feature dimensions with limited data.
- Feature Boosting: The expanded model combines frozen old features with new features, producing logits for both old and new categories.The new branch supports new-class learning, while its old-class logits fit residuals and exploit additional discriminative patterns.
- Feature Boosting: FOSTER trains a new feature extractor and classifier to fit residuals between the target labels and the previous model's outputs.This gradient-boosting-inspired expansion preserves the previous model while adding plasticity for new classes.
- Calibration for Old and New: Logits Alignment rescales old and new class logits using factors derived from normalized effective class numbers to reduce classification bias.The method uses 0 < γ1 < 1 and γ2 > 1, with class weighting based on instance counts and an adjustable β.
- Calibration for Old and New: Feature Enhancement trains the new feature representation to classify all seen categories and uses distillation to match the previous model on old categories.This addresses cases where residuals provide insufficient supervision for old classes and reduces overfitting from imbalanced one-hot training.
- Feature Compression: Feature compression removes redundant parameters and meaningless dimensions from the expanded model through knowledge distillation into a smaller single model.The dual-branch model supplies soft targets, and similar-domain unlabeled data may be sampled temporarily without occupying additional memory.
5 Experiments
FOSTER is evaluated against state-of-the-art methods on CIFAR-100 and ImageNet benchmarks, including ablations and robustness tests. It achieves strong incremental-learning performance while compressing expanded models back to one backbone with negligible degradation.
- Benchmark Results: FOSTER outperforms other state-of-the-art strategies in all six CIFAR-100 settings.It improves performance across both long-term and large-step incremental-learning tasks.
- Benchmark Results: 3.11% and 2.67% improvements are achieved under CIFAR-100 base-50 25-step and base-0 20-step settings, respectively.
- Benchmark Results: 1.71% and 3.06% improvements are obtained with 20 and 10 new classes per step, respectively.
- Compression and Efficiency: Compression causes negligible performance degradation, keeping the single-backbone FOSTER model close to the dual-branch FOSTER B4 model.The model's parameter count and feature dimensions do not increase with the number of tasks.
- Benchmark Results: 68.34% average top-1 accuracy is achieved on ImageNet-1000 with 10 steps, improving over 66.73% by 1.61%.
- Ablation Study: Logits alignment surpasses weight alignment by about 4% in final accuracy, while removing feature enhancement causes more than a 3% final-stage decline.Feature enhancement particularly improves performance on old categories by more than 4%.
- Robustness Testing: Robustness is tested across different β values and exemplar counts on CIFAR-100 B50 with 5 steps.
6 Conclusions
FOSTER applies gradient boosting to class-incremental learning through residual-fitting modules, feature enhancement, logits alignment, and distillation-based compression. Experiments on three benchmarks report state-of-the-art performance while returning the expanded model to a single backbone.
- FOSTER creates a new module at each step to learn residuals between the target and the original model.
- Logits alignment and feature enhancement address classification bias and balance representation learning between old and new classes.
- A distillation strategy removes redundant parameters and dimensions, compressing the expanded model into a single backbone.
- Experiments on three widely used incremental-learning benchmarks show state-of-the-art performance.
I Rationality Analysis of the Substitution.
The substitution replaces summing softmax outputs with applying softmax to summed logits. This keeps the combined boosting output within valid probability constraints while integrating both models' judgments.
- The substitution uses softmax of summed logits instead of summing softmax outputs.
- Summing softmax outputs can violate the desired output range because the combined class probability may reach 2.
- Softmax applied after logit summation constrains the boosting output between 0 and 1 while integrating the original and new models.
II Influence of the Initialization of the Weight O
The study compares fine-tune, all-zero, and all-zero-with-bias initialization strategies for the super-classifier weight O. Fine-tuning is slightly better, while the all-zero variants perform similarly.
- The default strategy initializes O to all zeros, making the original model's outputs for new categories zero.
- Fine-tune, all-zero, and all-zero-with-bias strategies are compared for initializing O.
- Fine-tune performs slightly better than all-zero and all-zero-with-bias initialization, but the difference is not significant.
- The performance gap between all-zero and all-zero-with-bias initialization is negligible, indicating weak bias influence.
III Introduction to Compared Methods
The compared methods address catastrophic forgetting through fine-tuning, replay, bias correction, weight alignment, distillation, or dynamic feature extractors, with different trade-offs in retention and model size.
- Baselines: Fine-tune updates parameters for each new task and suffers from catastrophic forgetting, while replay uses rehearsal to alleviate it.
- Classifier correction: WA corrects biased final-classifier weights by aligning new-class weight norms with old-class weight norms.The correction uses a scaling factor based on the mean norms of old and new classes.
- Distillation: PODNet applies spatial-based distillation throughout the model and performs better on long runs of small incremental tasks.
- Dynamic architectures: DER preserves old feature extractors and concatenates new ones into a higher-dimensional space, but parameters still increase with task count.
IV Visualization of Detailed Performance
The visualizations compare feature organization and classification behavior across incremental-learning methods. FOSTER maintains clearer category clusters and boundaries, while fine-tuning shows severe bias toward new categories.
- Visualizing Feature Representation: FOSTER forms good clusters with clear classification boundaries at every incremental learning stage, unlike fine-tuning.Fine-tuning’s category clusters become poor during incremental stages, with scattered feature points in the last stage.
- Visualizing Confusion Matrix: Fine-tuning’s confusion matrix shows strong classification bias toward new categories, indicating severe catastrophic forgetting.The matrix has brightest colors on the right and colder colors elsewhere.
- Visualizing Feature Representation: The t-SNE visualization uses normalized features and randomly selects one category from each group of five categories.
- Visualizing Confusion Matrix: Confusion matrices encode real labels vertically and predicted labels horizontally, with warmer colors representing more samples.