Source-linked AI summary
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
Minsoo Kang, Jaeyoo Park, Bohyung Han
TL;DR
Class-incremental learning must acquire new classes while retaining prior knowledge despite limited access to old-task data and catastrophic forgetting. The paper uses knowledge distillation with theoretically derived adaptive feature-map weighting, and reports large-margin improvements over existing methods.
Problem
Online learning across task sequences is difficult because previous-task data are limited, and fine-tuning causes catastrophic forgetting.
Method
The method estimates how feature-map representation changes affect loss increases, then adaptively weights knowledge distillation to preserve important features while allowing less critical features to change.
Results
The proposed approach outperforms existing methods by large margins in various scenarios.
Takeaways & Limitations
Adaptive feature consolidation balances robustness to old tasks with adaptivity to new data in class-incremental learning.
Takeaways & Limitations
Architectural alternatives require additional network components and multiple forward passes, while directly optimizing the loss-bound objective adds substantial computation or memory overhead.
Abstract
from arXiv · showhide
We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks. Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively. The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates. It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model. Based on the importance, the model restricts updates of important features for robustness while allowing changes in less critical features for flexibility. This optimization strategy effectively alleviates the notorious catastrophic forgetting problem despite the limited accessibility of data in the previous tasks. The experimental results show significant accuracy improvement of the proposed algorithm over the existing methods on the standard datasets. Code is available.
ECE†, ASRI†, & IPAI†,§
The passage contains contact information for the paper’s authors.
- The listed contact email is kminsoo,bellos1203,bhhan}@snu.ac.kr.
- The passage is an author-contact entry.
- No methodological or empirical content is provided in this passage.
1. Introduction
The paper addresses catastrophic forgetting when learning task sequences with limited access to earlier data, proposing adaptive feature-weighted knowledge distillation. It theoretically links feature-map distribution shifts to loss increases and reports large-margin gains over existing methods.
- Limited previous-task examples make online learning across task sequences challenging for deep neural networks.Memory constraints or privacy issues restrict access to earlier data.
- Fine-tuning can perform well on the current task while failing to generalize to previous tasks because of catastrophic forgetting.
- The proposed approach uses knowledge distillation with feature-map weighting for class-incremental learning under limited access to old-task data.
- The method theoretically minimizes an upper bound on loss increases by relating feature-map distribution shifts to changes in loss.
- Experimental results show that the technique outperforms existing methods by large margins across various scenarios.
2. Related Work
Prior class-incremental learning methods use architectural expansion, rehearsal, parameter regularization, or knowledge distillation. The paper distinguishes its approach by theoretically grounded adaptive weighting of feature maps.
- Class-incremental learning methods are grouped into architectural, rehearsal, parameter-regularization, and knowledge-distillation approaches.
- Architectural Methods: Architectural methods expand network capacity for incoming tasks but require extra components and multiple inference passes, increasing computational costs.
- Rehearsal Methods: Rehearsal methods store representative examples or estimate past data distributions to support learning new tasks.
- Parameter Regularization Methods: Parameter-regularization methods adjust individual parameter updates according to estimated importance, including Fisher information or optimization-path measures.
- Knowledge Distillation Methods: Knowledge-distillation methods transfer teacher knowledge through logits, output distributions, intermediate activations, attention maps, or feature representations.
- Proposed Approach: Unlike prior methods, the proposed approach estimates feature-map importance theoretically to adaptively weight distillation and minimize loss increases from model updates.
3. Proposed Algorithm
The method uses knowledge distillation to preserve representations from previous tasks while learning new classes. It estimates feature-map importance from representation-induced loss changes, then penalizes changes to important features using objectives that can incorporate current-task data.
- Model and objective: The learner initializes each incremental model from the previous model and uses knowledge distillation to mitigate catastrophic forgetting while learning new classes.The model is decomposed into feature-extracting and prediction subnetworks, with the classifier expanded for newly observed classes.
- Importance estimation: Feature-map distribution shifts are linked to loss increases through a first-order Taylor approximation and channel-wise loss-increase terms.The analysis uses gradients from the previous model and measures changes between corresponding feature maps.
- Importance estimation: The practical objective minimizes an upper bound on expected loss increases by weighting squared feature-map changes with estimated channel importance.The bound avoids storing feature-map gradients or performing additional backpropagation during training.
- Importance estimation: Importance is estimated by Monte Carlo integration at the previous stage, stored as a scalar per channel, and interpreted as a weight factor for feature preservation.Larger importance values correspond to feature maps whose perturbations have greater effects on the loss.
- Empirical behavior: On CIFAR100 with 50 incremental stages, larger samples for importance estimation produce more reliable importance estimates and better generalization performance.Figure 2 reports results across three class orders using ResNet-32, with mean and standard deviation shown in the legend.
- Robust objective: Using current-task data in the expectation yields an objective that minimizes the upper bound of previous-task loss increases under the stated unchanged class-conditional data assumption.The proposition states that the current-and-previous-data optimization bounds the previous-task objective when the data-generating process for each class remains unchanged.
4. Experiments
Experiments evaluate AFC across standard class-incremental benchmarks, memory budgets, feature-distillation variants, class orders, and task configurations. AFC achieves the strongest reported performance across these settings and remains effective as incremental learning becomes more challenging.
- Results on CIFAR100: AFC achieves the highest accuracy across the tested numbers of incremental stages on CIFAR100, with larger gaps as the number of stages increases.Both NME and CNN inference variants outperform the reported PODNet results and reach state-of-the-art performance.
- Results on ImageNet: AFC outperforms compared methods on ImageNet100 and ImageNet1000, with especially large margins for 50-stage ImageNet100.The authors identify 50-stage ImageNet100 as more prone to catastrophic forgetting.
- Effect of memory budget: AFC outperforms state-of-the-art methods across several memory budgets, while the preferred discrepancy-loss formulation depends on the budget per class.Using exemplar sets together with current-task data performs better at small budgets; with 50 examples per class, AFC with Eq. (10) is slightly better.
- Analysis and Ablation Studies: AFC outperforms seven feature-distillation alternatives, including fine-tuning and uniform importance, on CIFAR100 with 50 stages.Average accuracies across three class orders are used for the CIFAR100 comparison, and AFC is reported as more robust to catastrophic forgetting than PODNet throughout the stages.
- Analysis and Ablation Studies: AFC consistently outperforms previous algorithms when the initial task is reduced and each remaining task contains one class.This configuration increases the total number of incremental stages and makes the problem more challenging.
- Analysis and Ablation Studies: AFC is not sensitive to λdisc, which controls the balance between previous knowledge and the current task.The sensitivity analysis is conducted on CIFAR100 with 50 stages.
5. Conclusion
The paper presents adaptive feature consolidation for class-incremental learning, balancing adaptation to new data with robustness to old knowledge. It derives feature importance from distribution discrepancies and loss changes, and reports large-margin improvements over existing methods.
- Adaptive feature consolidation weights feature maps to balance adaptivity to new data and robustness to previously learned knowledge.The approach is designed for class-incremental learning scenarios.
- Feature importance is formulated by relating discrepancies in feature distributions to changes in the loss.
- Knowledge distillation is incorporated into a relaxed objective to reduce computational cost and memory overhead.
- Experimental results show that the proposed approach outperforms existing methods by large margins.
6. Appendix
The appendix derives an upper bound for expected loss changes, evaluates exemplar-memory strategies and alternative metrics, and discusses practical limitations. AFC performs strongly across exemplar policies and generally surpasses PODNet, especially in settings with more catastrophic forgetting.
- Derivation: The appendix derives the expected loss change from feature-map updates and obtains an upper bound using gradient and representation-change norms.The derivation applies an inequality relating the expectation of a product to the product of expected squared norms.
- Derivation: The appendix concludes the upper-bound derivation by applying the expectation inequality to feature-map gradients and representation changes.
- Other metrics: AFC outperforms PODNet on backward transfer and average accuracy in most evaluated cases.AFC is marginally worse on PODNet’s NME backward-transfer comparison at 10 and 5 stages, but clearly better at 50 and 25 stages.
- Exemplar selection: Nearest-mean exemplar selection is more effective than random selection for both AFC and PODNet, while AFC also beats PODNet with random exemplars.
- Limitation: Existing methods require old-task exemplars, while functional regularization methods add forward-pass costs in FLOPs, memory, and power.The appendix identifies resource-hungry systems as an important boundary for future algorithms.