Source-linked AI summary
Learning without Memorizing
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, Rama Chellappa
TL;DR
Incremental learning must expand a classifier to new classes while retaining base-class knowledge without storing old data. LwM adds Attention Distillation Loss to constrain teacher–student attention-map divergence alongside classification and prediction distillation losses. The approach consistently outperforms state-of-the-art performance on iILSVRC-small and iCIFAR-100, while the paper evaluates object classification and proposes segmentation as future work.
Problem
Incremental learning must add new classes while preserving base-class knowledge, but storing base-class data or models is memory-expensive and may be unavailable when training data is private.
Method
LwM combines classification and knowledge-distillation losses with Attention Distillation Loss, constraining the L1 distance between teacher and student attention maps without base-class data.
Results
LwM consistently outperforms state-of-the-art performance on the iILSVRC-small and iCIFAR-100 datasets.
Takeaways & Limitations
Constraining attention-map changes helps retain base-class information while the classifier incrementally learns new classes without storing base-class data.
Takeaways & Limitations
The study explores incremental learning for object classification; extending LwM to incremental segmentation is identified as future work.
Abstract
from arXiv · showhide
Incremental learning (IL) is an important task aimed at increasing the capability of a trained model, in terms of the number of classes recognizable by the model. The key problem in this task is the requirement of storing data (e.g. images) associated with existing classes, while teaching the classifier to learn new classes. However, this is impractical as it increases the memory requirement at every incremental step, which makes it impossible to implement IL algorithms on edge devices with limited memory. Hence, we propose a novel approach, called `Learning without Memorizing (LwM)', to preserve the information about existing (base) classes, without storing any of their data, while making the classifier progressively learn the new classes. In LwM, we present an information preserving penalty: Attention Distillation Loss ($L_{AD}$), and demonstrate that penalizing the changes in classifiers' attention maps helps to retain information of the base classes, as new classes are added. We show that adding $L_{AD}$ to the distillation loss which is an existing information preserving loss consistently outperforms the state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in terms of the overall accuracy of base and incrementally learned classes.
1. Introduction
Incremental learning seeks to add unseen classes while preserving recognition of base classes without storing their data. LwM addresses this memory-constrained setting by constraining attention-map divergence between teacher and student models, and reports improvements over established baselines.
- Incremental learning must add unseen classes while preserving performance on previously learned base classes.
- Storing base-class data or models is memory-expensive, may be impractical for lifelong learning and edge settings, and can be unavailable when training data remains private.
- Attention Distillation Loss complements prediction distillation by matching attention regions that encode which visual hints support base-class predictions.
- LwM preserves base-class information without storing base-class data by restricting divergence between teacher and student attention maps.
- Figure 2 contrasts degrading and stable attention regions, showing that prediction distillation is unaffected by degradation whereas Attention Distillation Loss is sensitive to it.
- LwM consistently outperforms state-of-the-art performance on the iILSVRC-small and iCIFAR-100 datasets.
2. Related work
Incremental learning expands a classifier to new classes while retaining knowledge of base classes, with class-incremental evaluation being harder because new and base classes are jointly classified. Related methods differ by task setting and whether they store base-class data; this work targets class-incremental learning without such data and extends distillation with attention maps.
- Incremental learning trains classifiers to recognize new classes while retaining knowledge of originally trained classes.
- Task-incremental methods evaluate on different datasets, whereas class-incremental methods learn new classes within the same dataset.
- Single-headed class-incremental evaluation is harder than task-incremental evaluation because new classes can be confused with base classes.
- Methods using base-class data reduce intransigence but increase memory requirements at each incremental step.
- This work focuses on class-incremental learning without base-class data and uses LwF-MC as its baseline.
- The proposed approach constrains teacher and student attention maps in addition to their prediction vectors, while previous-step teacher models cannot be accumulated under limited memory.
3. Background
The background defines distillation as an information-preserving penalty that aligns teacher and student predictions on base classes when new-class images are input. LwM adds attention-map preservation, using Grad-CAM-derived maps alongside classification and distillation losses while the previous model remains frozen.
- 3.1. Distillation loss (LD): Distillation loss LD aligns the student’s and teacher’s base-class prediction vectors when new-class images are provided.The vectors contain probability scores for N base classes.
- 3.2. Generating attention maps: Grad-CAM generates attention maps by weighting convolutional feature maps using gradients of a desired class score.
- 3.2. Generating attention maps: At incremental step t, the three losses train Mt on new-class images while Mt−1 remains frozen.
4. Proposed approach
LwM combines classification and distillation losses with an attention-based information-preserving penalty, LAD, to keep teacher and student attention representations similar while learning new classes.
- Proposed approach: LwM introduces LAD, an attention-map penalty combined with classification loss LC and distillation loss LD.The resulting objective is LLwM = LC + βLD + γLAD, where β and γ weight the two information-preserving terms.
- Proposed approach: At each incremental step, student Mt is initialized from teacher Mt−1 and must expand recognition from N base classes to N + k classes.For an input image i, the two models generate class-specific attention maps for comparison.
- Attention distillation loss (LAD): LAD compares normalized, vectorized attention maps for the top base class predicted by the student on an image from a new class.It uses the sum of element-wise L1 differences; L1 performed better than L2 on held-out data.
- Attention distillation loss (LAD): The penalty uses teacher attention outputs as traces of base data, encouraging the student to produce similar responses to regions resembling base classes.This constrains divergence between the models without explicitly storing base-class images.
- Attention distillation loss (LAD): Attention maps capture gradient-flow and spatial information that class-score distributions alone do not explicitly represent.Matching them is intended to preserve class-specific interpretation between teacher and student more completely than matching scores alone.
5. Experiments
The experiments evaluate incremental training protocols, datasets, and configurations using established baselines and iterative teacher-student updates. Attention-map examples compare LwM with C and LwF-MC across incremental steps.
- Qualitative results: Figure 4 compares C, LwF-MC, and LwM attention maps for base-class images across the initial teacher state and four incremental steps.The figure reports that LwM maps remain closer to the initial M0 maps over time than the other configurations.
- Baselines and configurations: The baseline LwF-MC uses classification and distillation losses, while the study compares experiment configurations identified by experiment IDs.The implementation follows iCaRL’s LwF-MC implementation because LwF-MC is the selected baseline.
- Datasets: The experiments use datasets from LwF-MC and additionally evaluate Caltech-101 and CUBS-200-2011 under disjoint class batches.Data preparation and evaluation follow the LwF-MC protocol for fair comparison.
- Experimental protocol: Before incremental learning, teacher M0 is trained on 10 base classes for 10 epochs with cross-entropy classification loss.Each later student Mt is initialized from Mt−1 and receives a new batch of images.
- Experimental protocol: At every incremental step, the student is trained for 10 epochs using classification loss on new classes together with an information-preserving penalty.The trained student then becomes the teacher for the next incremental step.
- Implementation details: ResNet-18 is used for iILSVRC-small, Caltech-101, and CUBS-200-2011, while ResNet-34 is used for iCIFAR-100.The learning rate is 0.01, and Grad-CAM generates attention maps from the final convolutional feature maps.
6. Results
LwM preserves base-class information while incrementally learning new classes, outperforming the compared baselines across the reported datasets and configurations. Its attention distillation loss further improves performance beyond classification and standard distillation losses alone.
- Qualitative results: LwM retains base-class attention regions across all shown incremental steps and produces attention maps most similar to the targets.These qualitative results support delayed forgetting of base-class knowledge.
- Quantitative results: More than 30% improvement over LwF-MC occurs on iILSVRC-small once the classifier reaches 40 or more classes.For 100 classes, the reported improvement exceeds 50%.
- Quantitative results: LwM outperforms iCaRL at every incremental step on iILSVRC-small despite iCaRL storing base-class exemplars during student training.The comparison is reported as an advantage for LwM under a setting without base-class data.
- Quantitative results: LwM outperforms LwF-MC for all incremental batch sizes—10, 20, and 50 classes—on iCIFAR-100.The paper concludes that LwM consistently outperforms LwF-MC on both iILSVRC-small and iCIFAR-100.
- Ablation: Performance improves progressively from classification loss alone to LwF-MC with distillation loss and then to LwM with added attention distillation loss.Classification loss alone performs poorly because of catastrophic forgetting.
7. Conclusion and future work
The paper concludes that combining distillation and attention distillation enables incremental object-class learning without base-class data. It reports broad gains in the investigated scenarios and identifies incremental segmentation as future work.
- Conclusion: LwM combines distillation loss with attention distillation to transfer base-class knowledge from teacher to student without base-class data during training.The approach uses attention maps to constrain teacher–student knowledge transfer.
- Conclusion: LwM outperforms the baseline in all investigated scenarios.This is the paper’s stated overall conclusion about the evaluated classification settings.
- Future work: The authors propose extending LwM to incremental segmentation, where scarce ground-truth maps make the task challenging.They motivate the extension by noting that visual attention is meaningful for segmentation.