Revisiting Distillation and Incremental Classifier Learning

1 Introduction

• Catastrophic forgetting prevents ANNs from learning new tasks without losing previously learned knowledge, unlike humans.Incremental learning aims to bridge this gap and support systems that acquire new tasks or classes over time.Source: §1 · p. 2
• The paper limits its scope to incremental classifier learning because it retains the central challenges of incremental learning while being more tractable for current ANNs.Source: §1 · p. 2

1.1 Our Contributions:

• iCaRL analysis shows that distillation, rather than NEM classification or herding, primarily accounts for its performance.The authors report that NEM can be replaced after correcting classifier bias, while herding did not outperform random selection in their experiments.Source: §1.1 · pp. 2–3
• The proposed Dynamic Threshold Moving Algorithm computes a scale vector to correct bias introduced by knowledge distillation at no additional cost.The method addresses a bias problem previously identified in knowledge distillation but not given a practical computation strategy.Source: §1.1 · p. 3
• The paper also releases an extensible incremental-learning framework and a reproducibility protocol for comparing datasets and methods.The framework is intended to support future work and easier reproduction of reported results.Source: §1.1 · p. 3

2 Related Work

• Earlier classifier-learning methods added classes by changing classification while keeping the feature extractor fixed, which limited adaptation to newer data.Source: §2 · pp. 3–4
• Recent incremental-learning techniques fall into rehearsal-based and knowledge-preserving categories.Source: §2 · p. 4
• Rehearsal methods retain information from earlier tasks by storing or generating prior-task data and using it during later training.iCaRL stores exemplars, whereas GAN-based methods generate older-class images for rehearsal.Source: §2.1 · p. 4
• iCaRL is a prominent rehearsal method that stores K exemplars from previous classes and applies distillation loss to preserve their knowledge.Source: §2.1 · p. 4
• Knowledge-preserving methods retain prior-task information through auto-encoders, protected weights, or distillation loss [4].These approaches constrain or guide updates so newer learning preserves older knowledge.Source: §2.2 · pp. 4–5

3 Overview of iCaRL

• The herding algorithm selects k class instances whose feature mean approximates the mean feature embedding of all class instances.It computes a class mean and iteratively adds instances that keep the selected-set mean closest to it.Source: §3 · p. 5
• iCaRL stores a bounded number of old examples and aims to maintain classification performance while limiting memory and computation.Its memory budget K specifies the maximum number of stored old instances.Source: §3 · p. 5
• During each increment, iCaRL combines ground-truth classification loss on new and stored data with a distillation loss for retaining prior knowledge.The distillation targets come from a copy of the classifier made before the increment.Source: §3 · p. 5
• At runtime, iCaRL classifies examples using exemplar-based approximate mean embeddings rather than the classifier learned through back-propagation.The resulting NEM classifier uses exemplar-set means and is presented as an approximation to NCM.Source: §3 · p. 6, §3.1 · p. 6
• The iCaRL authors attribute its performance to exemplar construction, NEM classification, and a distillation-loss variant without softened teacher probabilities [2, 4].Source: §3.1 · p. 6

4 Implementation Details

• The authors reimplemented iCaRL in PyTorch and compared their results with the original paper and open-source implementations.This validation supported development of a reusable experimental codebase.Source: §4 · p. 6
• The implementation decouples algorithms, models, and datasets so new components can be added without modifying existing code.Source: §4 · p. 6
• Each experiment automatically records its parameters and Git-repository version hash in metadata intended to make results reproducible.The metadata file is designed to contain the information needed to reproduce an experiment.Source: §4 · p. 6
• Experiments use a CIFAR-modified ResNet32, while MNIST images are resized to 32 × 32 so the same model can be used.CIFAR100 experiments run for 70 epochs per increment, whereas MNIST experiments use 10 epochs.Source: §4.1 · p. 7

Exemplar Selection using Herding

• Random exemplar selection performs remarkably similarly to herding, indicating that herding is unnecessary for class-incremental learning.The CIFAR100 experiment begins with ten classes and adds ten classes per increment; multiple runs use error bars representing one standard deviation.Source: §5.1 · pp. 7–8, Figure 1 · p. 7

Hypotheses Development

• The analysis tests predictions that higher distillation temperature and threshold moving can reduce the difference between the trained classifier and NEM.Both experiments provide conclusive support for the bias-based hypothesis.Source: Hypotheses Development · p. 9

Analysis of Prediction 2

• Threshold scaling makes the trained classifier’s difference from NEM insignificant on CIFAR100.Without scaling, NEM performs better; scaling uses a vector S to remove classifier bias.Source: Analysis of Prediction 1 · p. 9, Figure 2 · p. 9
• NEM’s superiority over the trained classifier is attributed to trained-classifier bias rather than an inherent advantage of NEM.The suspected bias arises from class imbalance and unequal distilled targets, with newer classes contributing disproportionately to the loss.Source: Hypotheses Development · p. 8, Analysis of Prediction 2 · p. 10

Analysis of Prediction 1

• Higher-temperature distillation (T=3) makes the trained classifier perform as well as NEM in the worst-case comparison.The result is stable across T values from 3–10; without temperature adjustment, the trained classifier performs significantly worse.Source: Hypotheses Development · p. 9, Table 1 · p. 10

6.1 Knowledge Distillation in Neural Networks

• The distillation loss can bias classifiers because new-class examples lack ground-truth labels and teacher outputs may underrepresent some old classes.The paper illustrates this with a whale class receiving near-zero teacher probability when new examples are terrestrial animals.Source: §6.1 · p. 11, §6.2 · p. 11

6.3 Algorithm for Scale Computation:

• Because the scale-vector computation mirrors the distillation loss, it can be performed during training at no additional cost.The final prediction applies point-wise multiplication between the classifier output and the scale factor.Source: §6.3 · pp. 12–13
• The scale vector sums target probability distributions from the distillation loss, measuring each class’s relative representation in training targets.Predictions are multiplied pointwise by the reciprocal expected class representation to normalize expected class predictions.Source: §6.3 · p. 12, §6.4 · p. 13

6.6 Results

• Scaling the classifier predictions recovers most of MNIST’s accuracy loss as classes are removed, achieving near-optimal NCM classifier results.The NCM comparison uses training data from all classes at test time, making it an idealized reference.Source: Figure 3 · p. 12, §6.6 · p. 14

6.4 Intuition Behind the Algorithm

• With five MNIST classes removed, threshold moving corrects older-class errors while preserving accurate classification of the new classes.The confusion matrix indicates that the unscaled model struggled on unseen classes, whereas scaling enabled accurate classification across all ten classes.Source: Figure 5 · p. 13, §6.6 · p. 14

6.3 Algorithm for Scale Computation:

• Higher temperature reduces bias: softened targets produce scale values closer to one, whereas T=2 requires substantially larger scaling than T=5.The MNIST visualizations compare threshold moving at T=2 and T=5 after removing classes from training.Source: Figure 4 · p. 12, Figure 3 · p. 12, §6.6 · p. 14

6.6 Results

• On CIFAR100, dynamic threshold moving improves the Trained Classifier toward the performance of the Trained Classifier Scaled.The reported CIFAR100 evaluation appears in Fig. 2(b).Source: §6.6 · p. 14

• Distillation loss on exemplars, rather than herding or Nearest Exemplar Mean classification, is identified as the primary reason iCaRL works well.Source: §7 · p. 14
• Dynamic threshold moving fixes learned classifier bias caused by distillation, with its effectiveness verified empirically.Source: §7 · p. 14
• The released incremental-learning framework supports extensibility to new datasets and models and enables rapid reproduction of the reported results.Source: §7 · p. 14