PaperlayerView exact revision on arXiv

Source-linked AI summary

Maintaining Discrimination and Fairness in Class Incremental Learning

Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, Shutao Xia

arXiv:1911.07053v1cs.CV

TL;DR

Class incremental learning suffers catastrophic forgetting because models trained on new classes can favor new-class predictions over old classes. The paper combines KD with Weight Aligning, which corrects biased FC-layer weights without extra parameters or a reserved validation set. Experiments on ImageNet-1000, ImageNet-100, and CIFAR-100 report better performance than previous methods.

  • Problem

    Class incremental learning suffers catastrophic forgetting because models trained on new classes can favor new-class predictions over old classes.

  • Method

    The method uses KD to maintain discrimination within old classes and Weight Aligning to correct biased FC-layer weights and maintain fairness between old and new classes.

  • Results

    The proposed method achieves better performance than previous methods on ImageNet-1000, ImageNet-100, and CIFAR-100.

  • Takeaways & Limitations

    The paper suggests that useful information hidden in trained models can support mitigating catastrophic forgetting through discrimination and fairness maintenance.

  • Takeaways & Limitations

    KD’s positive effect is limited when used alone, and its deviation can accumulate when more than two incremental steps make the impaired model a subsequent teacher.

Abstract

from arXiv · show

Deep neural networks (DNNs) have been applied in class incremental learning, which aims to solve common real-world problems of learning new classes continually. One drawback of standard DNNs is that they are prone to catastrophic forgetting. Knowledge distillation (KD) is a commonly used technique to alleviate this problem. In this paper, we demonstrate it can indeed help the model to output more discriminative results within old classes. However, it cannot alleviate the problem that the model tends to classify objects into new classes, causing the positive effect of KD to be hidden and limited. We observed that an important factor causing catastrophic forgetting is that the weights in the last fully connected (FC) layer are highly biased in class incremental learning. In this paper, we propose a simple and effective solution motivated by the aforementioned observations to address catastrophic forgetting. Firstly, we utilize KD to maintain the discrimination within old classes. Then, to further maintain the fairness between old classes and new classes, we propose Weight Aligning (WA) that corrects the biased weights in the FC layer after normal training process. Unlike previous work, WA does not require any extra parameters or a validation set in advance, as it utilizes the information provided by the biased weights themselves. The proposed method is evaluated on ImageNet-1000, ImageNet-100, and CIFAR-100 under various settings. Experimental results show that the proposed method can effectively alleviate catastrophic forgetting and significantly outperform state-of-the-art methods.

1. Introduction

Class incremental learning must acquire new classes while retaining prior knowledge, but vanilla fine-tuning suffers catastrophic forgetting and favors new-class predictions. The paper analyzes KD’s limited role and proposes combining KD-based discrimination maintenance with Weight Aligning for fairness.

  • Motivation: Class incremental learning gradually adds classes, requiring models to learn new data while retaining previously learned experiences.Standard DNNs are typically trained with a fixed number of categories, whereas incremental settings introduce classes from streaming data.
  • Motivation: Vanilla fine-tuning suffers catastrophic forgetting because old-class probabilities become relatively low, causing old samples to be predicted as new classes.The vanilla method expands the output layer to include old and new classes before fine-tuning on new data.
  • Key observations: Knowledge distillation maintains discrimination within old classes but does not alleviate prediction bias toward new classes, limiting its positive effect.The paper reports that KD’s benefits can be overwhelmed by superior outputs for new classes.
  • Proposed solution: The proposed method combines KD for old-class discrimination with Weight Aligning to correct biased FC-layer weights and improve fairness between old and new classes.WA uses information in the trained model without reserving a validation set or tuning additional parameters.
  • Contributions: The authors investigate KD’s positive and negative impacts and report that their method achieves better performance than previous methods.These claims are presented among the paper’s stated contributions and experimental findings.

2. Related Work

Prior work addresses catastrophic forgetting through parameter control, knowledge distillation, rehearsal, and class-imbalance correction. These strategies differ in their memory, parameter, validation, and modeling requirements, while this paper emphasizes a simpler class-imbalance treatment.

  • Parameter Control: Parameter-control methods constrain important old-model weights, but accurately measuring parameter importance across a series of tasks is difficult.EWC uses Fisher information, SI uses optimization-path integrals, and MAS uses gradients of network output.
  • Knowledge Distillation: Knowledge-distillation methods transfer key knowledge from a teacher to a student to preserve capabilities of the old model.Examples include LwF, LwF.MC, and M2KD, which distills from multiple previous model snapshots.
  • Rehearsal: Rehearsal methods replay stored old data or generate old samples, but generative approaches require an additional model and depend on generated-sample quality.The simplest rehearsal strategy stores a small number of old examples for replay during later incremental steps.
  • Class Imbalance: Class-imbalance methods include bias correction, cosine normalization, loss constraints, balanced fine-tuning, and score rectification, often with extra requirements.BiC requires a validation set, while other approaches combine multiple loss terms, skills, or memory contents.
  • Combined strategies: Combined strategies such as iCaRL and EEIL use distillation and rehearsal, whereas this paper targets class imbalance without additional parameters, hyperparameters, or a reserved validation set.The authors position their method as a simple alternative that achieves better performance than previous methods.

3. Motivation

Class incremental learning must acquire new classes while retaining knowledge of previously seen classes. The motivation analysis shows that knowledge distillation preserves discrimination among old classes but leaves a bias toward predicting new classes.

  • Problem: Class incremental learning trains on successive data batches while evaluating the model across all classes seen so far.At each step, the model learns from new data while retaining experiences from earlier data.
  • Baseline: The baseline expands the fully connected output layer and combines cross-entropy with knowledge distillation, using new data and limited rehearsal data.New-class output weights are initialized randomly, and both feature-extraction and fully connected parameters are updated during training.
  • Effect of Knowledge Distillation: After five incremental steps, the combined loss reduced old-to-old errors to 1,012 versus 1,333 with cross-entropy alone.The evaluation used 80 old classes, 20 new classes, 8,000 old-class test samples, and 2,000 new-class test samples.
  • Effect of Knowledge Distillation: Knowledge distillation did not reduce old-to-new errors, which increased to 4,314 from 4,027 with cross-entropy alone.Its loss compares outputs corresponding to old classes, making misclassification into new classes less costly than confusion among old classes.
  • Motivation: The resulting prediction bias toward new classes limits distillation’s positive effect and can accumulate when the biased model becomes a teacher in later steps.The analysis identifies maintaining discrimination within old classes as distillation’s successful role, while fairness between old and new classes remains unresolved.

4. Methodology

The method combines knowledge distillation to preserve discrimination among old classes with Weight Aligning to correct FC-layer bias between old and new classes. WA aligns new-class weight norms using the trained model itself, without extra parameters or a validation set, while preserving within-group relative magnitudes.

  • Maintaining Discrimination: Knowledge distillation transfers knowledge from the old model and maintains discrimination within old classes during training.
  • Maintaining Fairness: Weight Aligning corrects FC-layer bias by aligning the norms of new-class weight vectors to those of old-class vectors.
  • Biased Weights in the FC Layer: The FC layer maps extracted feature vectors to class logits through class-specific weight vectors, with the bias term set to zero for analysis.The weight matrix contains one d-dimensional vector for each seen class.
  • Biased Weights in the FC Layer: New-class weight-vector norms are much larger than old-class norms after incremental steps, which can increase new-class logits and prediction bias.The experiments use CIFAR-100 with five incremental steps and 20 classes per step.
  • Weight Aligning: WA normalizes new-class weights so their average norm equals the average norm for old classes, without changing relative magnitudes within either group.This design is intended to preserve separation among classes within each group.
  • Weight Aligning: After aligning weights, the method rescales new-class output logits by a coefficient, and experiments show reduced prediction bias.
  • Weight Aligning: Because weight norms may not reflect logits when weights contain large negative elements, the method restricts FC-layer weight elements to be positive using weight clipping.The assumption is that extracted features are non-negative, as with ReLU-activated features.

5. Experiments

Experiments evaluate the method across CIFAR-100 and ImageNet class-incremental settings, ablations, confusion matrices, and comparisons with prior methods. Results show that combining KD with WA preserves old-class discrimination and old/new-class fairness, while achieving strong performance without extra tuned parameters or validation data.

  • Effect of Weight Aligning: 11.3% is the average gain from combining KD and WA, exceeding the separate gains of 6.2% for WA and 1.1% for KD.The authors describe this as a synergistic “one plus one greater than two” effect.
  • Effect of Weight Aligning: KD reduces misclassifications among old classes, while WA corrects the tendency to predict old-class objects as new classes.Confusion matrices use log(1 + x) transformed entries for visibility.
  • Effect of Weight Aligning: Weight normalization during training does not significantly improve performance over the cross-entropy and WA variations.The authors attribute this to stricter FC-weight constraints increasing bias in feature extraction layers during adaptation.
  • Comparison to Other Methods: On ImageNet-100, ImageNet-1000, and CIFAR-100, the proposed method outperforms compared state-of-the-art methods across the reported incremental-learning settings.ImageNet experiments use 10 incremental steps, while CIFAR-100 experiments vary between 2, 5, 10, and 20 steps.

6. Conclusions

The paper identifies biased FC-layer weights as an important factor in catastrophic forgetting and combines KD with WA to preserve discrimination and fairness. Experiments on ImageNet-1000, ImageNet-100, and CIFAR-100 show better performance than previous methods.

  • Conclusions: The proposed method maintains discrimination through knowledge distillation and fairness through weight aligning.The conclusion frames these two components as the solution to catastrophic forgetting in class incremental learning.
  • Conclusions: Experiments across ImageNet-1000, ImageNet-100, and CIFAR-100 show better performance than previous methods.The conclusion also suggests that useful information remains hidden in trained models.
Loading 1911.07053v1…