PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Classifier Learning with Generative Adversarial Networks

Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Zhengyou Zhang, Yun Fu

arXiv:1802.00853v1cs.CV

TL;DR

Incremental classifier learning must retain old-class performance even though past data are unavailable, a setting associated with catastrophic forgetting. The paper combines cross-entropy and distillation, estimates old/new class bias, and uses GANs to generate historical data. It reports outperforming state-of-the-art methods across three datasets, while noting performance constraints from GAN-generated data.

  • Problem

    Past data are unavailable during incremental learning, and existing approaches face difficulty retaining old-class performance while balancing old and new classes.

    Source: §1 · p. 1

  • Method

    The method combines cross-entropy and distillation, estimates class bias, and uses GANs to generate historical data in place of retained exemplars.

    Source: §1 · p. 1, §4.1 · p. 3

  • Results

    The method outperforms LwF on CIFAR-100, Flower-102, and MS-Celeb-1M-Base, and the paper reports gains over iCaRL on CIFAR-100.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    GAN-generated historical data can support incremental learning without keeping real old-class data, and the paper reports performance gains over methods using no exemplars.

    Source: §1 · p. 1

    On CIFAR-100, performance using GANs drops from 63.87% to 54.9% compared with real images, which the authors attribute to the challenge of training GANs on that dataset.

    Source: §5.3.2 · p. 6

Abstract

from arXiv · show

In this paper, we address the incremental classifier learning problem, which suffers from catastrophic forgetting. The main reason for catastrophic forgetting is that the past data are not available during learning. Typical approaches keep some exemplars for the past classes and use distillation regularization to retain the classification capability on the past classes and balance the past and new classes. However, there are four main problems with these approaches. First, the loss function is not efficient for classification. Second, there is unbalance problem between the past and new classes. Third, the size of pre-decided exemplars is usually limited and they might not be distinguishable from unseen new classes. Forth, the exemplars may not be allowed to be kept for a long time due to privacy regulations. To address these problems, we propose (a) a new loss function to combine the cross-entropy loss and distillation loss, (b) a simple way to estimate and remove the unbalance between the old and new classes , and (c) using Generative Adversarial Networks (GANs) to generate historical data and select representative exemplars during generation. We believe that the data generated by GANs have much less privacy issues than real images because GANs do not directly copy any real image patches. We evaluate the proposed method on CIFAR-100, Flower-102, and MS-Celeb-1M-Base datasets and extensive experiments demonstrate the effectiveness of our method.

1. Introduction

Incremental learning can cause catastrophic forgetting because past data are unavailable, while existing methods struggle to retain old-class performance and balance old and new classes. The paper combines cross-entropy and distillation, estimates class bias, and uses GANs to generate historical data; it reports gains over iCaRL and LwF on the stated benchmarks.

1. Introduction

  • Past data are unavailable during incremental learning, contributing to catastrophic forgetting and making it difficult to retain old-class performance while balancing old and new classes.Existing exemplar approaches also face scalability and privacy concerns.Source: §1 · p. 1
  • The proposed loss combines cross-entropy and distillation on old exemplars and new samples, while a validation-estimated scalar removes bias toward new classes.The paper reports that bias estimation is stable across validation and test data and that the method outperforms iCaRL on CIFAR-100.Source: §1 · p. 1
  • Using GANs to replace exemplars is reported to outperform LwF without exemplars on CIFAR-100, Flower-102, and MS-Celeb-1M-Base.The authors note GAN image-quality and mode-dropping limitations while describing the approach as more scalable and less privacy-sensitive than retaining past data.Source: §1 · p. 1

2. Related Work

Related work spans incremental-learning methods that either avoid old data or retain some of it. The paper positions GAN-generated old data between these approaches, with the aim of avoiding long-term storage of real examples.

2. Related Work

  • Prior incremental-learning research includes linear, ensemble, and nearest-neighbor classifiers, followed more recently by deep-network methods.Deep-learning approaches are grouped by whether they require old data.Source: §2 · pp. 1–2
  • Methods that avoid old data include freezing old-task output layers, constraining important weights, and distilling old-class knowledge onto new data.The paper places its GAN approach in this category because generated data represent old data without retaining it.Source: §2 · p. 2
  • Other methods retain old data, including iCaRL’s selected exemplars; the paper contrasts these with GAN-generated data, which it expects to have fewer privacy or legal concerns.The Figure 1 comparison describes LwF as distilling on new data, iCaRL as keeping old exemplars, and the proposed method as generating old data with GANs.Source: §2 · p. 2, Figure 1 · p. 2

3. Incremental Classifier Learning

The paper formulates incremental classification as extending a pretrained classifier to additional labeled classes, then develops a combined loss and a bias-removal step. The design uses old-class exemplars alongside new data, with GAN data considered as a replacement for real exemplars.

3. Incremental Classifier Learning

  • Incremental classification extends a pretrained n-class classifier with m newly labeled classes to produce a classifier over n + m classes.The strict setting does not use the old-class training data, though iCaRL reports gains from retaining a small exemplar set.Source: §3 · p. 2

3.1. Design of Loss Function

  • The proposed loss combines distillation and cross-entropy on old exemplars and new samples, using exemplars to align old and new classifier predictions on old classes.Cross-entropy spans all old and new classes; the paper represents residual class imbalance with a scalar on new-class predictions and estimates it on validation data.Source: §3.1 · p. 3

3.2. Loss Function

  • The loss has distillation and cross-entropy terms, combined linearly with a scaler λ that balances their contributions.The distillation temperature scaler T is set to 2.Source: Figure 2 · p. 3, §3.2 · p. 3

3.3. Bias Removal

  • Because old exemplars are much fewer than new samples, the classifier is biased toward new classes; a multiplier β on new-class outputs represents this bias.The bias is estimated on a validation set and applied to the classifier; the paper reports consistency between validation and test sets.Source: §3.3 · p. 3

4. Generating Exemplars Using GANs

The paper uses GAN-generated samples as a representation of old-class data, labels them with the old classifier, and filters generated images before training the new classifier.

4. Generating Exemplars Using GANs

  • GANs are proposed to represent old-class data because selected exemplars may differ from the full training distribution, and generated samples are expected to raise fewer privacy concerns.The authors note that GANs do not copy real image patches.Source: §4 · p. 3

4.1. Combining GANs and Classifier

  • The generator is trained on all old-class training data, and the old classifier assigns a label to each generated image.The generator takes random noise as input; the method uses vanilla GANs, DCGAN architecture, and the WGAN earth-mover loss to encourage variation and coverage.Source: §4.1 · pp. 3–4

4.2. Data Selection

  • Because GANs may produce unrelated or mixed-class images, the method retains generated samples only when their maximum likelihood over the old classes exceeds threshold θ.The selected generated images and labels replace real exemplars when training the new classifier.Source: §4.2 · p. 4

5. Experiments

The experiments section introduces datasets and implementation details, then compares methods and discusses model parameters.

  • The experiments cover dataset descriptions and implementation details before presenting comparisons with state-of-the-art methods and model-parameter discussions.Source: §5 · p. 4

5.1. Datasets

The experiments use incremental class-learning setups on CIFAR-100, Flower-102, and MS-Celeb-1M-Base, with dataset-specific splits and training protocols.

5.1. Datasets

  • CIFAR-100 has 100 classes with 500 training and 100 testing images per class; experiments use P=2, 5, 10, or 20 incremental parts.Classes follow a fixed random order, with C=100/P classes per part; the first part trains a classifier that is then adapted to all 100 classes.Source: §5.1 · p. 4
  • Flower-102 has 102 categories; the authors train on the original 6,149-image test set and test on the original 1,020-image training set.The 102 classes are divided into two 51-class parts for incremental training.Source: §5.1 · p. 4
  • MS-Celeb-1M-Base contains 20,000 classes and 1.2 million aligned face images; the authors randomly split classes equally into two incremental parts.For each person, 80% of up to 30 images are randomly selected for training, with the remainder used for testing.Source: §5.1 · p. 4

5.2. Implementation Details

Implementation uses ResNet models with dataset-specific training schedules, and evaluates performance using standard Top-1 accuracy.

5.2. Implementation Details

  • CIFAR-100 uses a 32-layer ResNet with 70 epochs per step, batch size 128, and learning-rate reductions after epochs 49 and 63.Weight decay is 0.0002; the learning rate starts at 0.1 and drops to 0.01 and 0.001.Source: §5.2 · p. 4
  • Flower-102 fine-tunes an 18-layer ResNet pretrained on ImageNet ILSVRC 2012, while MS-Celeb-1M-Base uses a 34-layer ResNet.Both use 90 epochs, batch size 256, weight decay 0.0001, and a learning rate reduced by a factor of ten every 30 epochs.Source: §5.2 · p. 4
  • The reported evaluation metric is standard Top-1 accuracy, and the authors state that their code will be released on GitHub.Source: §5.2 · p. 4

5.3. Comparisons and Results

Across CIFAR-100, Flower-102, and MS-Celeb-1M-Base, Ours-Real improves on iCaRL with real exemplars, while Ours-GANs outperforms LwF without exemplars. Results also show gains across CIFAR-100 class splits, balanced confusion across batches, and a performance cost from using GAN-generated rather than real images.

5.3.1 Ours-Real vs iCaRL with Real Exemplars

  • On Flower-102, Ours-Real outperforms iCaRL by 1% using an exemplar set of 255.The training schedule runs for 90 epochs, with learning-rate decreases after epochs 30 and 60.Source: §5.3.1 · p. 5
  • Ours-Real outperforms iCaRL on all three datasets; on CIFAR-100, it gains 2.07% accuracy when incrementally learning from 50 to 100 classes.The authors consider their different ResNet-18 implementations on par based on full-data and first-50-class accuracy, and attribute the gain to their loss and unbalance removal.Source: Table 1 · p. 5, §5.3.1 · p. 5
  • For CIFAR-100 splits of 20, 10, and 5 classes, Ours-Real’s average gains over iCaRL are 2.36%, 2.03%, and 2.97%, respectively.Ours-Real performs better across all incremental batches; softmax leads sigmoid by a large margin initially, with the margin narrowing after several batches.Source: §5.3.1 · p. 5, Table 2 · p. 5, Table 3 · p. 5
  • Confusion is reported as balanced within and across batches for splits of 5, 10, 20, and 50 classes.Figure 3 presents the batch confusion matrices for those four split sizes.Source: §5.3.1 · p. 5, Figure 3 · p. 6
  • With all 50 old CIFAR-100 classes available as exemplars, Ours-Real matches batch-training performance in the incremental setting, while iCaRL does not reach its own batch-training upper bound.The paper attributes this difference to Ours-Real’s loss reducing to the batch loss when λ is zero, whereas iCaRL uses a different incremental-learning loss.Source: §5.3.1 · pp. 5–6, Table 5 · p. 6

5.3.2 Ours-GANs vs LwF without Real Exemplars

  • Ours-GANs outperforms LwF on all three datasets without exemplars, with a 25.39% advantage on Flower-102.The reported CIFAR-100 accuracy is 54.9% for Ours-GANs versus 52.5% for LwF.Source: §5.3.2 · pp. 6–7, Table 6 · p. 6
  • On CIFAR-100, accuracy drops from 63.87% with real images to 54.9% with GAN-generated images, though Ours-GANs still exceeds LwF at 52.5%.The authors attribute the drop to the challenge of training GANs on CIFAR-100 and report lower image quality; selected GAN and real images nevertheless share some shapes and colors.Source: §5.3.2 · pp. 6–7, Figure 4 · p. 7

5.4. Model Analysis

The analysis tests exemplar-selection sensitivity, the loss-balance parameter λ, bias correction β, and GAN data sampling. It finds similar performance with random exemplar selection, stable results across a range of λ, and gains from bias correction, while GAN-generated data show a validation–test mismatch.

5.4.1 Sensitivity to Exemplar-Selection

  • Random exemplar selection achieves performance similar to iCaRL exemplar management with an exemplar set of 2,000.Random selection results average five runs.Source: §5.4.1 · p. 7, Table 7 · p. 7

5.4.2 Loss Scalar λ

  • Results are stable for λ from 0.3 to 0.7, so the paper sets λ = 0.5 for real-data experiments.λ balances distillation and cross-entropy losses; using only either loss is reported as a poor choice.Source: §5.4.2 · pp. 7–8, Table 8 · p. 8
  • For GAN-generated data, λ is set empirically to 0.9 because estimating it is difficult given the apparent differences from real images.The authors reason that GAN data require a larger λ because outputs from the old classifier are more reliable than pseudo-labels.Source: §5.4.2 · p. 8

5.4.3 Bias Scalar β

  • β = 0.7 achieves 9.8% and 6.7% gains on the validation and test sets, respectively, with real exemplars.The validation and test accuracy-β curves are correlated; the bias-removed classifier also outperforms iCaRL by 2.07% in accuracy despite using less training data.Source: §5.4.3 · p. 8, Figure 5 · p. 8
  • For GAN-generated data, β = 0.8 gives 54.90% top-1 validation accuracy, while β = 0.7 gives the best test accuracy, 56.28%.The validation and test sets are not perfectly aligned for GAN data; the validation-based estimate is reported to incur only a 1.38% performance drop. Confusion matrices show bias shifting from new toward old classes as β decreases, with less confusion among old classes for GAN data, attributed to lower variation in generated data.Source: §5.4.3 · pp. 8–9, Figure 6 · p. 9

5.4.4 GANs Data Sampling

  • GAN sampling selects images above an empirical threshold θ, using top-K images when enough exceed the threshold.The settings are θ = 0.95, K = 50 for CIFAR-100; θ = 0.3, K = 60 for Flower-102; and θ = 0.3, K = 10 for MS-Celeb-1M-Base. The CIFAR-100 threshold differs because its classifier is not as good as the other two.Source: §5.4.4 · p. 9

6. Conclusions and Future Work

The conclusion presents a combined loss and validation-based bias correction for incremental classifier learning, alongside GAN-generated past data. The method is reported to outperform state-of-the-art methods on three datasets, while improving the GAN generator is left for future work.

  • The method addresses inefficient loss integration, imbalance between old and new classes, and exemplar scalability and privacy concerns.Source: §6 · p. 9
  • The proposed loss combines distillation and cross-entropy, while bias is represented by a multiplier on predictions for new classes and estimated from validation data.Source: §6 · p. 9
  • The method is reported to outperform state-of-the-art methods on CIFAR-100, Flower-102, and MS-Celeb-1M-Base; future work targets narrowing the gap between GAN data and real images.Source: §6 · p. 9

Related papers

Loading 1802.00853v1…