PaperlayerView exact revision on arXiv

Source-linked AI summary

Mnemonics Training: Multi-Class Incremental Learning without Forgetting

Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, Qianru Sun

arXiv:2002.10211v6cs.CVstat.ML

TL;DR

MCIL must learn new concepts without forgetting previous ones, yet limited exemplar replay can be ineffective when selected examples are unrepresentative. Mnemonics makes exemplars optimizable through bilevel training and improves incremental-learning performance, achieving 61.00% average accuracy and 19.17% forgetting on 25-phase ImageNet.

  • Problem

    MCIL faces catastrophic forgetting, while heuristically selected few exemplars may inadequately represent previous classes and weaken incremental classification.

  • Method

    Mnemonics parameterizes exemplars as optimizable inputs and alternates model- and exemplar-level learning through global and local bilevel optimization.

  • Results

    61.00% average accuracy and 19.17% forgetting rate are achieved by LUCIR with mnemonics on 25-phase ImageNet, alongside consistent improvements across methods.

  • Takeaways & Limitations

    Mnemonics exemplars can serve as flexible replacements for original data, often lying near class boundaries and yielding clearer class separation.

  • Takeaways & Limitations

    The framework targets the assumption that models trained on few exemplars also minimize loss on original data, which heuristic selection does not guarantee.

Abstract

from arXiv · show

Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a few examples of the previous concepts but the effectiveness of this approach heavily depends on the representativeness of these examples. This paper proposes a novel and automatic framework we call mnemonics, where we parameterize exemplars and make them optimizable in an end-to-end manner. We train the framework through bilevel optimizations, i.e., model-level and exemplar-level. We conduct extensive experiments on three MCIL benchmarks, CIFAR-100, ImageNet-Subset and ImageNet, and show that using mnemonics exemplars can surpass the state-of-the-art by a large margin. Interestingly and quite intriguingly, the mnemonics exemplars tend to be on the boundaries between different classes.

1. Introduction

MCIL systems suffer catastrophic forgetting because updates for new classes can overwrite previously learned knowledge, while heuristic exemplar selection provides weak protection. Mnemonics addresses this by optimizing parameterized exemplars end-to-end through alternating model-level and exemplar-level bilevel optimization.

  • Current machine-learning systems suffer catastrophic forgetting when continuous updates overwrite knowledge acquired from previous data, especially in multi-class incremental learning.
  • Around 25% accuracy is lost by iCaRL with herding when predicting 50 previous CIFAR-100 classes in the final 100-class phase, relative to using all examples.
  • Mnemonics parameterizes exemplars with image-size parameters and optimizes them end-to-end, learning new-class exemplars while adapting earlier exemplars to current data distributions.
  • A global bilevel optimization program alternates optimization of the conventional model and parameterized exemplars across incremental training phases.
  • A local bilevel program trains exemplars for new classes and adjusts old-class exemplars end-to-end, complemented by experiments, visualization, and feature-space explanation.

2. Related Work

Prior incremental-learning work distinguishes multi-task and multi-class settings, with this paper focusing on multi-class incremental learning (MCIL). Existing approaches replay herding exemplars or synthesize old samples with GANs, while the proposed framework instead alternates optimization of models and exemplars across phases.

  • Incremental-learning settings: Incremental learning studies settings where data from different classes arrive gradually, including multi-task and multi-class incremental learning; this work targets MCIL benchmarks.Multi-task methods use classes from different datasets, whereas multi-class methods use classes from the same dataset.
  • Exemplar replay: Knowledge-distillation methods replay old knowledge using a transfer set, while representation-learning approaches store herding exemplars selected near each class’s average sample.Castro et al. combined the same herding exemplars with balanced fine-tuning and temporary distillation.
  • Generative replay: Generative replay methods use GANs to synthesize old samples, but their performance depends on GANs that are difficult to train and their models require memory.These limitations may make GAN-based methods inapplicable to MCIL.
  • Bilevel optimization: The proposed method formulates training as a global bilevel optimization alternating MCIL-model and mnemonics-exemplar updates across phases, with a local bilevel optimization inside each phase.The global procedure optimizes model and exemplar parameters alternately, while the local procedure learns or adjusts mnemonics exemplars.

3. Preliminaries

This section defines MCIL as incremental classification over class-specific data phases, contrasting it with conventional classification, and specifies the losses and evaluation protocol used. Models are trained with classification and distillation losses and evaluated by average accuracy across phases.

  • MCIL setting: MCIL evaluates classification models learned incrementally from a sequence of data involving different classes, unlike conventional classification with all classes available initially.Its uniform setting is used in related work, and class samples arrive in different time phases.
  • Training losses: Training combines a classification loss with an MCIL-specific distillation loss to preserve prediction behavior between the current and previous models.The distillation loss encourages the new Θ_i and previous Θ_i−1 to maintain the same predictions.
  • Evaluation: After each phase, the model is evaluated on unseen data for all classes observed so far, and final performance is the average accuracy over all N +1 phases.This evaluation protocol follows prior MCIL work.
  • Distillation loss: The distillation temperature scalar τ is set above 1, assigning larger weights to smaller values.This temperature parameter appears in the distillation formulation.
  • Classification loss: The classification objective uses softmax cross entropy over the M classes present in D_0:i.The formulation uses the ground-truth label y and an indicator function δ_y=k.

4. Mnemonics Training

Mnemonics training alternates model and exemplar optimization across incremental phases through global and local bilevel optimization programs. It also adjusts old exemplars online and fine-tunes models on balanced exemplar sets to reduce classification bias.

  • Training framework: The framework alternates learning classification models and optimizable mnemonics exemplars across all incremental phases.Mnemonics exemplars are parameterized and adjusted online rather than treated only as fixed data samples.
  • Global bilevel optimization: A global BOP couples model-level learning on new data and old exemplars with exemplar-level optimization across phases.Each phase uses the model and exemplars to constrain one another, while stored exemplars approximate inaccessible old data.
  • Model-level optimization: The model-level problem updates Θ_i by gradient descent from Θ_i−1 using old mnemonics exemplars, after which Θ_i trains exemplar parameters.The objective balances distillation and classification losses with manually set scalar λ.
  • Exemplar-level optimization: A local BOP initializes new exemplars from a random subset of D_i, temporarily trains a model on them, and back-propagates validation loss on D_i to optimize exemplar parameters.This meta-level validation explicitly targets feasible approximation of original-data performance despite few exemplars.
  • Exemplar-level optimization: Old exemplars are split into E^A_0:i−1 and E^B_0:i−1, which alternately serve as optimization and validation subsets to adjust them online.The strategy can also use more than two subsets, optimizing each against its complement.
  • Bias correction: Fine-tuning Θ_i on E_i∪˜E_0:i−1 gives every class the same number of exemplar samples, alleviating bias from imbalanced training counts.The paper illustrates the imbalance with 1000 versus 20 samples between new-data and old-exemplar classes.

5. Weight transfer operations

Weight transfer operations learn layer-wise scaling and shifting parameters to transfer the previous network Θi−1 to Θi while preserving structural knowledge. The framework trains these parameters indirectly with classification and distillation losses, then reconstructs the updated network.

  • Weight transfer purpose: Weight transfer operations learn parameters Ti that transfer network weights Θi−1 to Θi while preserving Θi−1’s structural knowledge during new-class learning.The parameters are trained in the i-th incremental phase.
  • Layer-wise parameterization: For each layer, the method learns neuron-wise scaling parameters for weights and shifting parameters for biases.A layer with R neurons has R weight-scaling and R bias-shifting parameters.
  • Layer-wise parameterization: The complete transfer set Ti comprises the layer-specific scaling and shifting parameters applied sequentially across the network’s Q layers.The parameters operate on layer weights and biases using element-wise multiplication where specified.
  • Indirect optimization: Instead of directly training Θi, the method keeps Θi−1 fixed and trains Ti on Di ∪ E0:i−1 using classification and distillation losses.The loss terms are balanced by a manually set scalar λ.
  • Model reconstruction: After Ti is learned, the method computes the updated network Θi from the transferred parameters.This final step converts the optimized transfer parameters into the incremental model.

6. Algorithm

Algorithm 1 summarizes mnemonics training as alternating classification-model and exemplar learning, with optional memory updates accommodating two MCIL memory-budget settings.

  • Overall process: Algorithm 1 summarizes the overall mnemonics-training process across Steps 1–16, corresponding to the model and exemplar-learning procedures in Sections 4.1–4.3.The algorithm describes alternative learning of classification models and mnemonics exemplars.
  • Training phases: Each phase performs model-level training at Step 8 and exemplar-level training at Steps 11 and 14.The algorithm alternates these two optimization levels within each phase.
  • Memory settings: Step 17 is optional because MCIL settings differ in their memory budgets.The paper evaluates both a fixed number of exemplars per class and a consistently fixed total memory budget across phases.
  • Memory settings: Under a fixed total memory budget, earlier phases can store more exemplars per class, while later phases gradually discard old exemplars.The alternative setting keeps a fixed memory budget throughout all phases.

7. Experiments

Experiments on CIFAR-100 and ImageNet show that mnemonics training consistently improves incremental-learning performance, especially as the number of phases increases. Ablations and visualizations attribute these gains to adaptable, boundary-focused exemplars and dynamic adjustment of old exemplars.

  • Datasets and benchmark protocol: Experiments use CIFAR-100 and ImageNet under the LUCIR protocol, training on half the classes initially and adding the remainder evenly across incremental phases.CIFAR-100 contains 60,000 images from 100 classes, while ImageNet contains about 1.3 million images from 1,000 classes.
  • Compared to the state-of-the-art: Mnemonics training consistently improves baseline architectures, with LUCIR w/ ours reaching 61.00% average accuracy and 19.17% forgetting on 25-phase ImageNet.The method achieves the highest average accuracy and lowest forgetting rate in this setting.
  • Different total phases (N = 5, 10, 25): On ImageNet-Subset, LUCIR w/ ours gains 1.74% in the 5-phase setting and 8.30% in the 25-phase setting.The improvement from mnemonics training becomes larger as the number of phases increases, while related methods suffer performance drops.
  • Ablation study: Across all ablation settings, mnemonics training achieves the highest average accuracies and lowest forgetting rates, while adjusting old exemplars adds an average 0.34% on both datasets.The ablation compares mnemonics variants with random and herding exemplars.
  • Visualization results: Compared with herding, mnemonics exemplars produce clearer class separation and are optimized mainly along class boundaries, supporting high-quality classifiers.The visualization also shows stronger intra-class cohesion despite greater confusion when more classes are learned.

8. Conclusions

The paper introduces mnemonics training for multi-class incremental learning, using exemplars as efficient, flexible, optimizable, and adaptable parameters. The generic framework can be applied to existing methods and yields large-margin improvements.

  • Framework: Mnemonics training is proposed as a framework for multi-class incremental learning.The paper identifies mnemonics exemplars as its main contribution.
  • Mnemonics exemplars: Mnemonics exemplars function as efficient data samples and flexible, optimizable, adaptable parameters.These parameterized exemplars contribute to the flexibility of online systems.
  • Generality: The generic mnemonics training approach can be readily applied to existing methods to achieve large-margin improvements.The passage characterizes the approach as broadly applicable to existing methods.

A. Uniform memory budget experiments

This section evaluates uniform memory budgets, in which each phase stores a fixed total number of exemplars rather than a fixed number per class, and compares related methods with and without mnemonics training.

  • Uniform memory budget setting: Uniform memory budgets store 2,000 or 20,000 total exemplars per phase, allocating more exemplars to earlier classes and discarding some older exemplars.This contrasts with the main paper’s uniform exemplar setting of 20 exemplars per class.
  • Uniform memory budget setting: Table S2 compares related works with and without mnemonics training as a plug-in module, while Figure S1 reports phase-wise results for the best model.The related works are [25] [37].

B. Ablation study for distillation loss

The supplementary ablation study examines the role of distillation loss in LUCIR with the mnemonics training approach and as a plug-in module. It reports average accuracy and forgetting rates under with- and without-distillation settings, with weight transfer applied throughout.

  • Table S1 supplements the paper’s ablation study of the distillation loss.
  • The table reports average accuracies Ā (%) and forgetting rates F (%) for LUCIR with mnemonics training or [28] as a plug-in module.
  • The comparison distinguishes experiments with and without distillation loss, while applying weight transfer operations in every experiment.

C. More visualization results

The supplementary results extend the paper’s visualization analyses with phase-wise accuracy comparisons and exemplar-distance changes under a uniform memory budget. They also evaluate mnemonics training as a plug-in module alongside related methods using average accuracy and forgetting rate.

  • Uniform memory budget evaluation: Table S2 reports average accuracies and forgetting rates for related methods with and without mnemonics training as a plug-in module.The uniform memory budget setting uses random discarding of old exemplars and weight-transfer operations for MCIL model training.
  • CIFAR-100 setup: For CIFAR-100, the initial phase trains on 50 classes and the remaining classes are introduced evenly in subsequent phases.This setup contains 100 classes in total.
  • Phase-wise accuracy visualization: Figure S1 compares phase-wise accuracies for Upper Bound, LUCIR, BiC, iCaRL, LwF, and LUCIR with mnemonics training.Light-color ribbons show 95% confidence intervals, and average accuracies are reported in Table S2.
  • Exemplar-distance visualization: Figure S2 visualizes changes in average distances between exemplars and initial samples for LUCIR with mnemonics training.The curves are smoothed at a rate of 0.8, and the corresponding average accuracy is given in Table 1.
Loading 2002.10211v6…