PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-Incremental Continual Learning into the eXtended DER-verse

Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, Simone Calderara

arXiv:2201.00766v2cs.LGstat.ML

TL;DR

Class-incremental continual-learning models must acquire new classes without catastrophically forgetting earlier knowledge, while existing DER stores past model responses without incorporating later information about those classes. The paper proposes X-DER, which updates replay memories and prepares future classification heads; experiments report higher performance than the state of the art across multiple datasets, alongside ablations and analyses of forgetting and flatter minima.

  • Problem

    Class-incremental continual learning must balance adapting to new tasks with retaining previous knowledge, while DER does not incorporate later information about already stored classes.

  • Method

    X-DER extends DER by editing replay-memory contents with new information about past data and preparing future classification heads using past and present data.

  • Results

    X-DER delivers higher performance and outperforms the current state of the art across multiple datasets, with analyses and ablations supporting its effectiveness against forgetting.

  • Takeaways & Limitations

    Memory content editing and future preparation are supported as useful enhancements to DER, while knowledge distillation and flatter minima receive further analysis in continual-learning settings.

  • Takeaways & Limitations

    DER’s replayed information is limited to classes seen when each example entered memory, leaving it blind to later relations between old and newly discovered classes.

Abstract

from arXiv · show

The staple of human intelligence is the capability of acquiring knowledge in a continuous fashion. In stark contrast, Deep Networks forget catastrophically and, for this reason, the sub-field of Class-Incremental Continual Learning fosters methods that learn a sequence of tasks incrementally, blending sequentially-gained knowledge into a comprehensive prediction. This work aims at assessing and overcoming the pitfalls of our previous proposal Dark Experience Replay (DER), a simple and effective approach that combines rehearsal and Knowledge Distillation. Inspired by the way our minds constantly rewrite past recollections and set expectations for the future, we endow our model with the abilities to i) revise its replay memory to welcome novel information regarding past data ii) pave the way for learning yet unseen classes. We show that the application of these strategies leads to remarkable improvements; indeed, the resulting method - termed eXtended-DER (X-DER) - outperforms the state of the art on both standard benchmarks (such as CIFAR-100 and miniImagenet) and a novel one here introduced. To gain a better understanding, we further provide extensive ablation studies that corroborate and extend the findings of our previous research (e.g. the value of Knowledge Distillation and flatter minima in continual learning setups).

1 INTRODUCTION

The paper addresses catastrophic forgetting in class-incremental learning by extending DER with memory updating and preparation for future classes. X-DER combines these enhancements with analyses of logits replay and flatter minima, achieving higher accuracy than the state of the art across multiple benchmarks.

  • Human intelligence retains old knowledge while acquiring new tasks and can address novel problems with limited dependence on prior experience.
  • Artificial neural networks instead suffer catastrophic forgetting because new knowledge overwrites previously learned knowledge, motivating continual-learning methods.
  • Episodic memory is a reliable continual-learning strategy, but prior work largely leaves unexplored how stored experiences should be rewritten using later insights.
  • X-DER updates past memories with secondary information from present data and prepares future classification heads using past and present data.
  • X-DER achieves higher accuracy than the state of the art on Split CIFAR-100, Split miniImageNet, and the newly introduced Split NTU-60.
  • The paper reviews logits replay, evaluates X-DER’s flatter-minima behavior, and uses extensive ablations to examine the proposed intuitions.

2 RELATED WORK

Related work frames class-incremental continual learning around retaining prior knowledge while adapting to new tasks, with rehearsal and distillation as important approaches. Existing methods include regularization, replay buffers, and teacher-student knowledge transfer.

  • Continual learning seeks plasticity for distributional changes while preserving previously learned knowledge across sequential tasks.
  • Class-incremental learning is emphasized as more general and realistic than task-incremental learning because it focuses on a single-head setting without task information.
  • Regularization methods penalize changes to weights important for previous tasks but can struggle to scale to complex problems.
  • Rehearsal methods store exemplars from previous distributions and replay them with incoming data; Experience Replay provides a simple and effective example.
  • Knowledge Distillation transfers knowledge from a teacher model to a student model and can use previous model snapshots during learning of new tasks.

3 BACKGROUND

Class-Incremental Continual Learning trains on disjoint tasks sequentially, requiring a regularized objective and memory-based replay to preserve earlier knowledge. The background introduces DER and identifies bias and representation weaknesses motivating X-DER’s partition-specific objectives, future preparation, and memory updates.

  • CiCL formulation: CiCL trains a model on one task at a time, with disjoint label sets and distribution shifts between tasks.Its objective combines current-task empirical risk with regularization to support accurate predictions across all tasks.
  • CiCL formulation: Regularization prevents forgetting past knowledge and encourages task-specific classifiers to form a single harmonized classifier.Knowledge Distillation uses past network snapshots as teachers, while logit matching provides a direct implementation of this objective.
  • Replay methods: DER approximates unavailable past-task data by replaying stored examples with model outputs, while DER++ additionally replays ground-truth labels.The memory buffer stores examples and logits recorded when each example enters memory.
  • Output-space partitions: The output space is partitioned into present, future, and future-past logits according to the task being learned and when buffered examples were stored.Future logits correspond to unseen classes, whereas future-past logits describe classes discovered after an example entered the buffer.
  • DER++ limitations: DER++ has a blind spot for future-past information because replay targets contain only classes seen when each example entered memory.Later relations between old examples and newly discovered classes are therefore absent from replay, unlike checkpoint-based distillation methods whose teachers are updated at task boundaries.
  • DER++ limitations: DER++ accumulates bias toward present and future classes: current-task gradients dominate replay, while cross-entropy pushes unseen-class logits negative.X-DER addresses these issues with separate objectives, future-logit regularization, and regularly updated memory predictions; the cited comparison reports that it avoids future-logit bias accumulation.

4 PROPOSED APPROACH

X-DER extends DER by updating replayed information from past tasks and preparing future classification heads for unseen classes. It also mitigates class bias through selective loss application and activation constraints.

  • Memory Revision: X-DER updates memory entries with secondary information from the current task while attenuating future-past logits to limit present-class bias.The attenuation rate is controlled by γ, typically set to 0.75.
  • Future Preparation: X-DER prepares unseen classes by optimizing future logits to produce consistent responses for augmented examples sharing the same class.The method uses class-supervised contrastive learning across future heads and strong data augmentations.
  • Future Preparation: Future logits are also replayed and updated as new classes emerge, allowing the method to account for semantic drift in later tasks.The replay mechanism extends to future heads because unused heads acquire semantic information during training.
  • Bias Mitigation: X-DER restricts cross-entropy on current-task examples to present logits so past classes are not penalized by gradients dominated by novel classes.For memory examples, the method computes softmax over all classes seen so far.
  • Bias Mitigation: X-DER constrains past and future activations to remain below the ground-truth logit, preventing them from outgrowing present responses and causing trivial errors.The penalty strictness is controlled by m, typically set to 0.3.
  • Overall Objective: The overall objective combines DER loss, selective cross-entropy, future preparation, and past/future constraints using separate weighting hyperparameters.The loss-term coverage across classifier partitions is summarized visually in Fig. 5.

5 EXPERIMENTAL ANALYSIS

The experiments evaluate X-DER against continual-learning baselines on Split CIFAR-100, Split miniImageNet, and Split NTU-60 in a unified multi-epoch setup. Across all benchmarks, X-DER achieves higher accuracy and lower forgetting than the current state of the art, while ablations support both memory updating and future-class preparation.

  • Experimental Settings: Experiments use Split CIFAR-100, Split miniImageNet, and the introduced Split NTU-60 benchmark, covering image and graph-based action classification.Split CIFAR-100 has 10 tasks, Split miniImageNet has 20 tasks, and Split NTU-60 targets sequential classification of temporal skeletal graphs.
  • Experimental Settings: All models are trained from scratch in a multi-epoch setup, with hyperparameters selected through grid search and experiments rerun in a common environment.The unified environment is used because experimental settings across prior works can differ in meaningful ways.
  • Metrics and Baselines: Final Average Accuracy (FAA) summarizes accuracy after the last task, while Final Forgetting (FF) measures average maximum accuracy degradation during training.FF is bounded in [−100, 100], and the analysis also considers hyperparameter sensitivity, memory footprint, and training time.
  • Baselines and Competing Methods: Regularization through LwF.MC consistently underperforms online replay methods, supporting replay memory as important for solid class-incremental performance.GDumb can avoid bias by jointly training on stored old and new examples, but discarding stream data lowers its FAA relative to most online methods.
  • Baselines and Competing Methods: Among replay methods, ER-ACE lowers forgetting through separate loss treatment, while rehearsal combined with distillation generally outperforms simple replay.iCaRL balances accuracy across seen tasks, but its performance is sub-optimal on miniImageNet and NTU-60 under the reported settings.
  • Discussion: X-DER delivers higher accuracy and lower forgetting than the current state of the art across all benchmarks.Its incremental accuracy trends show stronger retention on past tasks as training progresses.
  • Discussion: Removing memory updates causes a significant performance drop, especially with smaller memories, while contrastive future preparation yields the lowest FF rates.Future heads and replayed logits remain preferable to omitting future heads, and indiscriminate cross-entropy on future heads further decreases accuracy.

6 MODEL ANALYSIS

The analysis explains X-DER’s behavior through richer distillation targets, future preparation, explanation quality, and flatter optimization minima. Across these analyses, X-DER preserves secondary information, improves transfer to unseen tasks, and reaches minima more tolerant of perturbations.

  • Towards Better “Continual” Teachers: X-DER, iCaRL, and LUCIR predict correct coarse classes more consistently and confidently than methods with less informative distillation targets.These methods achieve lower SS-ERR and SS-NLL, while DER(++)’s targets neglect future logits.
  • Towards Better “Continual” Teachers: X-DER’s updated logits improve secondary-information metrics, with average gains of 6% using logits alone and 6.25% when combined with labels over DER++.The analysis attributes this additional regularization to future past logits providing a better estimate of Bayes class-probabilities.
  • Towards Better “Continual” Teachers: X-DER achieves lower ECE than other approaches on Split CIFAR-100, including baselines using smoothed teachers such as DER(++).The analysis links this result to a better estimation of the underlying Bayes class-probability distribution.
  • Effect of Future Preparation on Unseen Classes: X-DER’s AUC advantage over other rehearsal methods widens as more tasks are seen, indicating better scaling to unseen-task generalization.The evaluation stops training after a seen task, fits nearest-neighbor classifiers on future heads, and summarizes shot-dependent performance with AUC.
  • Measuring the Flatness: X-DER and DER(++) reach flatter minima than other approaches, with X-DER showing greater perturbation tolerance and lower Tr(F) values.The perturbation analysis uses parameter-magnitude-weighted noise, while Tr(F) estimates the sum of the eigenvalues of F.
  • Analysis of Model Explanations for Secondary Targets: Knowledge-distillation methods perform better on both F1 score and pointing accuracy for stitched secondary targets, with X-DER standing out.This provides further evidence that X-DER retains richer secondary information and supports localization of secondary objects.

7 CONCLUSION

The conclusion identifies information loss and classification bias in DER, then presents X-DER as an enhanced method that edits memory and prepares for future classes. Experiments report higher performance, flatter-minima-related explanations, richer secondary-information retention, and favorable preparation for unseen classes, while future work targets predicting incoming tasks.

  • 7 CONCLUSION: DER discards semantic relations between old and novel classes and suffers from classification bias toward recently acquired knowledge.These shortcomings motivate the proposed extensions.
  • 7 CONCLUSION: X-DER introduces innovations including memory content editing and outperforms the current state of the art across multiple datasets.The paper also evaluates mechanisms beyond final accuracy, including memory, minima geometry, and secondary-information retention.
  • 7 CONCLUSION: The secondary-target evaluation is limited to patches from classes shared between CIFAR-100 and COCO 2017, using a 2x-upscaled CIFAR-100 version.Additional implementation details are reported in Appendix G.
  • 7 CONCLUSION: X-DER’s future-preparation technique favorably arranges the model for classes that have not yet been seen.This conclusion follows the paper’s analysis of future logits and unseen-class preparation.
  • 7 CONCLUSION: Future work will investigate strategies for mimicking future data distributions and overcoming the stability–plasticity schema through guesses of incoming tasks.The authors identify this as a direction for broader continual-learning advances.

APPENDIX A EXPERIMENTAL ILLUSTRATION OF L1

The experiments expose DER++’s blindness to future-past classes: replay targets preserve associations available when examples entered memory, favoring classes learned earlier. Reversing task order reverses which inter-class relations the model emphasizes.

  • L1: Future-past blindness: DER++ replay targets are blind to relationships with classes that had not yet been learned when memories were stored.The stored network responses encode secondary-class beliefs at insertion time, before later classes become available.
  • L1: Future-past blindness: DER++-ACE emphasizes relations with classes from previous tasks rather than classes introduced later.The pattern appears in class-wise average predictions for four Split CIFAR-100 classes from the fifth task.
  • L1: Future-past blindness: Reversing class order reverses the neglected and emphasized inter-class relations, showing the effect depends on when related classes are encountered.The reversal changes which associations appear in the model’s predictions.

A.2 X-DER compensates this shortcoming

X-DER addresses future-past blindness by updating replay logits and preparing future class heads, then evaluates the resulting distributions against DER++-ACE and Joint Training. X-DER produces more consistent activations across task orders and captures similarities with both future and past classes.

  • Experimental comparison: The experiment compares DER++-ACE, X-DER, and Joint Training using average output distributions for four classes.Joint Training supplies a non-continuous reference for inter-class similarities.
  • Experimental comparison: DER++-ACE captures only the mouse-hamster similarity in forward order and only the mouse-shrew similarity in reverse order.Hamsters, mice, and shrews are learned in different incremental tasks, so replay targets encode only associations available at storage time.
  • X-DER result: X-DER shows a more consistent activation pattern across both task orders and captures similarities with future as well as past classes.The authors ascribe this pattern to updating future-past targets in replay.
  • Algorithm: X-DER combines separate cross-entropy, DER, and future/past constraint losses into an overall training objective.The algorithm samples memory examples, computes augmented logits, updates stored logits, and takes a gradient step.
  • Algorithm: The future-past mechanism stores attenuated logits for relevant future-past class relationships while updating memory entries.The algorithm specifies an attenuation rate γ, defaulting to 0.8.

APPENDIX D BACKBONE ARCHITECTURES

The experiments use different backbone architectures for the image and skeleton benchmarks. Split CIFAR-100 uses ResNet18, Split miniImageNet uses EfficientNet-B2, and Split-NTU60 uses EfficientGCN-B0 on joint data.

  • Split CIFAR-100: Split CIFAR-100 experiments use ResNet18 with the original filter count and an initial 3 × 3 kernel.Additional architectural details are referred to the original backbone paper.
  • Split miniImageNet: Split miniImageNet experiments use EfficientNet-B2.The passage introduces MBConv as the mobile inverted bottleneck block used in the architecture.
  • Split-NTU60: Split-NTU60 experiments use EfficientGCN-B0, a graph-convolutional network applied only to joint skeleton data.The dataset represents skeletons as spatio-temporal graph data.

E.1 Data Augmentation Techniques

The appendix describes dataset-specific augmentations and a fairness-oriented hyperparameter-selection procedure. Hyperparameters are chosen by grid search, while several training quantities are fixed consistently across methods.

  • Data augmentation: Image benchmarks use random crop after 4-pixel padding, random horizontal flip, and normalization on inputs and memory-buffer samples.The augmentations are applied independently to the input stream and memory buffer.
  • Data augmentation: Split-NTU-60 augments skeletal joints with Gaussian noise and random 3D rotation.The reported Gaussian noise parameters are μ = 0.01 and σ = 0.02.
  • Hyperparameter choice: Hyperparameters for the evaluated methods are selected through grid search on a validation set sampled from 10% of the training data.The appendix lists considered values and highlights the chosen configuration.
  • Hyperparameter choice: Epoch count, learning-rate schedule steps, batch size, and replay batch size are fixed per dataset for all evaluated methods.These quantities are excluded from the hyperparameter-selection list to ensure fairness.

F.1 Sensitivity analysis of hyperparameters

The sensitivity analysis examines how lr, α, β, λ, and η affect final average accuracy on Split CIFAR-100. X-DER is robust to β when α is sufficiently high, while η must remain low enough to avoid overconstraining the model.

  • X-DER is robust to β when α is sufficiently high, approximately α≥0.3.The analysis focuses on the base parameters of Eq. 6.
  • η must be set low enough to avoid overconstraining the model.η regulates the constraint losses in Eq. 11.
  • λ is stable across the tested values.

F.2 Memory and Time usage

The resource analysis compares memory footprint and execution time for models on Split CIFAR-100 with memory size M500. X-DER has memory usage comparable to rehearsal methods but substantially higher runtime than DER++.

  • Analysis setup: Figures 20 and 21 report sensitivity analyses on Split CIFAR-100 for hyperparameters including lr, α, β, λ, and η.
  • Memory usage: X-DER has a memory footprint comparable to DER(++) and ER(-ACE).It is much less demanding than methods using more than one backbone network, including LwF, iCaRL, BiC, and LUCIR.
  • Execution time: X-DER increases runtime by 5× relative to DER++ and has time complexity comparable to GSS.The authors attribute this to repeated augmentation and forward propagation during self-supervised future preparation.

G.2 Primary and secondary datasets

The secondary-target evaluation uses a synthetic dataset formed by adding segmented COCO patches to CIFAR-100 images. Models are initialized on the primary dataset, then evaluated in a frozen-backbone multi-label classification setup.

  • Secondary dataset: The synthetic secondary-target dataset stitches COCO 2017 image patches onto CIFAR-100 examples.Patches are cut using ground-truth instance segmentation masks and upscaled through the CAI super-resolution API before stitching.
  • Secondary dataset: The evaluation uses 50,000 training and 10,000 test examples sampled from random pairs of shared classes.The shared class pool includes objects such as bed, cup, chair, bicycle, elephant, and train.
  • Evaluation procedure: Models are pretrained on upsampled Split CIFAR-100, frozen, and fine-tuned only through the final classifier for multi-label prediction.Each logit uses a sigmoid and training minimizes binary cross-entropy with SGD for 50 epochs; performance is evaluated using F1-score.
Loading 2201.00766v2…