PaperlayerView exact revision on arXiv

Source-linked AI summary

Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning

James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, Zsolt Kira

arXiv:2106.09701v2cs.CVcs.LG

TL;DR

The paper studies data-free class-incremental learning, where models must learn new concepts without storing past data despite catastrophic forgetting and replay constraints. It diagnoses synthetic replay’s feature-domain bias and introduces a modified distillation strategy. The method achieves state-of-the-art DFCIL performance, including a 25.1% increase in final task accuracy over DeepInversion, while outperforming several replay baselines.

  • Problem

    Data-free class-incremental learning asks how models can incorporate new information without storing past data, which matters under memory and data-legality constraints.

  • Method

    The method uses modified cross-entropy training, importance-weighted feature distillation, and linear head fine-tuning to separate new- and past-task feature embeddings.

  • Results

    25.1% increase in final task accuracy over DeepInversion is reported for common class-incremental benchmarks, and the method outperforms Naive Rehearsal and LwF with a coreset.

  • Takeaways & Limitations

    The method achieves state-of-the-art performance for DFCIL and is comparable to state-of-the-art replay-based approaches without storing replay data.

Abstract

from arXiv · show

Modern computer vision applications suffer from catastrophic forgetting when incrementally learning new concepts over time. The most successful approaches to alleviate this forgetting require extensive replay of previously seen data, which is problematic when memory constraints or data legality concerns exist. In this work, we consider the high-impact problem of Data-Free Class-Incremental Learning (DFCIL), where an incremental learning agent must learn new concepts over time without storing generators or training data from past tasks. One approach for DFCIL is to replay synthetic images produced by inverting a frozen copy of the learner's classification model, but we show this approach fails for common class-incremental benchmarks when using standard distillation strategies. We diagnose the cause of this failure and propose a novel incremental distillation strategy for DFCIL, contributing a modified cross-entropy training and importance-weighted feature distillation, and show that our method results in up to a 25.1% increase in final task accuracy (absolute difference) compared to SOTA DFCIL methods for common class-incremental benchmarks. Our method even outperforms several standard replay based methods which store a coreset of images.

1. Introduction

Class-incremental learning must incorporate new classes while avoiding catastrophic forgetting, but replay-based solutions demand storage that can conflict with device-memory and data-legality constraints. This paper diagnoses why synthetic replay can fail and proposes a DFCIL solution that improves final accuracy.

  • Class-incremental learning updates models with new environments or situations while risking catastrophic forgetting of earlier tasks.
  • Replay-based incremental learning methods require extensive memory, creating challenges for on-device applications and legally sensitive data.
  • Model inversion offers a DFCIL alternative to jointly training a generative replay model, which is more computationally and memory intensive and may memorize sensitive data.
  • Synthetic replay can make features distinguish real and fake images rather than separating task 1 and task 2.
  • The proposed method uses modified cross-entropy training, importance-weighted feature distillation, and linear head fine-tuning to separate new- and past-task embeddings.
  • 25.1% increase in final task accuracy is reported over DeepInversion on common class-incremental benchmarks, with performance also exceeding Naive Rehearsal and LwF with a coreset.

2. Background and Related Work

Prior incremental-learning methods mitigate forgetting through architectural expansion, replay, or regularization, while data-free distillation synthesizes images from a trained network without retaining training data. Existing data-free class-incremental methods remain limited in scope or fail on difficult benchmark settings.

  • Incremental-learning approaches include architecture expansion, stored-data replay, generative replay, and weight- or prediction-space regularization.
  • Knowledge distillation regularizes predictions or features using a frozen model and distillation images, but conventional methods require stored data.
  • Data-free knowledge distillation synthesizes distillation images using the trained inference network and its activation statistics, supporting settings where training data is sensitive or unavailable.
  • Only two class-incremental methods were identified as data-free, and DeepInversion succeeded only for short task sequences with very distinct image content.
  • The paper addresses DeepInversion’s failure on difficult class-incremental problems by proposing a solution for successful DFCIL.

3. Preliminaries

Class-incremental learning presents non-overlapping class subsets across sequential tasks and requires learning new classes while retaining performance on earlier ones. The model notation distinguishes training time, task history, and the logits associated with previously introduced classes.

  • A class-incremental model receives labeled data for M classes across N tasks, with each task containing a non-overlapping subset of classes.
  • The objective is to classify newly introduced object classes while retaining performance on previously learned classes.
  • θi,n denotes the model at time i trained with classes from task n, while θn,1:n denotes its logits for all tasks through n.

4. Baseline Approach for Data-Free Class Incremental Learning

The baseline synthesizes replay images from a frozen prior model and trains with losses for confidence, class diversity, feature-statistic alignment, smoothness, and knowledge distillation. Its distillation formulation aligns teacher and student outputs despite newly added classes.

  • The baseline trains an image generator or synthesized images from the prior discriminative model without retaining past training data.
  • Synthetic images are optimized for diverse class predictions so past-task classes are generated at approximately equal rates.
  • Content loss encourages confident predictions, while combining it with diversity loss makes synthetic images represent past-task classes.
  • Stat alignment loss matches synthetic-image intermediate features to batch-normalization statistics stored in the previous model.
  • A smoothness prior minimizes the L2 distance between each synthesized image and its Gaussian-blurred version.
  • Knowledge distillation appends zeros for new classes to the teacher probability vector, aligning teacher and student dimensions for transfer.

5. Diagnosis: Feature Embedding Prioritizes Domains Over Semantics

The analysis uses Mean Image Distance (MID) to compare embedded feature distributions and diagnose whether representations prioritize domain or semantics. DeepInversion makes synthetic past-task data more dissimilar from real past-task data than real current-task data, whereas the proposed method reduces this discrepancy.

  • MID measures the distance between mean embedded images, with higher scores indicating more dissimilar features and lower scores indicating more similar features.The distance is normalized by the standard deviation of the reference distribution to reduce the impact of highly deviating features.
  • For DeepInversion, real task 1 data is more distant from synthetic task 1 data than from real task 2 data.This comparison indicates that the embedding space prioritizes domain over semantic task identity.
  • For the proposed method, real task 1 and synthetic task 1 data have a much lower MID, indicating that embeddings prioritize semantics over domain.

6. A New Distillation Strategy for DFCIL

The proposed DFCIL strategy separately addresses new-task learning, past-task feature preservation, and class-boundary separation while reducing bias between real and synthetic data. It combines local cross-entropy, feature distillation, and importance weighting to balance plasticity with retention.

  • Overall strategy: The method balances learning new-task features, minimizing past-task feature drift, and separating overlap between new and past classes.
  • Learning current task features: Local cross-entropy trains new-task features using only the current task head, excluding synthetic data and past-task linear heads.
  • Minimizing feature drift: Feature distillation operates on penultimate-layer representations to directly reduce drift over previous-task data.The approach is motivated by domain differences between synthetic distillation images and real current-task images.
  • Minimizing feature drift: Standard knowledge distillation preserves important past-task components but can inhibit plasticity, whereas prediction distillation can permit feature drift and real-versus-synthetic bias.
  • Importance weighting: Importance-weighted feature distillation preserves high-impact features while allowing less important features to adapt to the new task.Frozen past-task linear heads estimate how feature changes affect class distributions.
  • Final objective: The final objective combines current-task, knowledge-distillation, and linear-head terms with task-balancing weights.The objective is visualized in Figure 3, and the model updates only the current-task parameters while earlier layers remain frozen in the described step.

7. Experiments

The experiments evaluate data-free class-incremental learning across CIFAR-100 and ImageNet settings, comparing the proposed method with data-free and replay-based baselines. Results show strong performance despite avoiding stored replay data, including close performance to replay on large-scale images.

  • Experimental setup: The evaluation compares the proposed method with Deep Generative Replay, Learning without Forgetting, DeepInversion, offline upper-bound training, and a classification-only Base model.These baselines do not store training data for rehearsal, while offline training provides an upper bound.
  • CIFAR-100 benchmark: CIFAR-100 experiments cover data-free class-incremental learning with 5, 10, and 20 tasks, reporting results averaged over 3 runs.The benchmark uses 100 classes of 32x32x3 images and a 32-layer ResNet.
  • CIFAR-100 benchmark: 25.1% increase in final task accuracy over DeepInversion is achieved on CIFAR-100, while the proposed method also outperforms the generative approach.The paper attributes the poor performance of standard distillation to feature embeddings prioritizing domains over semantics.
  • Replay comparison: The proposed method performs significantly better than LwF and Rehearsal, while landing roughly between BiC and LwF despite storing no replay data.BiC remains ahead, leaving a considerable expected gap between the data-free method and the strongest replay-based comparison.
  • ImageNet benchmark: On the large-scale ImageNet experiment, the proposed method uses 100 times fewer stored parameters than replay and performs reasonably close to replay while outperforming LwF.Replay stores 2000*224*224*3 = 3e8 parameters, compared with 3.3e6 temporarily stored for image synthesis.

8. Conclusions

The paper concludes that standard synthetic-data distillation performs poorly for class-incremental learning, while its method achieves state-of-the-art data-free performance comparable to replay-based approaches. Eliminating stored replay data could reduce memory and privacy burdens in several computer vision applications.

  • Conclusions: Existing class-incremental methods perform poorly when real current-task data is combined with synthetic distillation data to preserve past knowledge.The paper presents this failure as the motivation for its alternative incremental distillation method.
  • Conclusions: The proposed method achieves state-of-the-art data-free class-incremental learning performance and is comparable to state-of-the-art replay-based approaches.The claimed practical vision is to eliminate stored replay data in class-incremental learning.
  • Conclusions: Data-free incremental learning could reduce memory requirements and avoid transferring or tracking private data in autonomous vehicles, medical imaging, and personal-device recommendation systems.These applications are presented as examples of potential impact rather than evaluated deployment results.

A. Additional Results

Additional experiments extend evaluation across CIFAR-100 and Tiny-ImageNet, diagnose feature-space behavior with MMD, and examine scaling limitations. The method remains competitive under data-free constraints while approaching replay-based performance with substantially less stored state.

  • Our approach consistently outperforms LwF.MC on CIFAR-100, while E2E performs slightly worse than BiC.
  • Tiny-ImageNet results show the method outperforms all data-free approaches and performs slightly worse than replay methods storing 2000 images.The benchmark uses 20 tasks with 10 classes per task; the reported upper-bound performance drops from 69.9% to 55.5%.
  • 2.5e7 replay parameters exceed the 8.5e6 parameters temporarily stored for synthesis, and the temporary memory can be removed with additional computation.
  • Scaling to 224x224x3 images remains unresolved: the method and LWF.MC perform similarly, while scaling data-free generative distillation may require one generator per class.
  • MMD shows DeepInversion makes synthetic task-1 features more dissimilar to real task-1 features than real task-2 features, indicating domain-over-semantics prioritization.
  • For the proposed method, lower MMD between real and synthetic task-1 features indicates that feature embeddings prioritize semantics over domain.

C. Additional Experiment Details

The experiments use standard image augmentations, multiple GPU types, and synthesized samples from F at every training step. Hyperparameters are selected through restricted three-fold cross-validation rather than using holdout data from all tasks.

  • Training uses standard augmentations, including random horizontal flips and crops.
  • Experiments use Titan X and 2080 Ti GPUs, with synthesized images sampled from F at each training step.
  • Hyperparameters are tuned by grid search with three-fold cross-validation on half of the tasks.
  • The full task set is excluded from hyperparameter tuning because holdout data across all tasks may violate continual-learning assumptions.

E. Discussion of Class Shuffling Seeds

The appendix discusses evaluation variability from class and task ordering and provides supplementary result visualizations and tables. It also compares feature visualizations after the second task under identical distillation models and synthetic data.

  • Re-implemented baselines produce slightly lower results than previously reported work.
  • Figure A plots task number t on the x-axis and Ω up to task t on the y-axis.
  • Tables A and B report CIFAR-100 results across 5, 10, and 20 tasks, with Table B using a 2000-image replay coreset.
  • Table C covers data-free Tiny-ImageNet with 20 tasks and 5 classes per task, while Table D reports replay results under the same task structure.
  • Figure C compares t-SNE visualizations of DeepInversion and the proposed method after task two using identical teacher models and synthetic data.
  • Class and task orders are randomly shuffled for each experiment run because class order significantly affects end results.
  • The best-performing class order yields performance similar to prior reported results, while both shuffled and fixed orders are presented as defensible choices.

G. Training Time

Training-time comparisons on the twenty-task Tiny-ImageNet benchmark show that the proposed method occupies an intermediate position. It is faster than BiC but slower than the other compared methods, while inference time is otherwise matched by architecture.

  • Training time is measured in seconds per batch on a single Titan X Pascal GPU for twenty-task Tiny-ImageNet.
  • Our method trains faster than the state-of-the-art replay method BiC but slower than the other methods.
  • All methods use the same model architecture and therefore have the same inference time, except BiC’s small logit-weighting operation.
Loading 2106.09701v2…