PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Causal Effect of Data in Class-Incremental Learning

Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, Hanwang Zhang

arXiv:2103.01737v3cs.AI

TL;DR

CIL suffers catastrophic forgetting because new training loses the causal effect of old data. The paper models CIL causally, then distills a colliding effect without replay storage and removes incremental momentum bias. Across CIFAR-100 and ImageNet benchmarks, causal effect distillation consistently improves strong CIL baselines, with reported gains of 0.72%–9.06%.

  • Problem

    CIL must learn new classes without retaining all old data, but new-data training can override old representations and lose the old data’s causal effect.

  • Method

    The paper uses causal graphs to analyze CIL, distills the Colliding Effect as a storage-free analogue of replay, and removes Incremental Momentum Effect bias.

  • Results

    0.72%–9.06%: causal effect distillation improves LUCIR and PODNet across CIFAR-100 and ImageNet benchmarks.

  • Takeaways & Limitations

    The proposed causal solutions are model-agnostic and improve strong CIL baselines while avoiding replay storage for the distilled data effect.

Abstract

from arXiv · show

We propose a causal framework to explain the catastrophic forgetting in Class-Incremental Learning (CIL) and then derive a novel distillation method that is orthogonal to the existing anti-forgetting techniques, such as data replay and feature/label distillation. We first 1) place CIL into the framework, 2) answer why the forgetting happens: the causal effect of the old data is lost in new training, and then 3) explain how the existing techniques mitigate it: they bring the causal effect back. Based on the framework, we find that although the feature/label distillation is storage-efficient, its causal effect is not coherent with the end-to-end feature learning merit, which is however preserved by data replay. To this end, we propose to distill the Colliding Effect between the old and the new data, which is fundamentally equivalent to the causal effect of data replay, but without any cost of replay storage. Thanks to the causal effect analysis, we can further capture the Incremental Momentum Effect of the data stream, removing which can help to retain the old effect overwhelmed by the new data effect, and thus alleviate the forgetting of the old class in testing. Extensive experiments on three CIL benchmarks: CIFAR-100, ImageNet-Sub&Full, show that the proposed causal effect distillation can improve various state-of-the-art CIL methods by a large margin (0.72%--9.06%).

1. Introduction

Class-Incremental Learning forgets old classes when new-data training overrides old representations and loses the old data’s causal effect. The paper frames this process causally and proposes storage-free effect distillation, alongside momentum-effect removal, to recover old-class performance.

  • The CIL problem: CIL discards old data while learning new classes, allowing SGD updates to override old-data parameters and cause catastrophic forgetting.This is especially damaging for deep models because their representations are learned end to end from data.
  • The CIL problem: Fine-tuning only on new classes can replace old discriminative features, such as “stripe” and “furry,” with features for new classes.The resulting model loses discriminative power for old classes at later incremental steps.
  • Existing solutions: Existing anti-forgetting methods replay old data or distill old features and labels to impose effects from earlier training on later training.Replay adds old samples, whereas distillation constrains new-network behavior using features or logits from the old network.
  • Existing solutions: Feature and label distillation can conflict with new-class learning because output-end constraints may violate end-to-end representation learning, especially for out-of-distribution new data.The paper contrasts this with data replay, which preserves the end-to-end learning pathway.
  • Proposed causal approach: The proposed causal framework attributes forgetting to lost old-data effects and introduces Distilling Colliding Effect to recover the desired end-to-end effect without replay storage.The paper also proposes Incremental Momentum Effect Removal to address bias toward abundant new data.
  • Experimental outcome: 0.72%–9.06%: causal effect distillation improves LUCIR and PODNet across CIFAR-100 and ImageNet benchmarks.The method is reported as agnostic to methods, datasets, and backbones, with larger gains when old-data replay is smaller.

2. Related Work

Related work on CIL primarily addresses forgetting through replay-based or distillation-based methods. The paper positions causal inference as a way to explain these techniques through the recovery of vanished old-data effects.

  • Class-Incremental Learning: CIL methods seek a unified classifier that recognizes old and new classes as learning accumulates over successive data batches.The related-work discussion divides anti-forgetting approaches into replay-based and distillation-based branches.
  • Class-Incremental Learning: Replay-based methods include a small percentage of old data in the new training data.These methods address forgetting by retaining representative old examples.
  • Causal Inference: Causal inference provides a formulation in which forgetting reflects vanishing old-data effects and anti-forgetting retrieves those effects.This perspective is presented as the merit of applying causal inference to CIL.

3. (Anti-) Forgetting in Causal Views

The causal framework represents CIL through relationships among old data, new inputs, features, and labels, explaining forgetting as a loss of old-data influence on predictions. Existing anti-forgetting methods restore this influence by unblocking causal paths, though replay preserves end-to-end learning more coherently than feature distillation.

  • Causal Graphs: CIL is modeled as a causal graph whose nodes represent data, features, and labels across consecutive learning steps.The directed edges encode causalities among old data D, new inputs I, current and old features X and Xo, and current and old predictions Y and Yo.
  • Causal Effect: Causal intervention estimates the effect of old data by comparing predictions under the presence and absence of that data.The framework uses do(·) to represent externally assigning a variable and removing its incoming causal links.
  • Forgetting: In ordinary new-data training, all causal paths from old data to predictions are blocked by colliders, producing zero old-data effect and forgetting.One example is D → Xo ← I → X → Y, which is blocked at Xo.
  • Data Replay: Data replay restores a nonzero old-data effect by changing the new-data distribution while retaining an end-to-end path from inputs to predictions.The framework treats replay as introducing a path from D to I, with P(Y | I, D) = P(Y | I) because I mediates the effect.
  • Distillation: Feature and label distillation likewise unblock paths from old data, but feature distillation is not end-to-end in the same way as data replay.Feature distillation regularizes current features through the old network, whereas replay preserves end-to-end representation learning from data to label prediction.
  • Combined Methods: CIL methods can combine replay and distillation mechanisms, as illustrated by iCaRL and LUCIR.iCaRL combines replay with soft-label distillation, while LUCIR combines replay, feature distillation, and fixed old-class classifiers.

4. Distilling Causal Effect of Data

The method distills the colliding effect between old and new data to recover replay-like causal effects without storing replay data, then removes incremental momentum bias to balance predictions.

  • 4.1. Distilling Colliding Effect: Conditioning on the collider X_o makes old data D and testing input I correlated, enabling the colliding effect to be distilled into new training.The method uses classification probabilities and a weight term W(I, X_o, D) to represent the conditioned effect.
  • 4.1. Distilling Colliding Effect: The weight W is larger for new images whose old-network features are closer to the target old feature, so images are sorted by feature distance and truncated to a top-K fragment.The selected weights satisfy W_N1 ≥ W_N2 ≥ ··· ≥ W_NK and are normalized before final prediction.
  • 4.1. Distilling Colliding Effect: The proposed distillation method fundamentally equals the causal effect of data replay without replay-storage cost.It is designed to work with very limited or even no old data.
  • 4.2. Incremental Momentum Effect Removal: Old data scarcity and SGD momentum bias predictions toward new data, producing severe data imbalance across incremental steps.The paper characterizes the resulting distribution as long-tailed and notes that current rare classes may have been frequent in earlier steps.
  • 4.2. Incremental Momentum Effect Removal: Incremental momentum removal pursues the direct causal effect of feature X by subtracting a counterfactual biased head projection from the final logits.A dynamic head direction combines previous and current step directions, while the trade-off parameters α and β are learned by back-propagation.

5. Experiment

Experiments evaluate the proposed causal-effect distillation across CIL datasets, replay budgets, baselines, and incremental settings. The method improves state-of-the-art methods, especially when replay data are scarce, while MER remains effective across incremental-step counts.

  • 5.1. Settings: The evaluation uses CIFAR-100, ImageNet-Sub, and ImageNet-Full under varying incremental-step and replay settings.ImageNet-Sub contains 100 classes, whereas ImageNet-Full contains 1,000 classes; CIFAR-100 contains 100 classes.
  • 5.1. Settings: The experiments use ResNet backbones, SGD with momentum µ = 0.9, and a scaled cosine normalization classifier.A 32-layer ResNet is used for CIFAR100 and an 18-layer ResNet for ImageNet.
  • 5.2. Results and Analyses: 65.42% and 64.12% average accuracy are achieved on CIFAR-100, while ImageNet-Full reaches 67.51% and 65.77% with the LUCIR-based model.On CIFAR-100, DDE improves original LUCIR and PODNet by up to 1.7%; on ImageNet-Sub, it boosts them by 1.79% on average.
  • 5.2. Results and Analyses: 59.11% average accuracy without replay surpasses replaying five samples per class by 13.54%, requiring no stored old images.Traditional replay methods need around 1,000 stored old images to achieve the same result.
  • 5.2. Results and Analyses: Incremental Momentum Effect Removal remains significantly beneficial with 25 incremental steps, while the method steadily improves performance when the initial task has fewer classes.These tests evaluate robustness to incremental-step count and initial-task size.

6. Conclusions

The paper frames CIL anti-forgetting through causal effects and proposes storage-free distillation of the collider effect. It additionally removes incremental momentum effects during testing to balance old- and new-class prediction.

  • 6. Conclusions: Causal graphs explain CIL and anti-forgetting techniques in terms of recovering the causal effect of old data.The framework identifies lost old-training effects as the basis of forgetting and views replay and distillation as restoring them.
  • 6. Conclusions: Distilling the causal effect of the collider is causally equivalent to data replay without storing old data.The distilled quantity is the old feature of a new sample.
  • 6. Conclusions: Incremental Momentum Effect Removal is used during testing to achieve balanced prediction between old and new classes.The causal solutions are described as model-agnostic and improve two strong baselines.
Loading 2103.01737v3…