PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-incremental Learning via Deep Model Consolidation

Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, C. -C. Jay Kuo

arXiv:1903.07864v4cs.CVcs.LG

TL;DR

Incremental learning must add new classes without losing performance on old ones, even when legacy training data are unavailable. DMC consolidates separate old- and new-class models through double distillation on unlabeled auxiliary data, outperforming nonexemplar-based methods across image classification and object detection benchmarks.

  • Problem

    Incremental learning requires adding emerging classes without repeatedly retraining on complete data or causing severe performance degradation on old tasks.

  • Method

    DMC consolidates independently trained old- and new-class models using a double-distillation objective and publicly available unlabeled auxiliary data.

  • Results

    DMC significantly outperforms existing nonexemplar-based methods across image classification and object detection benchmarks, with stable performance across incremental settings.

  • Takeaways & Limitations

    DMC provides a nonexemplar-based approach for consolidating class-incremental models without storing legacy data across multiple vision tasks.

  • Takeaways & Limitations

    DMC leaves the effect of similarity between unlabeled auxiliary data and target data on incremental-learning performance for future study.

Abstract

from arXiv · show

Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) --- an abrupt degradation of performance on the original set of classes when the training objective is adapted to a newly added set of classes. Existing IL approaches tend to produce a model that is biased towards either the old classes or new classes, unless with the help of exemplars of the old data. To address this issue, we propose a class-incremental learning paradigm called Deep Model Consolidation (DMC), which works well even when the original training data is not available. The idea is to first train a separate model only for the new classes, and then combine the two individual models trained on data of two distinct set of classes (old classes and new classes) via a novel double distillation training objective. The two existing models are consolidated by exploiting publicly available unlabeled auxiliary data. This overcomes the potential difficulties due to the unavailability of original training data. Compared to the state-of-the-art techniques, DMC demonstrates significantly better performance in image classification (CIFAR-100 and CUB-200) and object detection (PASCAL VOC 2007) in the single-headed IL setting.

1. Introduction

The paper addresses class-incremental learning when old training data are unavailable, a setting where conventional retraining is impractical and catastrophic forgetting degrades old-task performance. Deep Model Consolidation combines separately trained old and new models through symmetric double distillation using generic unlabeled auxiliary data.

  • Problem setting: DMC targets strict class-incremental learning without access to original old-class data, which may be unavailable because of storage, privacy, proprietary, or recording constraints.The setting reflects dynamic environments where new categories emerge over time and retraining from scratch is prohibitively costly.
  • Deep Model Consolidation: DMC first trains a new-class model, then consolidates it with the old model using external unlabeled auxiliary data and a novel double-distillation objective.The objective promotes symmetric knowledge transfer and reduces bias from information asymmetry or over-regularization; the final model preserves old-class knowledge while recognizing new classes.
  • Motivation: Conventional deep-model training assumes complete data and known classes, while incremental updates can cause catastrophic forgetting and severe degradation on old tasks [15].Existing approaches that store partial old data remain costly and can conflict with copyright or privacy constraints [4] [7] [20] [35] [43].
  • Auxiliary data: The auxiliary data need not share target labels or the target distribution; they only need to be generic, diversified, and broadly related, allowing inexpensive web collection and disposal after learning.This provides a cheap substitute for storing old data without requiring additional dataset construction and maintenance.
  • Generalization: Because the two teacher models play symmetric roles, DMC can also merge arbitrary pretrained models downloaded from the Internet into one deployable model without original training data.This reduces deployment to a single model rather than two models.

2. Related work

Prior work addresses catastrophic forgetting through regularization, dynamic architectures, rehearsal, and pseudo-rehearsal, but class-incremental learning remains constrained by task-label requirements and old–new information asymmetry. Related research also motivates using unlabeled auxiliary data and extending incremental learning to modern object detectors.

  • Catastrophic forgetting occurs when retraining with new data overwrites memory of old data, motivating methods to alleviate this effect.
  • Learning without Forgetting uses knowledge distillation on new-class images to keep old-class outputs close to those of the original network [29].
  • Because new images may deviate from the old-data distribution, LwF retains old–new information asymmetry; using two teachers for one student is proposed to make information flow symmetric.
  • Dynamic network methods: Dynamic network methods allocate task-specific pathways but require task labels at test time, making them unsuitable for class-incremental learning without available task labels [36] [37] [56].
  • Incremental learning of object detectors: For object detection, Shmelkov et al. adapted LwF but limited it to externally generated proposals, whereas the presented method also applies to single-shot detectors such as RetinaNet [51] [31].
  • Exploiting external data: Unlabeled auxiliary data can support learning without requiring target labels or matching class labels and generative distributions, making it easier to obtain than typical semi-supervised or transfer-learning data.

3. Method

DMC addresses class-incremental learning without legacy data by training a new-class model and consolidating it with the old model through double distillation on unlabeled auxiliary data. The approach extends from image classification to object detection by jointly consolidating classification and bounding-box regression outputs.

  • Class-incremental setting: DMC learns new classes from Dnew, then consolidates the resulting new model with the previous s-class classifier into a t-class model without access to old training data.The incremental setting provides only newly available class data and the previous model, while requiring learning without catastrophic forgetting.
  • Image classification: The consolidation objective uses double distillation, treating the old and new models’ network responses as supervisory signals for jointly training one compact model.The consolidated outputs approximate the old-model scores for old classes and new-model scores for new classes.
  • Image classification: Unlabeled auxiliary data U approximates the target distribution, enabling consolidation despite unavailable legacy data and allowing on-the-fly use without persistent storage.The auxiliary data can be crawled, fed in mini-batches, and discarded after consolidation.
  • Image classification: Mean-subtracting the concatenated specialist logits calibrates old–new class bias by unifying their scales while retaining relative class magnitudes for symmetric information flow.The consolidated model should not be initialized from either specialist, and Dnew should not be reused during consolidation, to avoid intrinsic class bias.
  • Object detection: For object detection, DMC jointly consolidates classification and bounding-box regression subnetworks using outputs from old- and new-class specialist detectors.The method selects anchor boxes with the highest objectness scores and applies double distillation to their classification logits.

4. Experiments

Experiments evaluate DMC in the challenging single-head setting across incremental image classification and object detection benchmarks. DMC consistently outperforms exemplar-free baselines, limits forgetting, and achieves state-of-the-art detection accuracy with lower inference cost.

  • iCIFAR-100: DMC significantly outperforms all compared exemplar-free methods across every iCIFAR-100 group size, g = 5, 10, 20, and 50.Results use average top-1 accuracy over five class orderings and compare against EWC++ [6, 23], LwF [29], SI [57], MAS [1], and RWalk [6].
  • iCIFAR-100: Across incremental sessions, DMC maintains stable performance across group sizes, whereas regularization-based methods increasingly struggle to balance forgetting and intransigence.On the first task, DMC also has a considerably gentler forgetting curve than previous methods [1] [6] [23] [29] [57].
  • iCIFAR-100: Confusion matrices show Finetuning forgetting old classes, Fixed Representation favoring the first group, and LwF [29] retaining asymmetric errors toward recently learned classes.These patterns illustrate the differing old-class and new-class biases of the compared methods.
  • Ablation studies: DMC is generally insensitive to the distillation distance metric, while L2 and KD loss [18] with T = 2 perform slightly better; L2 is therefore used for simplicity and stability.The comparison varies the distance metric in Ldd on CIFAR-100 with g = 20.
  • Incremental object detection: DMC achieves state-of-the-art mAP on all learned PASCAL VOC 2007 classes while using half the model complexity and inference time of Inference twice.The evaluation covers 10 + 10 and 19 + 1 class increments, using COCO images from the 60 non-PASCAL categories as auxiliary data.

5. Conclusion

The paper introduces DMC, a class-incremental learning paradigm that consolidates independently trained models without storing legacy data. It achieves significant gains over existing nonexemplar-based methods, while suggesting several directions for extending and analyzing the approach.

  • DMC outperforms existing nonexemplar-based incremental-learning methods by a significant margin on large-scale image-classification and object-detection benchmarks.
  • DMC uses a novel double distillation objective and readily available unlabeled auxiliary data to consolidate two independently trained models without storing legacy data.
  • Future work includes analyzing auxiliary-data similarity, studying exemplar selection and rehearsal, and consolidating multiple models simultaneously.

Appendix Overview

This appendix provides detailed experimental results and analyses of Deep Model Consolidation (DMC) for class-incremental learning.

  • The supplemental document reports additional experimental results for Deep Model Consolidation (DMC).
  • It also presents detailed analyses of the proposed method.
  • The appendix focuses on DMC in the context of class-incremental learning.

A. Detailed experimental results of DMC for object detection

The experiments evaluate DMC for class-incremental object detection by adding each VOC category as a single new class to a 19-class RetinaNet detector.

  • Experimental setup: Using RetinaNet [31], the study incrementally learns 19+1 classes and reports one-class addition results for every VOC category in Table 10.Old Model is the 19-class detector trained on the original classes, while New Model is the 1-class detector trained on the added class.

B. Effect of the amount of auxiliary data for object detection

DMC remains effective for incremental object detection even when consolidation uses substantially less auxiliary data. Experiments on VOC 2007 evaluate random fractions of Microsoft COCO data in the 10+10 classes setting, including only 12.3k images.

  • B. Effect of the amount of auxiliary data for object detection: Using just 1/8 of the full auxiliary data, or 12.3k Microsoft COCO images, DMC can still perform effectively on VOC 2007 incremental object detection.The experiments use VOC 2007 with 10 old and 10 new classes.
  • B. Effect of the amount of auxiliary data for object detection: The study randomly samples 1/2, 1/4, and 1/8 of the full Microsoft COCO auxiliary dataset for consolidation and reports VOC 2007 test mAP.Table 6 defines classes 1–10 as old and classes 11–20 as new.

C. Implementation and training details

DMC uses standardized optimization and augmentation settings for image classification, with separate training and consolidation schedules. Object detection uses resized inputs, Adam optimization, GPU parallelism, augmentation, and NMS postprocessing.

  • Image classification: Image classification uses randomly initialized 32-layer ResNet models, with specialist training for 200 epochs and consolidation training for 50 epochs.The implementation uses PyTorch and follows iCaRL [43] with a 32-layer ResNet [17].
  • Image classification: Classification training uses SGD with momentum, an initial learning rate of 0.1 reduced at 7/10 and 9/10 of training, batch size 128, and weight decay 1 × 10−4.Momentum is 0.9, with padding-based random cropping or horizontal flipping for augmentation.
  • Object detection: Object detection resizes images to a 640-pixel shorter side, trains for 100 epochs with Adam at learning rate 1 × 10−3, and uses batch size 12 on two NVIDIA Tesla M40 GPUs.Aspect ratios are preserved and random horizontal flipping is used for augmentation.
  • Object detection: Detection predictions are postprocessed with standard non-maximum suppression using a 0.5 threshold to remove duplicate predictions.
  • Baseline hyperparameters: Baseline regularization strengths for EWC++ [6], SI [57], MAS [1], and RWalk [6] on iCIFAR-100 are selected using a held-out validation set and summarized in Table 7.Table 7 reports the λ̂ values used when incrementally learning g classes at a time.

D. Preliminary experiments of adding exemplars

DMC+ extends DMC to exemplar-based incremental learning by fine-tuning with a fixed memory budget, reducing training bias and achieving performance on par with iCaRL [43] on iCIFAR-100. These preliminary results suggest promise while leaving exemplar selection and rehearsal strategies for further study.

  • D. Preliminary experiments of adding exemplars: With a total memory budget K = 2000, DMC+ randomly samples exemplars from each class, fine-tunes the consolidated model for 15 epochs at a learning rate of 1 × 10−3, and then reduces the per-class exemplar set.The exemplar set is augmented with randomly sampled new-class images at each incremental session and maintained within the fixed budget.
  • D. Preliminary experiments of adding exemplars: Fine-tuning with exemplars further reduces DMC’s intrinsic training bias.This effect is reported in the confusion-matrix comparison between DMC+ and iCaRL [43].
  • D. Preliminary experiments of adding exemplars: The preliminary findings motivate further study of exemplar selection schemes and rehearsal strategies for improving DMC+.The paper characterizes these results as promising rather than definitive.
  • D. Preliminary experiments of adding exemplars: DMC+ is on a par with iCaRL [43] on iCIFAR-100, despite selecting exemplars by naive random sampling rather than herding [43].Figure 7 compares confusion matrices for exemplar-based methods when adding 10 classes per incremental step; Table 8 reports average incremental accuracies for groups of 5, 10, 20, and 50 classes.

E. Preliminary experiments of consolidating models with common classes

DMC extends to consolidating models trained on partially overlapping class sets by averaging normalized logits for common classes and using the other classes’ logits separately. In a CIFAR-100 experiment with two 55-class classifiers sharing 10 classes, it maintained the weaker model’s common-class accuracy without catastrophic forgetting or intransigence.

  • Method extension: For partially overlapping classes, DMC averages the normalized logits of the two models for common classes and uses each model’s logits for its other classes.The two models’ logits are first normalized according to Eq. 4 in the main paper before constructing the double-distillation regression target.
  • Method extension: The extension applies when the two consolidated models are trained on class sets that partially overlap rather than being entirely distinct.This generalizes the original DMC assumption of disjoint class sets.
  • Preliminary experiment: On CIFAR-100, DMC maintained at least the weaker model’s accuracy on 10 common classes while avoiding catastrophic forgetting and intransigence on the remaining classes.The experiment consolidated classifiers trained on Class 1-55 and Class 46-100, with Class 46-55 shared; results were reported in Table 9.

F. Enlarged plots

This section provides enlarged accuracy-curve plots for iCIFAR-100 to improve visibility, covering incremental groups of 5, 10, 20, and 50 classes.

  • F. Enlarged plots: Enlarged accuracy curves improve visibility for iCIFAR-100 incremental-learning results.
  • F. Enlarged plots: Table 10 reports VOC 2007 test per-class average precision for incrementally learning 19 + 1 classes.
  • F. Enlarged plots: Figure 8 covers increments of g = 5, 10, 20, and 50 classes at a time.
Loading 1903.07864v4…