Source-linked AI summary
Incremental Learning of Object Detectors without Catastrophic Forgetting
Konstantin Shmelkov, Cordelia Schmid, Karteek Alahari
TL;DR
Incremental object detection must add new classes without the original training data while avoiding catastrophic forgetting of old classes. The paper uses a dual-network distillation loss alongside new-class supervision, and reports successful repeated class addition on PASCAL VOC and COCO without forgetting the original classes.
Problem
CNN detectors suffer catastrophic forgetting when adapted to new classes without the original training data, degrading performance on previously learned classes.
Method
A frozen copy of the original detector supplies old-class responses and proposals while an expanded detector learns new classes using cross-entropy plus distillation loss.
Results
Experiments on PASCAL VOC and COCO show that new classes can be added incrementally without forgetting the original classes and without access to original training data.
Takeaways & Limitations
The framework supports sequential addition of further class sets by freezing each newly trained detector and reusing it for distillation.
Abstract
from arXiv · showhide
Despite their success for object detection, convolutional neural networks are ill-equipped for incremental learning, i.e., adapting the original model trained on a set of classes to additionally detect objects of new classes, in the absence of the initial training data. They suffer from "catastrophic forgetting" - an abrupt degradation of performance on the original set of classes, when the training objective is adapted to the new classes. We present a method to address this issue, and learn object detectors incrementally, when neither the original training data nor annotations for the original classes in the new training set are available. The core of our proposed solution is a loss function to balance the interplay between predictions on the new classes and a new distillation loss which minimizes the discrepancy between responses for old classes from the original and the updated networks. This incremental learning can be performed multiple times, for a new set of classes in each step, with a moderate drop in performance compared to the baseline network trained on the ensemble of data. We present object detection results on the PASCAL VOC 2007 and COCO datasets, along with a detailed empirical analysis of the approach.
1. Introduction
Incremental adaptation of object detectors to new classes is needed as datasets evolve, but fine-tuning can cause catastrophic forgetting of previously learned classes. The paper proposes balancing new-class learning with distillation to preserve old-class performance without original training data.
- Motivation: CNN-based detectors require incremental learning methods as datasets evolve with new classes and samples, while full retraining is costly.Pretraining and adaptation are presented as alternatives to training entire networks from scratch.
- Motivation: Fine-tuning can improve performance on new classes while dramatically degrading performance on old classes when all classes are not trained jointly.The output layer may be replaced or expanded, with newly initialized weights and network-wide tuning for the new objective.
- Motivation: Adding horse images to a detector trained on person and other classes enables horse detection but causes the adapted network to miss the previously localized rider.The example illustrates catastrophic forgetting even though the person class is not updated.
- Approach: The proposed loss balances cross-entropy for new-class predictions with distillation that minimizes discrepancies between old-class responses from the original and updated networks.The method uses only training samples for the new classes.
- Evaluation: Experiments on PASCAL VOC and COCO report incremental addition of classes without forgetting the original classes or accessing original training data.The paper also evaluates method variants and the influence of distillation and loss design.
2. Related work
Prior incremental-learning methods either retain collected data, use fixed representations, or address easier classification settings rather than object detection without previously seen samples. This work is positioned as end-to-end incremental object detection under that stricter setting.
- Incremental learning: Earlier approaches update training sets from Internet data but either use fixed representations or retain all collected data for retraining.Other methods learn classifiers without access to the full ensemble of data.
- Transfer learning: Transfer learning and domain adaptation reuse knowledge across tasks or related data distributions, with fine-tuning as a common computer-vision example.Classification CNNs are often adapted for detection and segmentation.
- Distillation: Distillation transfers knowledge by encouraging old and new networks to produce similar responses, with applications including domain adaptation and model compression.Earlier distillation work transferred knowledge from a large network to a smaller one for efficient deployment.
- Distillation: The paper differs from prior classification distillation by handling same-dataset class addition, object detection, and co-occurring old and new classes.The authors characterize this as more difficult than cross-dataset image-classification evaluation.
- Alternative approaches: Unlike approaches that use a subset of original data or add parameters, this method jointly learns representation and classifier end-to-end without original samples.The cited alternatives increase learned parameters or were evaluated only on image classification.
- Incremental learning: The paper identifies a gap in incremental object-detection classifiers trained without previously seen training samples.The stated gap distinguishes this setting from prior classifier-learning work.
3. Incremental learning of new classes
The method incrementally extends an object detector using only new-class data while preserving old-class responses through a frozen original network and distillation-based supervision.
- Dual-network learning: A frozen copy of the original detector selects distillation proposals and supplies responses used to constrain the adapted network.The adapted network expands the original outputs for new classes while retaining the original classes.
- Distillation targets: Distillation uses logits and bounding-box regression outputs from sampled RoIs, preserving both classification responses and localization refinements for old classes.The method samples 64 of 128 RoIs with the smallest background scores and applies an L2 loss to old-class responses.
- Loss function: The incremental objective combines the standard Fast R-CNN loss for annotated new classes with a weighted distillation loss for old-class behavior.The weighting parameter λ balances the two losses, and λ is set to 1 in the experiments.
- Sequential extension: After training, the adapted network can be frozen and reused for distillation, allowing additional classes to be added sequentially.The same extension can be repeated because the adapted network remains structurally compatible with the expanded detector.
- Sampling strategy: Selecting non-background proposals is advantageous because uniformly sampled proposals are mostly background and provide weak signals for conserving old categories.The authors report noticeably worse old-class detection performance with unbiased sampling from all 2000 proposals.
4. Experiments
Experiments on VOC and COCO show that distillation preserves old-class detection during incremental learning while retaining substantial performance on newly added classes. Results also identify trade-offs involving class count, data distribution, distillation strength, sequential updates, and bounding-box regression distillation.
- Addition of one class: VOC experiments show that freezing layers preserves old-class performance but learns new classes poorly, while fine-tuning without distillation causes severe forgetting.The authors conclude that convolutional layers must be fine-tuned for new classes, but this requires distillation to protect old classes.
- Addition of one class: 25.0% mAP without distillation rises to 68.3% with distillation on old VOC classes, compared with 69.6% for jointly trained Fast R-CNN.The new tvmonitor class remains below its 73.9% joint-training baseline, likely because only a few hundred training images provide limited negative diversity.
- Addition of one class: VOC performance across single-class additions ranges from 66.1% to 68.3% mAP, with mean 67.38% and standard deviation 0.6%.The reported behavior is similar across the VOC categories used as new classes.
- Addition of multiple classes: Adding all ten VOC classes yields 63.1% mAP with distillation versus 69.8% for joint training, while omitting distillation leaves only 12.8% mAP on old classes.Performance on new classes is slightly lower than with joint training, and omitting bounding-box distillation is inferior to the full method.
- Distillation analysis: Distillation works best when background-image distributions match across training stages, but remains effective with a different second-stage dataset.Setting λ to 1 balances learning new classes and preventing forgetting; weak distillation favors new classes, while strong distillation destabilizes training.
5. Conclusion
The paper presents an incremental object-detection approach that preserves old-class performance while learning new classes without the old training data. Experiments show it remains effective even when new classes are added one at a time, while future work targets learned proposals.
- The proposed loss optimizes new-class performance while preserving performance on old classes without access to their training data.
- The approach performs well even in the extreme case of adding new classes one by one.
- Future work will adapt the method to learned proposals such as RPN by adding an RPN-based knowledge-distillation term.