PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN

Can Peng, Kun Zhao, Brian C. Lovell

arXiv:2003.03901v2cs.CV

TL;DR

Incremental object detection must learn new categories without forgetting old ones when old data or annotations are unavailable. The paper analyzes RPN-based detection and introduces Faster ILOD with multi-network adaptive distillation. On PASCAL VOC and COCO, it reports superior results and a large-margin improvement over the state-of-the-art incremental detector in most cases.

  • Problem

    Incremental object detectors must retain old-category performance while learning new categories from incomplete data, where old exemplars are unavailable and old objects may be unannotated.

  • Method

    Faster ILOD uses an end-to-end Faster RCNN detector with multi-network adaptive distillation to preserve old-category knowledge during new-task training.

  • Results

    The method shows superior results on PASCAL VOC and COCO and outperforms the state-of-the-art incremental detector by a large margin in most cases.

  • Takeaways & Limitations

    RPN-based incremental detection can tolerate missing old-class annotations to some extent, while adaptive distillation further improves the detector.

  • Takeaways & Limitations

    Incremental-learning difficulty and accuracy improvements vary with training-data amount, old-category co-occurrence, and the percentage of missing annotations.

Abstract

from arXiv · show

The human vision and perception system is inherently incremental where new knowledge is continually learned over time whilst existing knowledge is retained. On the other hand, deep learning networks are ill-equipped for incremental learning. When a well-trained network is adapted to new categories, its performance on the old categories will dramatically degrade. To address this problem, incremental learning methods have been explored which preserve the old knowledge of deep learning models. However, the state-of-the-art incremental object detector employs an external fixed region proposal method that increases overall computation time and reduces accuracy comparing to Region Proposal Network (RPN) based object detectors such as Faster RCNN. The purpose of this paper is to design an efficient end-to-end incremental object detector using knowledge distillation. We first evaluate and analyze the performance of the RPN-based detector with classic distillation on incremental detection tasks. Then, we introduce multi-network adaptive distillation that properly retains knowledge from the old categories when fine-tuning the model for new task. Experiments on the benchmark datasets, PASCAL VOC and COCO, demonstrate that the proposed incremental detector based on Faster RCNN is more accurate as well as being 13 times faster than the baseline detector.

1 Introduction

Incremental object detection must learn new categories without losing old ones, despite inaccessible or incomplete old annotations. The paper analyzes RPN-based detection and proposes Faster ILOD with multi-network adaptive distillation.

  • Fine-tuning on new-class data causes catastrophic forgetting, while retraining on all labeled data is computationally expensive and often impractical.
  • Incremental detection is difficult because new-task images may contain old objects whose annotations are missing.
  • Faster ILOD uses multi-network adaptive distillation to address incremental learning for RPN-based object detectors.
  • RPN anchor selection gives incremental detection some tolerance to missing annotations for old classes.
  • The study evaluates Faster RCNN-based incremental detection on PASCAL VOC and COCO under several incremental settings.
  • The framework is designed to generalize to object detectors that use an RPN.

2 Problem Formulation

Incremental object detection retrains a detector using only data for new classes while retaining detection of previously learned classes. The setting assumes old objects may appear without annotations and no old-class exemplars are available.

  • At each incremental step, only training data for the new classes is accessible, while the detector must retain old-class detection.
  • Old classes are the categories learned in previous steps, and new classes are introduced during the current step.
  • The original detector is treated as the teacher model, and the retrained detector as the student model.
  • New-task images may contain old-class objects whose annotations are not provided.
  • The retrained detector must detect both current new classes and all classes learned previously.

3 Related Work

Prior incremental-learning work uses knowledge distillation, but incremental object detection remains constrained by proposal generation and unrealistic data settings. This paper targets end-to-end RPN-based detection under realistic missing-annotation conditions.

  • Knowledge distillation transfers old-model behavior to a new model and is widely used for incremental learning.
  • ILOD adapts LwF to object detection without old data but relies on Fast RCNN with an external fixed proposal generator.
  • The external proposal design makes ILOD non-end-to-end, motivating evaluation of a more efficient RPN-based detector.
  • Some prior end-to-end methods avoid missing annotations by excluding images containing objects from multiple class groups, unlike the realistic setting targeted here.
  • The paper targets high-performance incremental detection when new-task images contain old objects without their annotations, evaluating one-step and multi-step settings on VOC and COCO.

4 Evaluation of Robustness of RPN to Incremental Object Detection

RPN-based Faster RCNN remains relatively robust to missing annotations during incremental detection, although prior work had treated this issue as a major obstacle. Applying ILOD distillation to Faster RCNN consistently outperforms the original ILOD baseline across the reported VOC and COCO settings.

  • Comparison with ILOD: ILOD applied to Faster RCNN outperforms original ILOD by 0.45%, 3.63%, and 0.11% when adding 1, 5, and 10 VOC classes, respectively.These are one-step incremental experiments.
  • Comparison with ILOD: ILOD applied to Faster RCNN outperforms original ILOD by 4.1%, 3.3%, and 3.0% for adding 5, 10, and 40 COCO classes at 0.5 IoU.These are one-step incremental results.
  • Comparison with ILOD: ILOD applied to Faster RCNN improves over original ILOD by average margins of 1.32%, 4.11%, and 0.79% when adding 1, 2, and 5 VOC classes per step.These results come from multi-step incremental experiments.
  • Comparison with ILOD: ILOD applied to Faster RCNN improves over original ILOD by an average 3.2% at 0.5 IoU when adding one COCO class per step.This is the reported multi-step COCO comparison.
  • RPN robustness: RPN is relatively robust to missing annotations because random anchor sampling makes selected negatives unlikely to overlap well-localized old objects.The main remaining effect is reduced positive samples for old classes.
  • RPN robustness: Distillation applied only at final outputs back-propagates through the network, encouraging the RPN and feature extractor to recognize old classes.The authors report this helps explain the limited impact of missing annotations in the tested one- and several-step settings.

5 Faster ILOD for Robust Incremental Object Detection

Faster ILOD extends Faster RCNN with multi-network adaptive distillation across feature maps, RPN outputs, and final predictions. The design uses teacher outputs as lower bounds so the student can retain old-category knowledge while learning new categories and addressing missing annotations.

  • Motivation: The method targets an accuracy gap remaining between ILOD applied to RPN-based detectors and full-data training, including VOC multi-step results below 60% versus 69.50%.On COCO, the cited comparison is 24% at 0.5 IoU versus 42.71% for full-data training.
  • Framework: Faster ILOD combines adaptive feature-map and RPN distillation with conventional final-output distillation to preserve old knowledge and alleviate missing annotations.The framework uses a frozen teacher and an adapted student Faster RCNN detector.
  • Framework: Faster RCNN provides the end-to-end backbone through a CNN feature extractor, RPN, and class-level classification and regression network.The proposed method accounts for all three detector components.
  • Adaptive distillation: Adaptive distillation uses teacher outputs as lower bounds, allowing student activations or proposals to exceed teacher values when they may support new classes.This principle is applied to feature extraction and RPN proposal generation.
  • RCN distillation: The RCN distillation stage samples 64 of 128 teacher-selected RoIs with the smallest background scores and distills teacher outputs for old classes.Student outputs for new classes are excluded from this RCN distillation.
  • Loss design: The total loss weights standard Faster R-CNN, feature-distillation, RPN-distillation, and RCN-distillation terms, with λ1, λ2, and λ3 empirically set to 1.The loss is a weighted summation of the four components.

6 Experiments

Experiments on VOC and COCO evaluate Faster ILOD against ILOD variants across one-step and multi-step incremental detection settings. Faster ILOD generally achieves the strongest accuracy and substantially faster inference than the external-proposal baseline.

  • Experiments on VOC Dataset: 3.44% and 2.12% average gains are achieved over ILOD and ILOD applied to Faster RCNN when adding one VOC class across five steps.Faster ILOD outperforms both alternatives at every incremental step under this protocol.
  • Experiments on VOC Dataset: 5.78% and 1.67% average gains are achieved over ILOD and ILOD applied to Faster RCNN when adding two VOC classes across five steps.Faster ILOD performs best at all five incremental steps in this setting.
  • Faster ILOD outperforms ILOD and ILOD applied to Faster RCNN across VOC and COCO incremental detection experiments.The comparisons cover one-step and multi-step settings, with results reported using mAP-based detection accuracy.
  • Discussions: Incremental difficulty varies with training-data amount, old-category co-occurrence, and missing-annotation percentage.The authors conjecture that these factors explain differing accuracy improvements across category-learning scenarios.
  • 1396.66 ms versus 109.52 ms per VOC image makes ILOD about 13 times slower than Faster ILOD.The speed difference is attributed to ILOD's reliance on an external proposal generator.

7 Conclusion

The paper identifies RPN robustness to missing old-class annotations and proposes Faster ILOD, an end-to-end detector using adaptive multi-network distillation. It reports superior results on PASCAL VOC and COCO, with large gains over the state-of-the-art incremental detector in most cases.

  • Faster ILOD combines an end-to-end RPN-based framework with adaptive distillation across multiple networks.The method is designed to preserve old-class detection capabilities while limiting effects on learning new classes.
  • The RPN is relatively robust to missing annotations for old classes in incremental object detection.
  • Faster ILOD achieves superior results on PASCAL VOC and COCO and outperforms the state-of-the-art incremental detector by a large margin in most cases.
Loading 2003.03901v2…