PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Few-Shot Object Detection without Forgetting

Zhibo Fan, Yuchen Ma, Zeming Li, Jian Sun

arXiv:2105.09491v1cs.CV

TL;DR

Few-shot object detection must learn novel concepts while retaining base-class detection, since realistic test samples may contain either class type. Retentive R-CNN addresses this through Bias-Balanced RPN and Re-detector. Across established benchmarks and data settings, it preserves base-class performance, remains competitive on novel classes, and achieves state-of-the-art overall performance.

  • Problem

    Few-shot detectors often focus on novel categories while fine-tuning can degrade base-class detection, although test samples may contain both classes.

  • Method

    Retentive R-CNN combines Bias-Balanced RPN with a Re-detector to adapt to novel objects while retaining base-class knowledge.

  • Results

    Retentive R-CNN achieves state-of-the-art overall performance across few-shot detection settings, with no base-class degradation and competitive novel-class performance.

  • Takeaways & Limitations

    Few-shot learning without forgetting is achievable in object detection while detecting base and novel classes efficiently.

  • Takeaways & Limitations

    A substantial performance gap remains between few-shot classes and classes trained with abundant data, indicating that few-shot object detection remains difficult.

Abstract

from arXiv · show

Recently few-shot object detection is widely adopted to deal with data-limited situations. While most previous works merely focus on the performance on few-shot categories, we claim that detecting all classes is crucial as test samples may contain any instances in realistic applications, which requires the few-shot detector to learn new concepts without forgetting. Through analysis on transfer learning based methods, some neglected but beneficial properties are utilized to design a simple yet effective few-shot detector, Retentive R-CNN. It consists of Bias-Balanced RPN to debias the pretrained RPN and Re-detector to find few-shot class objects without forgetting previous knowledge. Extensive experiments on few-shot detection benchmarks show that Retentive R-CNN significantly outperforms state-of-the-art methods on overall performance among all settings as it can achieve competitive results on few-shot classes and does not degrade the base class performance at all. Our approach has demonstrated that the long desired never-forgetting learner is available in object detection.

1. Introduction

Few-shot object detection must recognize both novel and base classes efficiently, because realistic test images may contain either and fine-tuning can cause catastrophic forgetting. Retentive R-CNN addresses this with Bias-Balanced RPN and Re-detector, preserving base performance while achieving strong novel-class results.

  • Limited annotations and long-tailed data distributions motivate object detection under a low-data regime.
  • Most few-shot detection methods prioritize novel categories while neglecting base-class forgetting, despite test images potentially containing both class types.
  • Meta-learning approaches can become inefficient when detecting both base and novel classes because their complexity grows with the number of categories.
  • Transfer-learning methods support incremental full-class detection, but TFA still leaves a non-negligible base-class performance gap after fine-tuning.
  • Retentive R-CNN uses Bias-Balanced RPN and Re-detector to adapt proposals to novel objects, preserve base knowledge, and combine base and novel detectors with little extra cost.
  • The method achieves state-of-the-art overall benchmark performance across settings, with leading base-class metrics and competitive novel-class metrics.

2. Related Work

Related work spans few-shot learning, object detection architectures, and two main few-shot object detection paradigms. Meta-learning uses support examples but faces complexity and performance limitations, while transfer learning offers efficient full-class detection yet can retain base-class performance drops.

  • Few-Shot Learning: Few-shot learning commonly uses metric learning, meta-learning, and pretrained backbones to generalize from limited data, while fine-tuning can cause catastrophic forgetting.
  • Object Detection: RCNN generates object proposals from holistic image features, then classifies and refines those proposals using their localized features.
  • Few-Shot Object Detection: Few-shot object detection methods divide into meta-learning and transfer-learning streams.
  • Few-Shot Object Detection: Meta-learning methods condition detections on support examples as exemplar-based visual search, but their complexity can grow with the number of categories.
  • Few-Shot Object Detection: TFA uses two-stage fine-tuning to adapt to novel classes while retaining some base-class performance, but a performance drop remains.

3. Approach

Retentive R-CNN addresses generalized few-shot detection by preserving base-class knowledge while learning novel classes. It combines analysis-driven RPN debiasing with a parallel re-detector and consistency regularization.

  • Problem Statement: Generalized few-shot detection learns a detector for base and novel classes from few novel samples without forgetting base capabilities.
  • Problem Statement: Transfer-learning methods can efficiently detect all classes, but finetuning tends to degrade base-class performance because of forgetting or limited balancing data.
  • Analysis on Transfer Learning based Few-Shot Object Detection: Analysis finds that pretrained detectors can reject salient novel objects, while their RPNs are biased toward seen classes because novel instances were labeled non-object.
  • Retentive R-CNN: Retentive R-CNN uses Bias-Balanced RPN and Re-detector to retain base knowledge while detecting both base and novel classes.
  • Re-detector: The Re-detector runs unchanged base and finetuned joint-class heads in parallel, using consistency loss to make the finetuned head resemble the base head on base-class entries.
  • Bias-Balanced RPN: Bias-Balanced RPN ensembles pretrained and finetuned objectness predictions with a max operation, while shared regression and convolution layers limit additional overhead.

4. Experiments

Retentive R-CNN is evaluated on MS-COCO and Pascal VOC across multiple few-shot settings, with overall and base-class performance as the primary focus. It consistently preserves or improves base-class results while achieving competitive novel-class detection.

  • Experimental Settings: Experiments cover MS-COCO 5-, 10-, and 30-shot settings and Pascal VOC 1-, 2-, 3-, 5-, and 10-shot settings.The evaluations follow established class and data splits for fair comparison.
  • MS-COCO Results: Retentive R-CNN significantly improves MS-COCO AP and bAP without degrading base-class performance, while remaining competitive on nAP.It is state-of-the-art for 10-shot nAP and on-par with state-of-the-art for 5- and 30-shot nAP.
  • Pascal VOC Results: Retentive R-CNN consistently achieves the best overall AP across all Pascal VOC data settings.Its novel-class results are competitive under most settings, while MPSR often sacrifices base-class performance for stronger nAP.
  • Ablation Study: Max ensembling performs best for the Bias-Balanced RPN, while geometric averaging degrades performance and unfreezing box regression adds unnecessary overhead.Novel-class AP tracks RPN average recall closely, whereas base-class AP remains stable with slightly inferior RPN recall.
  • Ablation Study: The current Re-detector design maximizes overall performance, and unfreezing more layers lowers performance.Consistency loss also makes base-class logits more similar to the base model while producing more confident novel-class responses.

5. Conclusion

Retentive R-CNN addresses generalized few-shot detection by combining base and novel detectors while preserving base-class knowledge. Experiments show competitive novel-class performance without base-class degradation and state-of-the-art overall results.

  • Few-shot learning without forgetting is achievable in object detection.
  • Retentive R-CNN combines base and novel detectors using Bias-Balanced RPN and Re-detector.Bias-Balanced RPN alleviates pretrained-RPN bias, while Re-detector finds base and novel objects.
  • Retentive R-CNN preserves base-class performance while remaining competitive on novel classes.
  • Retentive R-CNN achieves state-of-the-art overall performance across the evaluated data settings.The conclusion reports this outcome on established few-shot detection benchmarks.
  • The remaining performance gap between few-shot and general object detection shows that data-limited detection remains difficult.

Supplementary Material

The supplementary material includes a publication metadata line and a figure showing randomly selected pretrained-detector results.

  • The document identifies the paper as arXiv:2105.09491v1, dated 20 May 2021.
  • Figure 1 visualizes ground-truth novel objects with black boxes and base-detector detections with green boxes and predicted labels.

1. Implementation Details

Retentive R-CNN uses two-stage transfer-learning training and inference procedures designed to balance base and novel predictions. The supplementary material also describes comparison settings and typical failure-case visualizations.

  • Retentive R-CNN is trained first on Db and then on a balanced dataset containing Cb ∪ Cn.The first stage follows TFA hyperparameters and learning schedule.
  • During inference, base-detector logits receive zero-valued novel-class entries before softmax scoring.This balances score scales when the base and novel detectors have different class-entry counts.
  • Figure 2 visualizes Retentive R-CNN and TFA w/cos[2] under Pascal VOC split1 2-shot settings alongside typical failure cases.
  • On MS-COCO 10-shot, novel-detector base-class predictions are included in non-maximum suppression and improve base AP by 0.05-0.1 over the pretrained model.
  • Meta R-CNN and FsDetView are compared using the same finetuning samples, while their original procedures used up to three times more base than novel samples.

2. Examples for the Base Detector Rejecting Novel Class Instances

Randomly selected examples show that the pretrained base detector often rejects visually salient novel-class instances rather than producing false positives. Retentive R-CNN uses this property to preserve base-class behavior.

  • Figure 1 marks unrecognized novel instances with black boxes and detected objects with green boxes and predicted categories.
  • The examples are randomly selected from the first 100 MS-COCO 2014 minival images ordered by image id, without cherrypicking.
  • The pretrained base detector strongly ignores novel classes, so false positives seldom occur on unseen objects.
  • Retentive R-CNN utilizes the pretrained detector’s rejection of novel instances to maintain base-class performance.

3. More Detection Results & Failure Case Analysis

Qualitative results show that Retentive R-CNN typically improves base-class detection and reduces confusion on novel instances, while failures remain in few-shot detection, especially for localization and class discrimination.

  • Qualitative Comparison: Retentive R-CNN typically performs better on base classes and reduces object confusion on novel instances than TFA w/cos[2].It detects many objects ignored by TFA w/cos[2] under Pascal VOC split1 2-shot settings.
  • Failure Analysis: Few-shot and abundant-data classes retain a substantial metrics gap, showing that few-shot object detection remains difficult.
  • Failure Analysis: False positives can arise when base and novel predictions overlap on the same unseen object.The base detector sometimes produces false positives on unseen objects, including overlapping boxes from both class types.
  • Failure Analysis: Insufficiently discriminative features cause confusion among few-shot classes and allow base classes to dominate novel classes.
  • Failure Analysis: Pretraining-related backbone deactivation can make novel objects difficult to detect in transfer learning methods.
  • Failure Analysis: Box regression remains a failure source because accurate localization for complex-shaped categories is difficult under low-shot training.

4. Results over Multiple Runs

Across 10 random sample sets on COCO, Retentive R-CNN improves AP, bAP, and nAP under 5-, 10-, and 30-shot settings.

  • Retentive R-CNN obtains better AP, bAP, and nAP across 10 random runs for every 5-, 10-, and 30-shot setting.The runs use exactly the same samples as TFA[2] for direct comparison.
Loading 2105.09491v1…