PaperlayerView exact revision on arXiv

Source-linked AI summary

DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection

Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, Chi Zhang

arXiv:2108.09017v1cs.CV

TL;DR

Few-shot object detection must generalize to novel classes from extremely few examples, but Faster R-CNN lacks tailored treatment of its multi-stage and multi-task contradictions. DeFRCN adds Gradient Decoupled Layers and an offline Prototypical Calibration Block to address them, and reports new state-of-the-art performance across benchmarks.

  • Problem

    Few-shot object detection requires rapid adaptation to novel classes with limited data, while Faster R-CNN lacks tailored consideration for its multi-stage and classification-localization conflicts.

  • Method

    DeFRCN extends Faster R-CNN with Gradient Decoupled Layers for multi-stage decoupling and an offline Prototypical Calibration Block for multi-task score calibration.

  • Results

    DeFRCN achieves new state-of-the-art performance on various benchmarks and is reported as superior to existing approaches.

  • Takeaways & Limitations

    The framework provides a simple fine-tuning-based approach that addresses Faster R-CNN contradictions in few-shot object detection.

  • Takeaways & Limitations

    The evaluation follows a two-stage fine-tuning setting with nonoverlapping base and novel classes.

Abstract

from arXiv · show

Few-shot object detection, which aims at detecting novel objects rapidly from extremely few annotated examples of previously unseen classes, has attracted significant research interest in the community. Most existing approaches employ the Faster R-CNN as basic detection framework, yet, due to the lack of tailored considerations for data-scarce scenario, their performance is often not satisfactory. In this paper, we look closely into the conventional Faster R-CNN and analyze its contradictions from two orthogonal perspectives, namely multi-stage (RPN vs. RCNN) and multi-task (classification vs. localization). To resolve these issues, we propose a simple yet effective architecture, named Decoupled Faster R-CNN (DeFRCN). To be concrete, we extend Faster R-CNN by introducing Gradient Decoupled Layer for multi-stage decoupling and Prototypical Calibration Block for multi-task decoupling. The former is a novel deep layer with redefining the feature-forward operation and gradient-backward operation for decoupling its subsequent layer and preceding layer, and the latter is an offline prototype-based classification model with taking the proposals from detector as input and boosting the original classification scores with additional pairwise scores for calibration. Extensive experiments on multiple benchmarks show our framework is remarkably superior to other existing approaches and establishes a new state-of-the-art in few-shot literature.

1. Introduction

Few-shot object detection is constrained by scarce annotations, while standard Faster R-CNN lacks tailored handling of its multi-stage and multi-task conflicts. DeFRCN addresses these conflicts with gradient decoupling and prototype-based score calibration, achieving superior benchmark performance.

  • Motivation: Few-shot object detection seeks rapid generalization to novel classes from extremely few annotated examples.Existing vision systems depend heavily on large annotated datasets, limiting applications in data-scarce settings.
  • Problem: Standard Faster R-CNN jointly optimizes class-agnostic RPN and class-relevant RCNN through a shared backbone, creating a multi-stage conflict.Its classification and localization tasks also require mismatched translation-invariant and translation-covariant features.
  • Method: DeFRCN extends Faster R-CNN with two Gradient Decoupled Layers that adjust decoupling among the backbone, RPN, and RCNN.GDLs use learnable affine feature transformations and scaled backward gradients between the shared backbone and detector modules.
  • Method: An offline Prototypical Calibration Block operates parallel to the box classifier to calibrate classification scores and decouple classification from localization.The framework is designed for straightforward end-to-end fine-tuning in the data-scarce setting.
  • Results: DeFRCN is reported as remarkably superior to state-of-the-art approaches across multiple benchmarks.The introduction identifies GDL and PCB as modules for decoupling Faster R-CNN components and boosting classification performance.

2. Related Work

Object detection commonly uses either two-stage proposal-based or one-stage proposal-free architectures, while few-shot detection includes meta-learning and fine-tuning approaches. DeFRCN builds on the fine-tuning family while modifying Faster R-CNN for few-shot detection.

  • General Object Detection: Two-stage detectors generate potential objects with an RPN before performing category classification and box localization.One-stage detectors instead directly produce detections without a proposal stage.
  • Few-Shot Learning: Few-shot learning research includes meta-learning and fine-tuning paradigms for extracting novel concepts from few examples.Few-shot detection is more challenging than few-shot classification and conventional object detection.
  • Meta-Learning Methods: Meta-based few-shot detectors use mechanisms including channel-wise attention, RoI attention, feature aggregation, or support information to improve novel detection.Examples include FSRW, Meta R-CNN, FSDView, and FSOD.
  • Fine-Tuning Methods: Fine-tuning-based methods include RepMet and TFA, while DeFRCN follows fine-tuning and jointly modifies the Faster R-CNN framework.TFA fine-tunes only the last detector layer on rare classes, whereas RepMet uses a modified prototypical-network classification head.

3. Methods

The method adapts Faster R-CNN to few-shot object detection by decoupling its stages and tasks. GDL controls gradient interactions among backbone, RPN, and RCNN, while PCB calibrates classification scores offline using prototypes.

  • Revisiting Faster R-CNN: Few-shot Faster R-CNN jointly optimizes modules with mismatched goals: RPN proposes class-agnostic regions, while RCNN classifies and localizes objects.The shared backbone also serves classification and localization, which require translation-invariant and translation-covariant features, respectively.
  • Decoupled Faster R-CNN: DeFRCN decouples the backbone, RPN, and RCNN with two Gradient Decoupled Layers and decouples classification from localization with an offline Prototypical Calibration Block.The GDLs adjust inter-module decoupling, while PCB improves RCNN classification during inference.
  • Gradient Decoupled Layer: GDL applies a learnable affine transformation during the forward pass and scales backward gradients by a decoupling coefficient λ.Its affine layer uses learnable channel-wise weights and bias; λ controls the gradient contribution passed to preceding modules.
  • Gradient Decoupled Layer: λ1 = 0 or λ2 = 0 stops one branch’s gradient at the shared backbone, whereas λ1, λ2 ∈ (0, 1] scales its contribution.Setting λ1 = λ2 = λ̃ slows shared-backbone updates relative to RPN and RCNN updates.
  • Prototypical Calibration Block: PCB receives proposals from the fine-tuned detector, extracts proposal features with RoIAlign, and computes cosine similarity against class prototypes.The prototype bank is formed from support examples grouped by class.
  • Prototypical Calibration Block: PCB shares no parameters with the detector and operates offline, allowing plug-and-play integration with other detector architectures.This separation preserves classification-oriented features while decoupling classification and regression within RCNN.

4. Experiments

Experiments across VOC, COCO, cross-domain detection, ablations, and conventional detection evaluate DeFRCN and its decoupling design. Results report strong benchmark performance, module effects, and the influence of gradient-decoupling settings.

  • Experimental settings: DeFRCN is evaluated on multiple benchmarks using FSOD and G-FSOD protocols with established VOC and COCO data splits.VOC uses three 15-base/5-novel splits with 1–10 shots; COCO uses 60 base and 20 novel classes with 1–30 shots.
  • VOC results: Up to 21.4%, DeFRCN exceeds recent state-of-the-art methods on VOC under both FSOD and G-FSOD settings.Table 1 reports AP50 across three VOC splits, with results averaged over multiple runs.
  • VOC results: 40.2% →66.5% versus 53.6% →60.8%, G-FSOD performance grows faster than FSOD as shot count increases.The paper attributes this pattern to more negative samples under G-FSOD.
  • Cross-domain results: 55.9%, DeFRCN achieves the best reported 10-shot cross-domain FSOD performance from COCO base classes to VOC novel classes.This is reported as a 13.6% improvement over MPSR.
  • Ablation study: 7.9%/12.2%, plain FRCN reaches these 10/30-shot COCO results, while the ablation progressively examines GDL and other DeFRCN modules.The plain model is described as severely over-fitting because of limited training data.
  • Decoupling analysis: λrpn values close to 0 and an appropriate λrcnn produce higher performance across base training and novel fine-tuning.The four-corner analysis reports RPN effects of 39.01 versus 38.39 and RCNN effects of 31.56 versus 38.39 for backbone optimization.

5. Conclusion

DeFRCN is a simple fine-tuning framework that addresses contradictions in conventional Faster R-CNN for few-shot object detection using GDL and PCB. It achieves new state-of-the-art results across various benchmarks.

  • DeFRCN alleviates contradictions in conventional Faster R-CNN for data-scarce few-shot object detection.It introduces the novel Gradient Decoupled Layer and Prototypical Calibration Block.
  • DeFRCN achieves new state-of-the-art performance on various benchmarks.The paper describes the method as effective and versatile despite its simplicity.

Supplementary Material

The supplementary material adds experimental analyses and visualizations that provide further insight into DeFRCN. It covers generalized few-shot detection, PCB analysis, GDL extensions, and qualitative results.

  • The supplementary material provides additional experimental analyses and visualization details.These materials are intended to develop further insights into the proposed approach.
  • Supplementary topics include G-FSOD results, PCB analysis, GDL extensions, and qualitative visualizations.

A.1. Implementation Details

The G-FSOD evaluation retains base-class performance after fine-tuning and uses a balanced set containing base and novel classes. DeFRCN is fine-tuned under this protocol.

  • G-FSOD requires evaluating base-class performance after the fine-tuning stage.This distinguishes it from the FSOD protocol.
  • DeFRCN is fine-tuned on a small balanced training set containing both base and novel classes.

A.2. Experimental Results of G-FSOD Setting

In G-FSOD experiments, DeFRCN improves novel and overall detection performance over TFA while maintaining comparable or sometimes better base-class performance. The pattern holds across VOC and COCO, with larger gains as support shots increase on COCO.

  • PASCAL VOC: Novel AP is usually over 7% points higher than TFA across three VOC data splits.
  • PASCAL VOC: DeFRCN outperforms TFA on VOC split 2 by +1.9% ∼ +3.7% AP but is slightly worse on splits 1 and 3.
  • COCO: On COCO, DeFRCN has comparable base-class performance but far superior novel and overall results compared with TFA.
  • COCO: On COCO, performance improvements increase as the number of support shots increases.

B.1. Boost Other Approaches with PCB

PCB consistently improves prior few-shot detection approaches on novel classes across methods and shot settings on COCO.

  • +1.0% ∼+3.0% points on novel classes across FRCN-ft, TFA, and MPSR, regardless of the number of shots.The results are reported on the COCO dataset in Table 8.

B.2. Employ Other Pre-trained Models

PCB remains effective with different ImageNet pre-training paradigms, while stronger pre-training yields larger FSOD improvements; GDL’s coupling range also affects optimization and performance.

  • Employ Other Pre-trained Models: PCB improved performance regardless of whether the classifier used self-supervised, supervised, or weakly supervised pre-training.The compared models were IN-SwAV, IN-1K, and IG-WSL.
  • Employ Other Pre-trained Models: Stronger pre-trained models improved FSOD performance more when used with PCB.This observation was reported for experiments on PASCAL VOC.
  • Why PCB Works ?: PCB prototype similarity maps activated distinct image regions, including for a novel Person category absent from ImageNet-1K classifiers.The analysis used channel-wise cosine similarity between few-shot RoI prototypes and test-image feature maps.
  • The value range of λ: The best GDL results occurred with λ_rpn and λ_rcnn in [0, 1], whereas larger values caused degradation through faster backbone updates and over-fitting.Values above approximately 5 could lead to unreasonable saddle points and collapse solutions.

D. More Visualization of Our Approach

The paper supplements its experiments with visualizations of PCB activations and DeFRCN detections, including both successful and failed novel-object cases.

  • More Visualization of Our Approach: The qualitative failure analysis considers misclassification, mislocalization, and missed detections for novel objects.These error types are shown alongside successful detections.
  • More Visualization of Our Approach: PCB visualizations show that different K-shot prototypes activate distinct areas of the same COCO validation image.The figure uses K = 10 samples and marks prototype types with ⋆.
  • More Visualization of Our Approach: The accompanying experiments cover G-FSOD on PASCAL VOC and COCO, PCB effectiveness across COCO shot counts, pre-training comparisons, and conventional cross-domain detection.The supplied table captions identify these evaluations but do not report their numerical outcomes.
  • More Visualization of Our Approach: DeFRCN visualizations display COCO 10-shot detections with scores larger than 0.7, distinguishing success and failure cases by green and red boxes.The failure cases are intended for qualitative inspection of detection behavior.
Loading 2108.09017v1…