PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Positive Sample Refinement for Few-Shot Object Detection

Jiaxi Wu, Songtao Liu, Di Huang, Yunhong Wang

arXiv:2007.09384v1cs.CV

TL;DR

FSOD must learn novel classes from few annotations, but its sparse and biased scale distribution creates a distinctive detection challenge. MPSR addresses this by generating multi-scale positive object pyramids and refining predictions through an auxiliary branch integrated with Faster R-CNN and FPN. Experiments on PASCAL VOC and MS COCO report state-of-the-art results and significant improvements over counterparts.

  • Problem

    Few-shot object detection is difficult because novel-class annotations sparsely cover object scales, while existing few-shot research has focused mainly on classification.

  • Method

    MPSR generates multi-scale positive samples as object pyramids and refines predictions through a shared-weight auxiliary branch integrated with Faster R-CNN and FPN.

  • Results

    MPSR reports better scores than current state-of-the-art methods on PASCAL VOC and MS COCO, including 16.2% mAP improvement over the best baselines on the 1-shot split-1.

  • Takeaways & Limitations

    Enriching positive-sample scales while limiting improper negatives provides an effective FSOD solution across the evaluated datasets and settings.

  • Takeaways & Limitations

    Multi-scale training can introduce extra improper negative samples, which harms performance under FSOD’s sparse and biased data distribution.

Abstract

from arXiv · show

Few-shot object detection (FSOD) helps detectors adapt to unseen classes with few training instances, and is useful when manual annotation is time-consuming or data acquisition is limited. Unlike previous attempts that exploit few-shot classification techniques to facilitate FSOD, this work highlights the necessity of handling the problem of scale variations, which is challenging due to the unique sample distribution. To this end, we propose a Multi-scale Positive Sample Refinement (MPSR) approach to enrich object scales in FSOD. It generates multi-scale positive samples as object pyramids and refines the prediction at various scales. We demonstrate its advantage by integrating it as an auxiliary branch to the popular architecture of Faster R-CNN with FPN, delivering a strong FSOD solution. Several experiments are conducted on PASCAL VOC and MS COCO, and the proposed approach achieves state of the art results and significantly outperforms other counterparts, which shows its effectiveness. Code is available at https://github.com/jiaxi-wu/MPSR.

1 Introduction

Few-shot object detection addresses detection with scarce annotations, but scale variation is especially difficult because novel-class examples sparsely cover the scale space. MPSR enriches positive-sample scales through object pyramids and refinement, improving FSOD performance.

  • Motivation: Few-shot object detection is needed because detectors trained with inadequate annotations suffer performance drops, while collecting and labeling some categories is difficult.Existing few-shot research has focused mainly on classification, leaving FSOD comparatively underexplored.
  • Scale challenge: FSOD is challenged by sparse and divergent scale distributions for novel classes, making scale variation harder than in ordinary detection.The limited labels of novel classes can leave large gaps between few-shot and original scale distributions.
  • Scale challenge: Multi-scale inputs can introduce improper negative samples containing features from positive objects, which harms recognition in the few-shot setting.Abundant data can help suppress such local patterns, but scarce FSOD data cannot reliably learn this diversity.
  • Approach: MPSR generates multi-scale positive samples as object pyramids and refines predictions at different scales without largely increasing improper negatives.It uses an auxiliary branch with Faster R-CNN and FPN, sharing weights with the original detector.
  • Contributions: MPSR is presented as the first work to discuss FSOD scale problems and is evaluated through quantitative and qualitative analyses and comprehensive experiments.The study reports significant improvements demonstrating the approach’s advantage.

2 Related Work

Related work spans few-shot classification, generic object detection, limited-annotation detection, and FSOD. Prior FSOD methods mainly transfer, metric-learn, or reweight detector knowledge, while scale sparsity remains a distinct challenge.

  • Few-Shot Image Classification: Few-shot image classification methods adapt models to unseen categories using weight initialization, parameter prediction, or metric learning.These techniques have commonly influenced approaches to few-shot object detection.
  • Generic Object Detection: Generic object detectors are commonly categorized as one-stage or two-stage systems, with two-stage detectors generating region proposals before classification.Scale invariance is addressed through multi-scale feature maps, scale normalization, and image pyramids.
  • Few-Shot Object Detection: Unlike generic detection, FSOD suffers more seriously from scale variations because its few samples sparsely distribute across the scale space.This motivates addressing scale distribution specifically rather than relying only on prior generic-detection techniques.
  • Object Detection with Limited Annotations: Limited-annotation detection includes weakly supervised methods using image-level labels and semi-supervised methods using additional assumptions or supervision.Weakly supervised work often ranks and classifies region proposals through multiple-instance learning.
  • Few-Shot Object Detection: FSOD methods use transfer learning, distance-metric classifiers, similarity and relationship modeling, or meta-model-based feature reweighting.These approaches modify detector representations or classification components to adapt to novel classes with few examples.

3 Background

This section defines the FSOD training protocol and Faster R-CNN baseline, then examines FPN and image pyramids as ways to enrich scale representation. Image pyramids introduce improper negative samples that are harmful under FSOD’s sparse and biased annotations.

  • Few-Shot Object Detection Protocols: FSOD divides classes into well-annotated base classes and few-shot novel classes, using base training followed by fine-tuning on balanced few-shot data.The protocol follows a two-step training paradigm.
  • Basic Detection Model: Faster R-CNN is adopted as the basic two-stage detector, with an RPN generating proposals before the detection head predicts classes and box offsets.The RPN produces objectness scores and bounding-box regression offsets; proposals are converted into RoI features for the detection head.
  • FPN for Multi-Scale Detection: FPN generates semantic feature maps at multiple scales and remains practically useful for FSOD under restricted training conditions.This motivates Faster R-CNN with FPN as a second baseline.
  • Image Pyramids for Multi-Scale Training: Image pyramids enrich object scales through multi-scale training, but anchor matching produces increasing numbers of improper negative samples.These samples can contain features belonging to positive samples and may be true positives in other contexts.
  • Image Pyramids for Multi-Scale Training: Extra improper negative samples further hurt performance because sparse and biased FSOD annotations provide insufficient context for suppressing misleading local patterns.With abundant data, networks can learn diverse contexts; this protection is limited in FSOD.

4 Multi-Scale Positive Sample Refinement

MPSR enriches positive-object scale diversity by extracting and resizing objects into pyramids, then selecting scale-consistent FPN features to refine Faster R-CNN’s RPN and RoI predictions.

  • Object Pyramids: Objects are cropped independently with a square window and resized to multiple scales, forming object pyramids that enrich positive-sample scale diversity.The resize scales include 32^2, 64^2, 128^2, 256^2, 512^2, and 800^2 pixels.
  • Scale-Consistent Feature Selection: Specific FPN feature maps are selected for each object scale before refinement rather than relying on standard anchor matching for cropped single objects.The selection preserves scale consistency while avoiding wasteful matching and additional improper negatives.
  • Scale-Consistent Feature Selection: For each object, two specific FPN feature maps are activated and fed separately to the RPN and detection heads.Table 1 summarizes feature-map selection across different object scales.
  • RPN Refinement: RPN refinement selects centric 2^2 features and places anchors with 1:2, 1:1, and 2:1 aspect ratios at the sampled locations.Only the feature map with the object’s consistent scale is activated for refinement.
  • Framework and Training: The training framework runs Faster R-CNN with FPN and the refinement branch in parallel, sharing weights; the extra branch is removed for inference.The detection-head loss weights MPSR positive RoI classification by λ=0.1 by default because these positives are relatively few.

5 Experiments

Experiments on PASCAL VOC and MS COCO show that MPSR improves few-shot detection across settings by enriching sparse object scales and refining positive samples.

  • 5.1 Datasets and Settings: MPSR is evaluated on PASCAL VOC and MS COCO under established few-shot detection protocols, using mAP on VOC and standard COCO metrics.VOC uses 1-/3-/5-/10-shot settings; COCO uses 10-/30-shot settings.
  • 5.2 Results: 82.1%/82.7%/82.9% mAP is achieved on VOC base classes across the three splits before few-shot fine-tuning.These values establish the pretrained MPSR performance before novel-class adaptation.
  • 5.2 Results: 16.2% mAP improvement is obtained over the best baseline on the 1-shot first VOC split, while MPSR improves Baseline-FPN across all sample counts and splits.MPSR also reports state-of-the-art VOC scores against the compared few-shot detectors.
  • 5.2 Results: 14.1% mAP is reported on COCO, improving the SOTA score from 12.4% and increasing 30-shot performance over Baseline-FPN by 0.4%.The method improves recognition of small, medium, and large objects simultaneously.
  • 5.2 Results: 42.3% is achieved in cross-dataset COCO-to-VOC evaluation, compared with 39.3% for Baseline-FPN and 37.4% for Meta R-CNN.The authors report this result as indicating better cross-domain generalization.
  • 5.3 Analysis of Sparse Scales: Scale-limited 10-shot datasets cause a 28.9% Baseline-FPN performance drop for bus, while MPSR relieves the reduction.The experiment demonstrates the effect of extremely sparse and biased scale distributions.
  • 5.3 Analysis of Sparse Scales: Approximately 12% improvement over multi-scale training is obtained when only one instance is available per category, supporting MPSR's scale-enrichment strategy.MPSR outperforms scale augmentation and image pyramids on novel classes in the compared VOC experiments.
  • 5.4 Ablation Studies: RPN-only and RoI-only refinement both exceed Baseline-FPN, while combining them reaches the top score because the branches play complementary roles.The ablation evaluates the contributions of the two refinement branches separately and jointly.

6 Conclusions

The paper addresses sparse scale distributions in few-shot object detection with MPSR, which generates and refines multi-scale positive samples through an auxiliary Faster R-CNN with FPN branch.

  • 6 Conclusions: MPSR generates multi-scale positive samples as object pyramids and refines detector predictions at different scales.The approach enlarges positive-sample scale coverage while limiting improper negative samples.
  • 6 Conclusions: Integrating MPSR with Faster R-CNN and FPN yields better scores than current state-of-the-art methods on PASCAL VOC and MS COCO.The experiments are described as extensive and demonstrate the approach's advantage.

A.1 Ablation of Refinement in the Two-Step Training

The refinement branch helps at both base training and few-shot fine-tuning stages, with the strongest results when applied throughout both stages. Its fine-tuning contribution is especially pronounced when novel classes have only 1 or 3 instances.

  • Ablation of Refinement in the Two-Step Training: Applying refinement during both base training and few-shot fine-tuning achieves the best results.The two training stages provide complementary benefits.
  • Ablation of Refinement in the Two-Step Training: Refinement during base training alone outperforms Baseline-FPN.This indicates that detection across various scales benefits from refinement even before few-shot adaptation.
  • Ablation of Refinement in the Two-Step Training: Refinement during few-shot fine-tuning alone exceeds base-only refinement and Baseline-FPN by a large margin at 1- or 3-shot settings.The gain is largest when the number of novel-class instances is extremely small.
  • Ablation of Refinement in the Two-Step Training: Table 8 reports novel-class mAP (%) on VOC split-1 for refinement applied at different training stages.The table evaluates where refinement is used within the two-step training process.

A.2 Complete Results on PASCAL VOC

The paper reports complete few-shot results on PASCAL VOC, including class-wise performance and mean scores across novel classes. It also includes 2-shot results for consistency with the broader evaluation.

  • Complete Results on PASCAL VOC: Complete results are presented on the few-shot PASCAL VOC datasets.The results follow the reporting format used in prior work.
  • Complete Results on PASCAL VOC: The VOC evaluation includes 2-shot experimental results for consistency.This supplements the complete results reported in the section.
  • Complete Results on PASCAL VOC: Table 9 reports AP (%) for each novel class and mAP (%) across novel classes.The table organizes results by class split and includes a mean score.
  • Complete Results on PASCAL VOC: The reported novel classes include bird, bus, cow, mbike, sofa, boat, cat, and sheep.These class names appear in the VOC result listings.
Loading 2007.09384v1…