PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector

Qi Fan, Wei Zhuo, Chi-Keung Tang, Yu-Wing Tai

arXiv:1908.01998v4cs.CV

TL;DR

Object detection usually needs substantial annotated data, while detecting unseen categories from only a few examples remains difficult because proposals can miss objects or include background false detections. The paper introduces an Attention-RPN, Multi-Relation Detector, and Contrastive Training strategy together with a 1000-category FSOD dataset. The resulting model achieves state-of-the-art performance across ImageNet Detection and MS COCO in the few-shot setting and can detect novel categories without fine-tuning.

  • Problem

    Few-shot object detection must localize unseen-category objects from few annotated examples despite missed proposals and false detections in cluttered backgrounds.

  • Method

    The paper combines Attention-RPN, Multi-Relation Detectors, and Contrastive Training with a 1000-category, high-quality FSOD dataset.

  • Results

    The model achieves new state-of-the-art performance on ImageNet Detection and MS COCO in the few-shot setting without further network adaptation.

  • Takeaways & Limitations

    A model trained on FSOD can detect objects from novel categories without pre-training or further network adaptation.

  • Takeaways & Limitations

    FSOD contains substantially fewer training images and bounding boxes than large-scale benchmarks such as MS COCO.

Abstract

from arXiv · show

Conventional methods for object detection typically require a substantial amount of training data and preparing such high-quality training data is very labor-intensive. In this paper, we propose a novel few-shot object detection network that aims at detecting objects of unseen categories with only a few annotated examples. Central to our method are our Attention-RPN, Multi-Relation Detector and Contrastive Training strategy, which exploit the similarity between the few shot support set and query set to detect novel objects while suppressing false detection in the background. To train our network, we contribute a new dataset that contains 1000 categories of various objects with high-quality annotations. To the best of our knowledge, this is one of the first datasets specifically designed for few-shot object detection. Once our few-shot network is trained, it can detect objects of unseen categories without further training or fine-tuning. Our method is general and has a wide range of potential applications. We produce a new state-of-the-art performance on different datasets in the few-shot setting. The dataset link is https://github.com/fanq15/Few-Shot-Object-Detection-Dataset.

1. Introduction

The paper targets few-shot object detection, where localizing unseen-category objects amid clutter is difficult and existing approaches require fine-tuning. It proposes a matching-based network with attention, multi-relation detection, and contrastive training, achieving state-of-the-art results without fine-tuning.

  • Problem: Few-shot object detection must localize unseen-category objects from few annotations while avoiding missed objects and false background detections.The paper identifies this localization problem as distinct from few-shot classification and links it to low-scoring proposals from region proposal networks.
  • Method: The proposed model detects novel categories without re-training or fine-tuning by matching support and query objects across multiple network stages.Its design combines a weight-shared network with contrastive training, an attention module on the RPN, and a detector module.
  • Method: Attention improves early proposal quality, while the multi-relation detector suppresses and filters false detections in confusing backgrounds.These modules address complementary stages of the detection pipeline.
  • Results: The model achieves new state-of-the-art performance on the ImageNet Detection and MS COCO datasets in the few-shot setting.The reported result applies across both named evaluation datasets.
  • Contributions: The paper contributes a well-annotated few-shot object detection dataset containing 1000 categories.The dataset is intended to support training and evaluation for novel-category detection.

2. Related Works

Prior object detection methods commonly use region proposals followed by classification, whereas few-shot learning research has focused mainly on classification and often requires retraining for new categories. This work extends matching-based few-shot learning to detection through a Faster R-CNN framework with attention RPN and multi-relation detection.

  • General Object Detection: Proposal-based detectors generate class-agnostic regions with an RPN before refining and classifying them, while proposal-free detectors omit explicit proposals.The related-work discussion contrasts the two main object-detection paradigms.
  • Few-Shot Learning: Few-shot learning research has largely emphasized classification, with comparatively less work on object detection and other computer-vision tasks.The paper notes that some earlier few-shot detection methods require retraining for new categories.
  • This Work: The proposed approach follows matching-network research and learns a matching metric between image pairs using a Faster R-CNN framework.It equips the framework with an attention RPN and multi-relation detector trained by contrastive learning.

3. FSOD: A Highly-Diverse Few-Shot Object Detection Dataset

The FSOD dataset is designed for generalization to unseen categories, addressing limited category diversity, noisy annotations, inconsistent labels, and overlapping train/test categories in existing datasets. It contains 1000 semantically organized categories with challenging visual variation, while remaining compact relative to large benchmarks.

  • Motivation: Existing datasets have limited categories, noisy or inconsistent annotations, and train/test categories that overlap, weakening evaluation of unseen-category generalization.These shortcomings motivate a dataset specifically designed for few-shot object detection.
  • Dataset Construction: The dataset merges semantically equivalent labels, removes poor-quality images and improper boxes, and splits training and test categories without overlap.Its test set contains 200 categories selected for large semantic distance from training categories, while the training set contains 800 categories.
  • Dataset Properties: FSOD contains 83 parent semantics and 1000 leaf categories, with strict splits that create substantial semantic differences between training and test images.The label tree organizes categories across broad semantic groups such as mammals, clothing, and weapons.
  • Dataset Properties: The test set includes large variation in box sizes and aspect ratios, with 26.5% of images containing at least three objects.It also contains many boxes from categories outside the label system, increasing detection difficulty.
  • Scope: FSOD has fewer training images and bounding boxes than MS COCO, reflecting its design as a compact dataset for few-shot learning.MS COCO is cited as containing 123,287 images and around 886,000 bounding boxes.

4. Our Methodology

The method matches support and query images through a weight-shared few-shot detection network, using attention-guided proposals, multi-relation matching, and contrastive training. Its attention RPN improves proposal quality, while the training strategy distinguishes target objects from background and other categories.

  • Deep Attentioned Few-Shot Detection: The network uses a weight-shared query-support framework to learn matching relationships for few-shot detection.The query branch contains Faster R-CNN components, while support branches provide category information for matching.
  • Attention-Based Region Proposal Network: Attention RPN uses support-query similarity to generate relevant proposals while suppressing proposals from other categories.Support features are used in depth-wise cross correlation with query features before proposal generation.
  • Multi-Relation Detector: The multi-relation detector combines global, local, and patch relation heads to match query proposals with support objects.These heads model global image similarity, pixel-wise correspondence, and one-to-many patch relationships.
  • Two-way Contrastive Training Strategy: The contrastive strategy trains matching pairs and three non-matching pair types involving positive or negative supports and foreground or background proposals.Sampling balances foreground-positive, background-positive, and negative-support pairs at a 1:2:1 ratio.
  • Attention-Based Region Proposal Network: 0.9130 versus 0.8804 top-100 proposal recall at IoU 0.5 was achieved by attention RPN versus regular RPN.Attention RPN also obtained ABO of 0.7282 versus 0.7127 and improved final AP50/AP75 across training strategies.

5. Experiments

Experiments evaluate the model across standard few-shot benchmarks, novel-category detection, real-world applications, and training-data diversity. The approach achieves strong results, including state-of-the-art performance without fine-tuning in several settings.

  • ImageNet Detection dataset: 1.7% AP50 gain over the state of the art is achieved on ImageNet-based 50-way 5-shot detection.The comparison uses the evaluation protocol and testing dataset of LSTD and RepMet, with the same MS COCO training set.
  • ImageNet Detection dataset: 41.7% AP50 is obtained by directly applying the FSOD-trained model without fine-tuning on the ImageNet detection test set.This result is reported as better than the fine-tuned model.
  • MS COCO dataset: 6.4% higher AP75 is obtained by the FSOD-trained model than by the fine-tuned model on MS COCO.The FSOD-trained model also has slightly better AP50 performance and can achieve the best performance after further fine-tuning.
  • MS COCO dataset: 7.9%/12.2%/9.5% improvements on AP/AP50/AP75 are achieved over the comparison methods on MS COCO novel categories without further fine-tuning.The fine-tuned model with the same MS COCO training data exceeds Meta R-CNN by 2.4%/1.3%/4.0% on AP/AP50/AP75.
  • Realistic Applications: 3.3%/5.9% gains over LSTD and 4.5%/6.5% gains over Faster R-CNN are reported on AP50/AP75 across 200 testing categories.The comparison is for novel-category detection with five fixed supports per test category.
  • Realistic Applications: 37.0% versus 38.5% AP50 on Cityscapes and 67.4% versus 64.1% on KITTI are obtained against DA Faster R-CNN for wild car detection.The model uses 10-shot supports without further retraining or fine-tuning, while DA Faster R-CNN uses much more related-domain data.
  • Training-data analysis: Category diversity is found essential for few-shot object detection when training image counts are kept similar.The experiments compare MS COCO with FSOD variants having different numbers of training categories.

6. Conclusion

The paper presents a few-shot object detection network and a 1000-category FSOD dataset. Across quantitative and qualitative evaluations, the model detects novel categories without pre-training or further network adaptation.

  • 6. Conclusion: The proposed network combines Attention-RPN, Multi-Relation Detectors, and Contrastive Training for few-shot object detection.The paper also contributes FSOD, a high-quality dataset containing 1000 object categories with few examples per category.
  • 6. Conclusion: The FSOD-trained model detects novel-category objects without pre-training or further network adaptation.The paper reports validation through extensive quantitative and qualitative results on different datasets.

Appendix A: Implementation Details of Multi-Relation Detector

The Multi-Relation Detector compares support and query proposal features through global, local, and patch relation heads. Their outputs are combined to produce matching scores and, for the patch head, bounding-box predictions.

  • Multi-Relation Detector: The detector receives 7 × 7 × C support and query proposal features and sums the three heads' matching scores.The summed score is used as the final matching score.
  • Global-Relation Head: The Global-Relation Head concatenates support and query features, average-pools them, and uses an MLP to generate matching scores.The concatenated feature has size 7 × 7 × 2C and is pooled to a 1 × 1 × 2C vector.
  • Local-Relation Head: The Local-Relation Head applies a shared 1 × 1 × C convolution, computes depth-wise similarity, and generates matching scores.The depth-wise similarity uses S = H = W = 7.
  • Patch-Relation Head: The Patch-Relation Head processes concatenated features with a patch-relation module and predicts matching scores and bounding boxes.Padding, convolution, ReLU, and pooling reduce the feature map from 7 × 7 to 1 × 1.

B.1. Training and Fine-tuning details

The appendix describes preprocessing and evaluation choices used when matching the experimental settings of prior few-shot detection methods. Training on MS COCO removes very small boxes, while target-dataset fine-tuning follows prior protocols.

  • Training and Fine-tuning details: Target-dataset fine-tuning follows the same settings as other methods for fair comparison.The appendix introduces these details as implementation information for Section 5.2.
  • Training and Fine-tuning details: Images with boxes smaller than 32×32 are removed when training on MS COCO.The paper says these boxes are usually low visual quality and hurt training as support examples.

B.2. Evaluation details

The evaluation details describe two protocols: episode-based ablations and standard full-way, N-shot comparisons. The latter can discard support branches after extracting support features.

  • Evaluation setting 1: Ablation experiments use the episode-based evaluation protocol defined in RepMet, with 600 random evaluation episodes.The episodes are intended to give every test image a high probability of evaluation.
  • Evaluation setting 1: Each N-way K-shot episode contains K support images per category and 10 query images per category.Thus, an episode contains K × N supports and 10 × N query images, with categories and images randomly selected.
  • Evaluation setting 2: Baseline comparisons use standard object-detection evaluation with full-way, N-shot support.After support features are obtained, the support branches can be discarded and those features serve as model weights during forward processing.

Appendix D: FSOD Dataset Class Split

This appendix describes the training/testing class split for the proposed FSOD Dataset and states that the split is used in the experiments.

  • Appendix D: FSOD Dataset Class Split: The appendix presents the training/testing class split used for the proposed FSOD Dataset.The passage introduces the split without listing its classes.
  • Appendix D: FSOD Dataset Class Split: The described class split is the split used in the paper’s experiments.No further split composition is specified in the passage.
  • Appendix D: FSOD Dataset Class Split: The section frames dataset organization in terms of separate training and testing classes.This is the only class-split information stated in the passage.

Training Class Split

The training class split section enumerates object categories assigned to the split, spanning animals, foods, household objects, clothing, vehicles, and other items.

  • Training Class Split: The listed training classes include animals and animal-related categories such as crocodile, antelope, whale, lobster, and ostrich.The passage also includes categories such as woodpecker, penguin, lion, shark, and gorilla.
  • Training Class Split: The split includes food categories such as cookie, strawberry, doughnut, grapefruit, corn, taco, avocado, and watermelon.Other listed food-related classes include meat loaf, spaghetti squash, okra, litchi, and muffin.
  • Training Class Split: Household and everyday-object classes include box, scissors, alarm clock, refrigerator, sofa, rocking chair, blender, and briefcase.The lists also contain kitchen, furniture, appliance, and office-related objects.
  • Training Class Split: Clothing and accessory classes include miniskirt, pajama, kimono, bridal gown, abaya, fedora, earrings, and military uniform.The enumerated categories cover multiple types of apparel and wearable accessories.
  • Training Class Split: Vehicle and transportation-related classes include taxi, train, school bus, river boat, bicycle helmet, beach wagon, and segway.The passages list both vehicles and transportation-associated equipment.
Loading 1908.01998v4…