PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-shot Object Detection on Remote Sensing Images

Jingyu Deng, Xiang Li, Yi Fang

arXiv:2006.07826v2cs.CV

TL;DR

The paper addresses remote-sensing object detection when deep CNN methods need many annotations and generalize poorly to unseen categories. It proposes a few-shot, multi-scale detector that learns from base classes and support images, and reports better performance than established baselines on two benchmark datasets.

  • Problem

    Deep CNN-based remote-sensing detectors require many annotations and can overfit or generalize poorly when trained with only a few samples from new classes.

  • Method

    The method combines a meta-feature extractor, support-image feature reweighting module, and bounding-box prediction module in a multi-scale few-shot detector.

  • Results

    The method achieves better performance than the compared few-shot detector and YOLOv3 on novel classes across the NWPU VHR-10 and DIOR benchmarks.

  • Takeaways & Limitations

    Experiments support using base-class meta-knowledge and few annotated novel-class samples for remote-sensing object detection.

Abstract

from arXiv · show

In this paper, we deal with the problem of object detection on remote sensing images. Previous methods have developed numerous deep CNN-based methods for object detection on remote sensing images and the report remarkable achievements in detection performance and efficiency. However, current CNN-based methods mostly require a large number of annotated samples to train deep neural networks and tend to have limited generalization abilities for unseen object categories. In this paper, we introduce a few-shot learning-based method for object detection on remote sensing images where only a few annotated samples are provided for the unseen object categories. More specifically, our model contains three main components: a meta feature extractor that learns to extract feature representations from input images, a reweighting module that learn to adaptively assign different weights for each feature representation from the support images, and a bounding box prediction module that carries out object detection on the reweighted feature maps. We build our few-shot object detection model upon YOLOv3 architecture and develop a multi-scale object detection framework. Experiments on two benchmark datasets demonstrate that with only a few annotated samples our model can still achieve a satisfying detection performance on remote sensing images and the performance of our model is significantly better than the well-established baseline models.

I. INTRODUCTION

The paper addresses remote-sensing object detection when CNN-based methods require extensive annotations and generalize poorly to unseen categories. It introduces a few-shot, multi-scale method that learns from base classes and detects novel classes using only a few annotated samples.

  • CNN-based remote-sensing detectors require large, diverse datasets, while few samples from new classes can cause overfitting and reduced generalization.
  • The paper introduces a few-shot learning method that learns a detection model from base classes for detecting unseen classes with few annotations.
  • The proposed model combines a meta-feature extractor, feature reweighting module, and bounding-box prediction module.
  • All three modules use multi-scale architectures to support multi-scale object detection.
  • Experiments on two public benchmark datasets demonstrate the method’s effectiveness for few-shot object detection in remote sensing images.

II. RELATED WORK

Related work spans two-stage and one-stage object detectors, remote-sensing detection methods, and deep or few-shot learning approaches. YOLO-family methods provide multi-scale one-stage detection, while deep learning methods address feature extraction for remote-sensing imagery.

  • Two-stage detectors generate region proposals before detecting objects, whereas one-stage detectors directly predict objects from input images.
  • YOLO is a fast one-stage detector, and YOLOv2 and YOLOv3 improve performance using stronger backbones and multi-scale detection.
  • Remote-sensing object detection methods include template matching, knowledge-based, OBIA-based, and machine learning-based approaches.
  • Deep learning methods automatically learn robust features from input images instead of relying on hand-crafted features.
  • Remote-sensing research has extended CNN detectors including R-CNN, Faster R-CNN, SSD, and regression-based one-stage methods.

C. Few-shot detection

Few-shot remote-sensing detection learns from adequately sampled base classes and applies transferable knowledge to novel classes with few annotations. The proposed episodic pipeline uses support images to condition multi-scale query-image detection.

  • Few-shot detection learns transferable knowledge for recognizing new classes when only a few annotated examples are available.
  • The task separates adequately sampled base classes from novel classes represented by only a few annotated samples.
  • The setting reflects applications where collecting large-scale annotations for newly targeted categories is time-consuming.
  • Each episode contains annotated support images and query images from the same class set, with support images supplying meta-knowledge for query detection.
  • FSODM extracts query-image meta-features at three scales, reweights them using labeled support images, and predicts objectness, boxes, and classes at three scales.

B. Meta Feature Extractor

The model extracts multi-scale query features and uses support-image-derived vectors to recalibrate those features for novel-class detection. Channel-wise reweighting produces class-specific feature maps for independent detection across scales.

  • B. Meta Feature Extractor: The meta-feature extractor uses DarkNet-53 and FPN to extract robust representations at multiple scales because remote-sensing objects vary in size.
  • B. Meta Feature Extractor: The selected feature-map scales are 1/32x, 1/16x, and 1/8x, with channel dimensions 1024, 512, and 256, respectively.
  • C. Feature Reweighting Module: The reweighting module maps each labeled support image to a scale-specific vector that adjusts the contribution of meta-features.
  • C. Feature Reweighting Module: Reweighted feature maps are computed by channel-wise multiplication of each meta-feature map with its corresponding reweighting vector.
  • C. Feature Reweighting Module: The module creates three groups of reweighted maps, with each map responsible for detecting one of the support-defined object classes.

D. Bounding Box Prediction

The bounding box prediction module uses multi-scale feature maps and anchor boxes to predict object locations, objectness, and class probabilities. It represents each prediction with location offsets plus objectness and classification scores.

  • Multi-scale anchors: At each scale, three anchor boxes are generated at every feature-map pixel to support multi-scale bounding-box prediction.The feature maps use different scales, with anchor settings illustrated for 800×800 inputs.
  • Prediction outputs: Each anchor produces six outputs: four location values, one objectness score, and one classification score.The first four outputs encode object location, while the remaining outputs score object presence and class membership.
  • Box parameterization: Bounding-box coordinates are computed from four predicted offsets, sigmoid-transformed center coordinates, cell offsets, scale factors, and anchor dimensions.The sigmoid and scale transformation map predictions from feature-map coordinates to the input-image coordinate system.
  • Scores: Objectness is computed with a sigmoid, while class-specific scores are normalized with a softmax across the N predicted class scores.Together, objectness and classification probabilities indicate whether an object is present and which class it belongs to.

E. Loss function

The loss function combines localization, objectness, and classification objectives. Positive anchors contribute to coordinate loss, while objectness weights balance the many negative anchors against positive ones.

  • Localization loss: Object localization uses mean-square error between predicted and ground-truth bounding-box coordinates.Coordinate loss is computed over the four coordinate representations and uses only positive anchors.
  • Anchor assignment: Positive anchors have IoU above a threshold such as 0.7, negative anchors have IoU below a threshold such as 0.3, and the highest-IoU anchor is also positive.These rules assign anchors relative to ground-truth bounding boxes.
  • Objectness loss: Objectness uses binary cross-entropy, with separate weights for object and non-object terms.The weights address the imbalance caused by having substantially more negative than positive boxes.
  • Classification loss: Object classification uses cross-entropy to align predicted classes with ground-truth classes, while background is excluded from classification loss.Objectness already determines whether a predicted box contains an object, so background is ignored in the class loss.

F. Training and Inference

Training uses episodic query-support tasks that separate base-class learning from few-shot adaptation to novel classes. During inference, support images generate class-specific reweighting vectors used to predict boxes in query images.

  • Episode construction: Each training episode contains a query image with annotations and one masked support image from each class-specific group.Query images and support images are processed separately, with query annotations serving as training ground truth.
  • Support representation: Support images are regrouped by object category, and pixels inside each ground-truth bounding box form a binary support mask.The mask assigns one to pixels inside the box and zero elsewhere.
  • Base and novel classes: The dataset is divided into base and novel classes, with many samples used for base training and only a few annotated samples used for novel classes.Novel classes are treated as a new detection task during few-shot tuning.
  • Training procedure: Training first learns network parameters on base classes, then fine-tunes the model on novel classes with few samples.The base-training stage is data-intensive, whereas novel-class tuning is designed to be fast and repeatable when classes are added.
  • Inference procedure: Testing extracts query features, computes support-derived reweighting vectors, forms reweighted feature maps, and generates predicted bounding boxes.These operations follow the testing loop described in Algorithm 1.

IV. EXPERIMENTS AND RESULTS

The model is evaluated for few-shot object detection on two public benchmark remote sensing image datasets and compared with two prior methods.

  • Evaluation setting: Experiments evaluate the few-shot detector on two public benchmark RSI datasets and compare it with methods and [25].The comparison is intended to assess the model’s performance relative to those baselines.

A. Dataset

The study evaluates few-shot detection using NWPU VHR-10 and DIOR, partitioning each dataset into base and novel classes. DIOR provides greater scale, object-size variation, and five novel classes, while NWPU VHR-10 uses four novel classes.

  • NWPU VHR-10: NWPU VHR-10 contains 800 remote sensing images, 10 object categories, and manually annotated positive and negative samples.The dataset includes 650 positive samples and 150 negative samples.
  • DIOR: DIOR contains 23,463 images and 192,472 instances across 20 classes, with 800×800 images and spatial resolutions ranging from 0.5m to 30m.Object sizes vary widely in DIOR.
  • Class partitioning: NWPU VHR-10 uses airplane, baseball diamond, tennis court, and one additional class as novel classes, with the remaining classes used as base classes.The passage states that four classes are novel but names only three explicitly.
  • Class partitioning: DIOR uses airplane, baseball field, tennis court, train station, and windmill as novel classes, with the remaining classes used as base classes.
  • Input processing: Training uses multi-scale square inputs ranging from 384 to 640 pixels, while DIOR images are cropped into 1024×1024 patches with a 512-pixel stride.

C. Comparing methods

FSODM is compared with YOLOv3 and another few-shot detector on novel classes from NWPU VHR-10 and DIOR. It consistently outperforms the few-shot baseline, while few-shot methods outperform YOLOv3 with fewer samples.

  • Evaluation: The evaluation uses mAP, calculated as the average of 11 precision values at recall levels from 0 to 1 in increments of 0.1.
  • NWPU VHR-10: FSODM achieves 166.6%, 120.8%, and 62.5% higher mean mAP than in the 3-, 5-, and 10-shot NWPU VHR-10 settings.
  • NWPU VHR-10: Even with 20 shots, YOLOv3 reaches only 0.28 mAP on NWPU VHR-10, below FSODM’s performance with 3 shots.
  • Category variation: Baseball diamond performs better than other NWPU VHR-10 categories for all compared methods, likely because its size variation is smaller.
  • DIOR: FSODM outperforms on DIOR novel classes, while both few-shot methods substantially outperform YOLOv3 with fewer annotated samples.
  • DIOR: Increasing annotated samples for novel classes consistently improves detection performance for FSODM,, and YOLOv3 on DIOR.
  • Category variation: Baseball field and tennis court achieve better DIOR detection performance, probably because they have smaller in-category variation.

V. DISCUSSION

The discussion examines whether FSODM preserves base-class performance while using few samples for novel classes. Results indicate comparable or better base-class performance and near-baseline performance with substantially fewer novel-class samples.

  • Base-class performance: FSODM and the compared methods achieve similar mAP on NWPU VHR-10 base classes, with only slight differences.
  • Base-class performance: On DIOR base classes, FSODM performs better than and maintains base-class performance under few-shot detection.
  • Qualitative results: Figure 5 compares 10-shot NWPU VHR results and 20-shot DIOR results using red true-positive, yellow false-positive, and blue false-negative boxes.
  • Shot sensitivity: Figure 6 plots detection performance across different shot counts on NWPU VHR-10 against horizontal lines representing performance with all samples.
  • Base-class performance: FSODM achieves the same mAP as conventional YOLOv3 when a large amount of data is available.

B. Number of shots

The study evaluates how detection performance changes with few-shot support and examines the learned reweighting vectors. With limited novel-class annotations, the method approaches the baseline using all training samples, while higher-dimensional vectors yield clearer category clustering.

  • Number of shots: 60 shots, or 8% of novel-class training samples, achieve almost the same detection performance as YOLOv3 trained on all samples.This result is reported on NWPU VHR-10 novel categories.
  • Number of shots: On the baseball diamond class, 20 annotated samples achieve almost the same performance as the all-sample baseline.The paper attributes this to smaller in-category variation and recognizable structural patterns.
  • Reweighting vectors: The reweighting module produces vectors whose relationships are visualized with t-SNE after dimensionality reduction.The vectors are generated from 400 DIOR support images, with 20 images per category.
  • Reweighting vectors: Reweighting vectors from the same categories tend to aggregate, suggesting that they characterize object-class information from support masks.The 1024-dimensional visualization shows better clustering than the 256- and 512-dimensional visualizations.
  • Method: The method combines a meta feature extractor, feature reweighting module, and bounding box prediction module in a multi-scale architecture.It learns from base classes and generalizes to novel classes with few annotated samples.
Loading 2006.07826v2…