PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Object Detection in Optical Remote Sensing Images

Gong Cheng, Junwei Han

arXiv:1603.06201v2cs.CV

TL;DR

Object detection in optical remote sensing images is important yet challenging, and generic-object literature lacks a deep review. This paper surveys the field across methods, resources, and research directions, while identifying scope limitations in weakly supervised approaches.

  • Problem

    Generic object detection in optical remote sensing images lacks a deep literature review despite its importance for aerial and satellite image analysis.

  • Method

    The paper reviews object-detection methods, including template matching, knowledge-based, OBIA-based, and machine learning-based approaches, together with their processing strategies.

  • Results

    The survey covers recent progress in generic optical RSI object detection, including five datasets, three evaluation metrics, and approximately 270 publications.

  • Takeaways & Limitations

    The paper proposes deep learning-based feature representation and weakly supervised learning-based geospatial object detection as two future research directions.

  • Takeaways & Limitations

    Existing weakly supervised geospatial object-detection methods work only for a specific object class.

Abstract

from arXiv · show

Object detection in optical remote sensing images, being a fundamental but challenging problem in the field of aerial and satellite image analysis, plays an important role for a wide range of applications and is receiving significant attention in recent years. While enormous methods exist, a deep review of the literature concerning generic object detection is still lacking. This paper aims to provide a review of the recent progress in this field. Different from several previously published surveys that focus on a specific object class such as building and road, we concentrate on more generic object categories including, but are not limited to, road, building, tree, vehicle, ship, airport, urban-area. Covering about 270 publications we survey 1) template matching-based object detection methods, 2) knowledge-based object detection methods, 3) object-based image analysis (OBIA)-based object detection methods, 4) machine learning-based object detection methods, and 5) five publicly available datasets and three standard evaluation metrics. We also discuss the challenges of current studies and propose two promising research directions, namely deep learning-based feature representation and weakly supervised learning-based geospatial object detection. It is our hope that this survey will be beneficial for the researchers to have better understanding of this research field.

1. Introduction

Object detection in optical remote sensing images supports many aerial and satellite image-analysis applications but faces substantial visual, data-scale, and application-driven challenges. This survey addresses the lack of a deep review of generic object detection by synthesizing methods, datasets, metrics, challenges, and research directions.

  • Optical RSI object detection determines whether an image contains target objects and locates each predicted object.
  • Object detection in optical RSIs supports environmental monitoring, hazard detection, LULC mapping, GIS updates, precision agriculture, and urban planning.
  • Visual-appearance variation, occlusion, clutter, illumination, shadows, rapidly growing RSI data, and new application requirements create increasing challenges.
  • Higher-resolution satellite and aerial imagery provides detailed spatial and textural information, making more man-made objects separately recognizable and opening prospects for automatic detection.
  • The survey reviews about 270 publications spanning template matching, knowledge-based, OBIA-based, and machine learning-based detection methods.
  • It also summarizes five publicly available datasets and three standard evaluation metrics, discusses challenges, and proposes deep-learning feature representation and weakly supervised learning directions.

2. Taxonomy of methods for object detection

The survey organizes optical RSI object-detection research into four overlapping methodological categories and examines the main processing stages and representative techniques within them.

  • 2. Taxonomy of methods for object detection: Object-detection studies are generally divided into template matching-based, knowledge-based, OBIA-based, and machine learning-based methods.
  • 2. Taxonomy of methods for object detection: These categories can overlap because a single method may belong to different categories.
  • Template matching-based methods: Template matching methods are classified as rigid or deformable according to the selected template type.
  • OBIA-based methods: OBIA-based detection generally consists of image segmentation followed by object classification.
  • Machine learning-based methods: Machine learning-based methods are reviewed through feature extraction, optional feature fusion and dimension reduction, and classifier training.
  • Machine learning-based methods: The reviewed feature types include HOG, BoW, texture, sparse-representation, and Haar-like features, while classifiers include SVM, AdaBoost, kNN, CRF, SRC, and ANN.

3. Template matching-based object detection

Template matching-based detection generates object templates and searches images for the best matches under allowable geometric changes. The survey distinguishes rigid and deformable templates, reviewing their applications, mechanisms, and limitations across remote-sensing objects.

  • 3. Template matching-based object detection: Template matching is among the simplest and earliest object-detection approaches, categorized into rigid and deformable template matching.
  • 3. Template matching-based object detection: The framework first generates a class-specific template by hand-crafting or learning, then matches it across an image using distortion or correlation measures.Common measures include SAD, SSD, NCC, and ED, while matching accounts for translation, rotation, and scale changes.
  • 3.1 Rigid template matching: Rigid templates support applications including road tracking, building extraction, coastline extraction, and petroleum-tank detection.Examples use rectangular templates, road profiles, morphological HMT operators, and multivariate image analysis.
  • 3.1 Rigid template matching: Rigid matching is limited by its need for precise templates, making it sensitive to shape and density variation when viewpoint or intra-class variation prevents exact geometric templates.Road trackers also encounter irregular deformation from junctions, material changes, vehicle occlusions, shadows, and lane markings.
  • 3.2 Deformable template matching: Deformable templates are more flexible than rigid matching because they impose geometric constraints while integrating local image evidence to handle shape deformations and intra-class variations.The survey divides them into free-form and parametric deformable templates.
  • 3.2.1 Free-form deformable templates: Free-form deformable templates represent arbitrary shapes through constrained snake models, whose energy-minimizing contours combine deformation behavior with regularization.Applications include highway extraction, vehicle detection, regular-shaped geospatial-object extraction, aircraft recognition, and building detection.
  • 3.2.2 Parametric deformable templates: Parametric deformable templates encode a specific object class and its variations using density functions, parametric formulas, or prototypes with deformation modes.These approaches are commonly used when prior information about geometrical shape is available, including building and aircraft detection.
  • 3.2.2 Parametric deformable templates: Parametric examples include a DRV parameter for building-neighborhood gray-level variation and a PCA- and kernel-based aircraft shape model for coarse-to-fine recognition.

4. Knowledge-based object detection

Knowledge-based object detection translates prior knowledge about target objects and their context into explicit rules, then tests object hypotheses against those rules. The survey covers geometric and contextual knowledge, including shape, radiometric, spatial, and shadow cues.

  • The survey reports extensive knowledge-based detection research for buildings, roads, and other geospatial objects.The reviewed applications include landslides, bridges, vehicles, urban land changes, crops, drainage channels, and forests.
  • Knowledge-based methods formulate object detection as hypothesis testing by establishing knowledge and rules.Knowledge and rule establishment is identified as the most important step.
  • 4.1 Geometric knowledge: Geometric knowledge uses parametric or generic shape models and object properties to generate and verify hypotheses.Examples include rectangular building models and road models combining geometric and radiometric properties.
  • 4.2 Context knowledge: Context knowledge exploits spatial relationships between target objects and neighboring regions or background.Shadow evidence is repeatedly used to verify building hypotheses and infer building boundaries or locations.
  • Rules that are too strict can miss target objects, whereas rules that are too loose can produce false positives.The central design challenge is converting implicit object understanding into effective explicit detection rules.

5. OBIA-based object detection

OBIA-based detection first segments imagery into meaningful objects and then classifies those objects using spectral, geometric, textural, and contextual information. Its flexibility and GIS integration are useful, but segmentation parameters and accuracy assessment remain difficult to make objective and reproducible.

  • OBIA-based detection consists of image segmentation followed by object classification.Segments represent relatively homogeneous groups of pixels selected using scale, shape, and compactness criteria.
  • OBIA provides a framework for incorporating spatial context and object shape, overcoming limitations of conventional pixel-based classification.The approach has been applied to landslide, land-cover, land-use, and change-detection mapping.
  • Segmentation quality directly influences subsequent classification accuracy, making appropriate parameters for varying objects a central challenge.Parameters must accommodate varying object sizes, shapes, and spatial distributions while satisfying user requirements.
  • MRS partitions images using shape, compactness, and scale, with larger scale values allowing greater internal heterogeneity and larger objects.Scale heavily affects classification accuracy, while traditional selection often relies on subjective trial and error and is difficult to reproduce.
  • Object features can include spectral information, size, shape, texture, geometry, and contextual semantic information.Expert knowledge and GIS-like functionality can make classification context-aware and capable of using multiple sources.
  • Object-based accuracy assessment should consider both class-label accuracy and the spatial characteristics of represented objects.No single all-purpose accuracy measure exists, and suitable sample sizes and measures depend on the application.

6. Machine learning-based object detection

Machine learning-based object detection treats detection as classification by learning appearance and viewpoint variation from labeled or weakly labeled training data. The pipeline represents candidate regions, predicts object labels, and relies heavily on feature extraction, fusion, and classifier training.

  • Machine learning approaches recast object detection as a classification problem and report significant improvements with powerful representations and classifiers.
  • A learned classifier captures variation in object appearances and views using supervised, semi-supervised, or weakly supervised training.
  • The classifier takes sliding windows or object proposals with feature representations and outputs predicted object-or-not labels.
  • Feature extraction, feature fusion, optional dimension reduction, and classifier training are central components of detection performance.The survey therefore focuses primarily on these stages.

6.1 Feature extraction

Feature extraction maps raw optical RSI pixels into discriminative representations for machine-learning detection. The survey reviews gradient, texture, and visual-word features, emphasizing their robustness properties and use in geospatial object detection.

  • Feature extraction maps raw image pixels into a discriminative high-dimensional space used by high-performance detectors.The survey focuses on widely used features for optical remote sensing images.
  • HOG feature: HOG represents gradient distributions across cells and normalized blocks, providing robustness to part-location variability and limited rotation invariance.Its extensions target rotation handling and applications such as ship detection.
  • HOG feature: HOG has achieved broad success in geospatial object detection, with part-model and sparelets methods built on the feature also reporting impressive performance.
  • BoW model: BoW is valued for simplicity, efficiency, and invariance to viewpoint changes and background clutter, supporting geographic classification and object detection.
  • BoW model: The BoW model detects keypoints, computes local descriptors, builds a visual vocabulary, encodes descriptors, and pools them into a global histogram.Spatial pyramid matching preserves keypoint layout by constructing histograms over increasingly finer subregions.
  • Texture features: Gabor and LBP features describe texture through scale-orientation filtering and local pixel-pattern frequencies, respectively.Gabor responses depend on matching local texture orientation and frequency, while LBP compares neighboring pixels with a center pixel.

6.2 Feature fusion and dimension reduction

This section reviews feature fusion for combining complementary visual cues and dimension reduction for producing compact, distinctive representations. It also contrasts unsupervised and supervised reduction, noting dataset-dependent generalization limits for unsupervised methods.

  • Feature fusion: Feature fusion combines multiple visual cues to form more comprehensive representations for object detection.Complementary feature fusion can significantly improve detection.
  • Feature fusion: Linear vector concatenation fuses N feature vectors by connecting them into one longer feature vector.Normalization may be required to avoid bias from scale differences.
  • Feature fusion: Nonlinear fusion methods include heterogeneous feature machines and sparse multimodal learning.Prior work also used boosting to integrate and select heterogeneous features.
  • Dimension reduction: Dimension reduction generates compact low-dimensional features from high-dimensional representations, balancing compactness and distinctiveness.It supports reduced computational complexity and includes unsupervised and supervised approaches.
  • Dimension reduction: PCA preserves projections with maximal variance, but unsupervised reduction has limited generalization because it is dataset-dependent.Supervised methods use class information and therefore hold an advantage for classification.
  • Dimension reduction: LDA can remove correlations in HOG features to produce Whitened Histograms of Orientation.Other supervised approaches include partial least squares and Fisher discriminant analysis.

6.3 Classifier training

This section surveys classifier training for machine-learning-based object detection, covering common classifiers and their operating principles. It highlights trade-offs involving kernels, annotation quality, confidence, and neighborhood size.

  • Classifier training: Classifiers are trained after feature processing to minimize training-set misclassification error, using approaches such as SVM, AdaBoost, kNN, CRF, SRC, and ANN.The survey presents these as alternatives for classifier training in object detection.
  • Support vector machine: SVM supports binary and multiclass classification, with kernel variants projecting samples into higher-dimensional spaces for nonlinear separation.Its soft-margin formulation uses slack variables and C to trade accepted errors against margin maximization.
  • Support vector machine: SVM classifies a test sample by the sign of its decision function, while nonlinear performance depends on selecting an appropriate kernel and parameters.With optimal kernel parameters, SVM can give very good classification results.
  • AdaBoost: AdaBoost iteratively adds weak classifiers, updates example weights, and combines the weak classifiers into a strong classifier.The sign of the final classifier predicts the label, while its absolute value measures confidence.
  • AdaBoost: Gentle AdaBoost is reported to outperform Discrete and Real AdaBoost because it is less sensitive to training-set errors.This property is useful when accurate annotations cannot always be guaranteed in complicated scenes.
  • k-nearest-neighbor: kNN classifies a test sample using the most frequent class among its k closest labeled training samples.It is simple and tunes only one free parameter, but different k values produce different performance and search costs.
  • k-nearest-neighbor: A very large k increases neighbor-search time, whereas a small k may reduce prediction accuracy.Variants have been proposed to address this neighbor-size trade-off.

7. Datasets and evaluation metrics

This section introduces publicly available datasets and standard metrics to make remote-sensing object-detection results more comparable. It summarizes five datasets and explains precision-recall curves, F-measure, and average precision.

  • Datasets: Public benchmarking datasets and standard evaluation metrics are important for comparing aerial- and satellite-image detection algorithms.The survey summarizes five publicly available datasets and three standard metrics.
  • Datasets: NWPU VHR-10 contains 800 very-high-resolution optical remote-sensing images spanning ten object classes with manually annotated bounding boxes.The classes include airplanes, ships, storage tanks, sports fields, harbors, bridges, and vehicles.
  • Datasets: The SZTAKI-INRIA dataset contains rectangular footprints of 665 buildings across nine aerial or satellite images from multiple European locations.The imagery includes RGB aerial and satellite images from several sources.
  • Datasets: The TAS dataset contains 30 color Google Earth images with 1,319 manually labeled cars and axis-aligned bounding boxes.The images are 792×636 pixels, with average car windows of approximately 45×45 pixels.
  • Datasets: OIRDS contains about 900 aircraft-captured overhead images and around 1,800 annotated vehicles, with GSD values from 0.0838-m to 0.3048-m.It is designed for vehicle-detection algorithms.
  • Datasets: The IITM road-extraction dataset contains 200 multispectral 512×512 satellite images split evenly between developed and emerging countries.Each image has a human-operator road-map ground truth.
  • Evaluation metrics: Precision measures the fraction of detections that are true positives, while recall measures the fraction of positives correctly identified.The precision-recall curve supports pixel-level and object-level evaluation.
  • Evaluation metrics: F-measure combines precision and recall through a weighted harmonic mean, with β = 1 assigning them equal importance.Average precision is the area under the precision-recall curve, with higher AP indicating better performance.

8. Promising research directions

The survey proposes deep learning-based feature representation and weakly supervised geospatial object detection as promising directions. It also identifies data, computational, multiclass, and accuracy challenges that remain unresolved.

  • Deep learning-based feature representation: Deep learning-based feature representation is proposed because deep networks automatically learn features and can provide more powerful, semantically abstract representations.Stacked neural-network layers can strengthen feature representation power.
  • Deep learning-based feature representation: Deep learning features depend on large, diverse training datasets to avoid overfitting.Reducing this dependency remains a challenging issue.
  • Deep learning-based feature representation: Deep neural-network feature extraction remains costly for online object detection, motivating further work on computational efficiency.The survey identifies time cost as a notable challenge.
  • Weakly supervised learning-based geospatial object detection: Weakly supervised learning can reduce annotation requirements by training from image-level binary labels indicating whether a target object is present.This direction is motivated by the expense and time required to manually annotate varied objects in large VHR image collections.
  • Weakly supervised learning-based geospatial object detection: Existing weakly supervised methods cannot be directly transferred from natural scenes because remote-sensing images contain appearance variation and complex background clutter.Prior work has shown feasibility for geospatial object detection, but RSI-specific challenges remain.
  • Weakly supervised learning-based geospatial object detection: Current weakly supervised geospatial detectors generally handle only one object class, although remote-sensing images often contain multiple interesting classes.The survey calls for frameworks that identify multiple classes simultaneously.
  • Weakly supervised learning-based geospatial object detection: Existing weakly supervised methods still have unsatisfactory performance and require efforts to improve detection accuracy.This is identified as a second challenge alongside single-class operation.

9. Conclusion

The survey reviews object detection in optical remote sensing as a fundamental and challenging problem, organizing prior methods and summarizing datasets, metrics, challenges, and future directions.

  • Object detection in optical remote sensing images is a fundamental but challenging problem in aerial and satellite image analysis.
  • The survey reviews recent progress in methods for detecting different object types.
  • The methods are broadly categorized into template matching-based, knowledge-based, and machine learning-based approaches.
  • Five publicly available datasets and three standard evaluation metrics are summarized.
  • The survey discusses current challenges and proposes two promising future research directions.
  • The survey aims to help researchers better understand the field of optical remote sensing object detection.
Loading 1603.06201v2…