PaperlayerView exact revision on arXiv

Source-linked AI summary

The Effects of Super-Resolution on Object Detection Performance in Satellite Imagery

Jacob Shermeyer, Adam Van Etten

arXiv:1812.04098v3cs.CV

TL;DR

The paper asks whether super-resolution can improve object detection in satellite imagery, where small objects and limited evidence make detection difficult. It evaluates VDSR and RFSR enhancements with SIMRDWN detectors across multiple resolutions, finding the largest mAP gains at the finest resolution and little benefit at coarser resolutions.

  • Problem

    The effect of super-resolution on object detection in satellite imagery has not been quantified across multiple resolutions, despite the difficulty of detecting small objects.

  • Method

    The study applies VDSR and RFSR at multiple enhancement levels and resolutions, then evaluates custom SIMRDWN object detectors on native and super-resolved imagery.

  • Results

    Super-resolving native 30 cm imagery to 15 cm yields a 13−36% improvement in mAP, while YOLT gains little to no boost at 120−480 cm.

  • Takeaways & Limitations

    Super-resolution is most beneficial for object detection at the highest tested resolutions and provides limited gains for coarser imagery.

  • Takeaways & Limitations

    Inconsistent xView labels, including missed and mislabeled objects, constrain the evaluation and can increase false-positive and misclassification rates.

Abstract

from arXiv · show

We explore the application of super-resolution techniques to satellite imagery, and the effects of these techniques on object detection algorithm performance. Specifically, we enhance satellite imagery beyond its native resolution, and test if we can identify various types of vehicles, planes, and boats with greater accuracy than native resolution. Using the Very Deep Super-Resolution (VDSR) framework and a custom Random Forest Super-Resolution (RFSR) framework we generate enhancement levels of 2x, 4x, and 8x over five distinct resolutions ranging from 30 cm to 4.8 meters. Using both native and super-resolved data, we then train several custom detection models using the SIMRDWN object detection framework. SIMRDWN combines a number of popular object detection algorithms (e.g. SSD, YOLO) into a unified framework that is designed to rapidly detect objects in large satellite images. This approach allows us to quantify the effects of super-resolution techniques on object detection performance across multiple classes and resolutions. We also quantify the performance of object detection as a function of native resolution and object pixel size. For our test set we note that performance degrades from mean average precision (mAP) = 0.53 at 30 cm resolution, down to mAP = 0.11 at 4.8 m resolution. Super-resolving native 30 cm imagery to 15 cm yields the greatest benefit; a 13-36% improvement in mAP. Super-resolution is less beneficial at coarser resolutions, though still provides a small improvement in performance.

1. Introduction

The study addresses the largely unexplored relationship between super-resolution and object detection in satellite imagery. It evaluates whether enhanced imagery improves detection across resolutions, object types, and rapid detection frameworks.

  • Research gap: Super-resolution and object detection remain insufficiently studied together in satellite or overhead imagery.Prior work applied super-resolution as preprocessing but did not quantify its effect on satellite-image detection across multiple resolutions.
  • Motivation: Small, densely clustered, rotation-invariant objects and limited labeled datasets make satellite-image detection difficult.Cars may occupy as few as 10 pixels, while few datasets provide suitable object labels.
  • Practical motivation: The study is motivated by the possibility of enhancing coarser imagery with smaller, cheaper satellites and broader field of view.The paper presents this as a potential cost-benefit rationale rather than an established outcome.
  • Super-resolution background: Single-image super-resolution derives a high-resolution image from one low-resolution image by learning likely high-resolution features.The paper describes VDSR as a deep network that learns residual images to transform low-resolution inputs into high-resolution outputs.
  • Study scope: The study tests object detection across five GSD resolutions and 35 resolution profiles, using native and super-resolved imagery.The analysis includes six super-resolution outputs per resolution and compares performance as a function of resolution.
  • Detection framework: The authors pair super-resolution with rapid detection algorithms in SIMRDWN to analyze real-world satellite imagery efficiently.SIMRDWN incorporates SSD, Faster-RCNN, R-FCN, and YOLT for satellite-image processing.

3. Dataset

The study uses xView imagery and aggregated object classes to evaluate detection under simulated coarser resolutions. Dataset labeling inconsistencies and object-scale variation are important constraints on this evaluation.

  • Dataset: The xView dataset contains 1,415 km2 of 30 cm WorldView-3 imagery from 56 global locations across six continents.Its object-detection labels include 1 million object instances across 60 classes.
  • Label quality: xView contains mislabeled and missing objects, causing false positives and inaccurate post-inference misclassification labels.The paper illustrates incorrectly sized cars, erroneous boat labels, and missing cars in Figure 1.
  • Class schema: The analysis aggregates xView labels into five classes: Small Aircraft, Large Aircraft, Small Vehicle, Bus/Truck, and Boat.The aggregation combines related original xView classes, including multiple vehicle and vessel categories.
  • Resolution simulation: The study simulates coarser imagery by applying Gaussian blur and interarea decimation to native 30 cm data.The simulated resolutions are 60, 120, 240, and 480 cm, with corresponding object-size histograms measured in pixels.
  • Evaluation split: Ground-truth data and super-resolution outputs are split into 60% training and 40% validation sets using consistent images across resolutions.Using the same images across resolution conditions supports comparison of validation scores.

4. Super-Resolution Techniques

The study uses VDSR and a custom RFSR method to enhance imagery by 2×, 4×, and 8× across resolutions from 30 cm to 4.8 m. RFSR is designed as a faster, less computationally intensive alternative, while VDSR uses a convolutional neural network.

  • 2×, 4×, and 8× enhancements are generated across five native resolutions ranging from 30 cm to 4.8 m.
  • RFSR is a custom random-forest super-resolution method designed for minimal training time and high inference speeds without requiring GPUs.It is evaluated against the near-state-of-the-art VDSR approach.
  • RFSR parameters were tuned to maximize PSNR while keeping training time at 4 hours or less per enhancement level on a 64GB RAM CPU.Deeper trees and more estimators could mildly improve PSNR, but would increase training time.
  • Both methods train on luminance-based residual image pairs created by degrading high-resolution imagery and emphasizing edge effects.The process uses shifted low-resolution images and removes homogeneous areas through residual normalization.

5. Object Detection Techniques

The study uses YOLT and SSD within SIMRDWN to detect objects in native and super-resolved satellite imagery. This produces models spanning detection architectures, enhancement methods, and resolutions for detailed performance comparisons.

  • YOLT and SSD are selected within SIMRDWN because prior results ranked them above Faster R-CNN and RFCN.
  • Models are trained on native imagery and on RFSR- and VDSR-processed outputs for object-detection comparison.

6. Metrics

The study evaluates super-resolution using both image-quality and object-detection metrics, comparing native and enhanced satellite imagery across resolutions and detection models. Detection performance declines sharply with coarser resolution, while super-resolution offers its clearest gains at the finest resolutions.

  • Evaluation metrics: PSNR and SSIM evaluate super-resolution image quality, but do not directly measure object-detection improvement.The study therefore compares visual enhancement with downstream detection performance.
  • Super-resolution performance: VDSR generally produces the strongest super-resolution results, while RFSR is comparable in some cases and both methods degrade with greater enhancement.Performance is strongest on 60 cm imagery, where features are considered easier to detect and enhance.
  • Object detection performance: At native 30 cm resolution, YOLT reaches mAP = 0.53 versus mAP = 0.30 for SSD, a 77% advantage for YOLT.The study computes mAP on a 338-image test set spanning six continents and 632 sq. km.
  • Object detection performance: 22−27% performance loss occurs from 30 cm to 120 cm, followed by 73−100% loss from 120 cm to 480 cm across broad object classes.These results quantify detection degradation as native resolution becomes coarser.
  • Super-resolution gains: +13% mAP improvement occurs when 30 cm imagery is enhanced to 15 cm, while 60 cm to 15 cm yields a 14−20% improvement.Enhancing 60 cm to 30 cm is substantially less effective than enhancing 60 cm to 15 cm.
  • Super-resolution gains: With SSD, super-resolution improves performance at all resolutions except 60 cm to 30 cm, although the 480 cm gain is driven partly by native mAP = 0.0.Performance increases significantly once objects exceed approximately 20 pixels in extent.

8. Conclusions

The study evaluates super-resolution for object detection across satellite-image resolutions using VDSR and RFSR paired with SIMRDWN detectors. Detection performance declines as resolution coarsens, while super-resolution helps most at the finest resolutions.

  • 8. Conclusions: Over 250,000 labeled objects across diverse environments support evaluation of VDSR and RFSR with advanced object detection methods.The imagery is degraded from 30 cm to coarser resolutions to assess super-resolution effects across sensor resolutions.
  • 8. Conclusions: 22−27%: baseline object detection performance decreases when resolution degrades from 30 cm to 120 cm for YOLT and SSD.Both models are part of the SIMRDWN object detection framework.
  • 8. Conclusions: 13−36% improvement in mAP: super-resolving native 30 cm imagery to 15 cm provides the greatest benefit at the highest resolutions.The reported improvement applies across both object detection frameworks.
  • 8. Conclusions: −0.02 to +0.04 change in mAP: super-resolution provides little to no boost for YOLT at coarser resolutions from 120 cm to 480 cm.With SSD, coarse-resolution super-resolution provides a greater boost, but most classes remain worse than YOLT using native imagery.

1. Super-Resolution Outputs

The supplemental examples show how vehicle appearance changes with resolution and how VDSR and RFSR differ in recovering car shape. At finer inputs and moderate enhancement, both methods produce favorable outputs, while stronger enhancement exposes differences between them.

  • 1. Super-Resolution Outputs: Cars rapidly degrade into amorphous pixelated blobs as image resolution becomes coarser.The examples emphasize the difficulty of preserving small-object structure during resolution degradation.
  • 1. Super-Resolution Outputs: 2× enhancement: VDSR and RFSR perform favorably for 30 cm data and super-resolved outputs.The example maps 60 cm input imagery to 30 cm super-resolved output.
  • 1. Super-Resolution Outputs: 4× enhancement: VDSR generally recovers the 30 cm car shape, whereas RFSR struggles to do so.This comparison uses 120 cm input imagery and produces 30 cm super-resolved output.
  • 1. Super-Resolution Outputs: The supplemental comparison pairs ground-truth 30 cm imagery with simulated 60 cm input imagery and shows 2× RFSR beside 2× VDSR outputs.The 60→30 models are presented with RFSR on the left and VDSR on the right.

2. Super-Resolution Scores- Bicubic Decimation with No Blurring

Table 1 reports PSNR and SSIM for 2×, 4×, and 8× super-resolution across five output GSDs under an idealized bicubic-decimation setting without blurring.

  • 2. Super-Resolution Scores- Bicubic Decimation with No Blurring: Table 1 reports average PSNR / SSIM scores for 2×, 4×, and 8× scales across five super-resolution output GSDs.The xView validation dataset contains 281 images.
  • 2. Super-Resolution Scores- Bicubic Decimation with No Blurring: The test imagery is not blurred, bicubically decimated, and then bicubically upsampled before evaluation.These are described as ideal or traditional super-resolution settings.
  • 2. Super-Resolution Scores- Bicubic Decimation with No Blurring: Bicubic scores provide a baseline by simply upscaling low-resolution images to the high-resolution image size.The validation dataset used is xView.

3. Object Detection Performance

The figures illustrate YOLT detections across native and super-resolved imagery at multiple resolutions. Their visual encoding distinguishes object classes, while the examples consistently use a low detection threshold that trades fewer false negatives for more false positives.

  • 3. Object Detection Performance: YOLT bounding boxes encode cars in green, buses/trucks in blue, small aircraft in red, and large aircraft in yellow.The color scheme applies across the resolution examples.
  • 3. Object Detection Performance: Native 30 cm imagery is evaluated with YOLT at a 0.1 detection threshold.This low threshold yields fewer false negatives but more false positives.
  • 3. Object Detection Performance: VDSR 2× super-resolved 15 cm imagery is evaluated with YOLT at the same 0.1 detection threshold.The figure uses imagery produced from the VDSR 2× model.
  • 3. Object Detection Performance: RFSR 4× super-resolved imagery is shown at both 30 cm and 60 cm output resolutions using YOLT with a 0.1 threshold.These examples use the RFSR 4× model and retain the fewer-false-negatives, more-false-positives trade-off.
  • 3. Object Detection Performance: VDSR 4× super-resolved 120 cm imagery is also evaluated with YOLT using a 0.1 detection threshold.The threshold favors fewer false negatives at the cost of more false positives.

4. Object Detection Performance Curves and Tables

This section presents object-detection performance curves and tables across object classes, sensor resolutions, native imagery, and super-resolved outputs, including statistical differences from baseline models.

  • Statistical comparison: The section describes statistical comparison procedures for super-resolved and baseline models, including joint error estimates and model differences.The procedure follows the approach used in Table 4 of the main text.
  • Performance curves: Performance is evaluated for multiple object classes using native-resolution YOLT and SSD models, alongside super-resolved YOLT data.The figures cover boats, small and large aircraft, small vehicles, and buses and trucks.
  • Super-resolution comparisons: Performance-change figures compare native-resolution results with 2x and 4x super-resolved YOLT data for each object class.Separate comparisons are provided for boats, small aircraft, large aircraft, small vehicles, and buses and trucks.
  • Performance tables: Table 2 compares object-detection performance in mAP between native imagery and super-resolved outputs across input sensor resolutions.RFSR and VDSR results are reported with errors and statistical differences from the baseline model.
  • Performance curves: The resolution curves plot average precision against sensor resolution for boats, aircraft, vehicles, buses, and trucks.Each class-specific figure identifies sensor resolution on the lower axis and average precision on the y-axis.
  • Performance tables: Tables 3–7 report class-specific performance for boats, small aircraft, large aircraft, small vehicles, and trucks and buses.For RFSR and VDSR at each resolution, the tables include error estimates and statistical differences from baseline models.
Loading 1812.04098v3…