PaperlayerView exact revision on arXiv

Source-linked AI summary

Remote Sensing Image Super-resolution and Object Detection: Benchmark and State of the Art

Yi Wang, Syed Muhammad Arsalan Bashir, Mahrukh Khan, Qudrat Ullah, Rui Wang, Yilin Song, Zhe Guo, Yilong Niu

arXiv:2111.03260v1cs.CVcs.AIcs.LGeess.IV

TL;DR

Small-object detection in remote-sensing imagery needs better benchmarking. This paper reviews existing approaches, introduces the RSSOD benchmark, and proposes MCGR with an auxiliary YOLOv5 detector, reporting high-resolution imagery that emphasizes small objects and detector accuracies surpassing two-stage methods.

  • Problem

    Benchmarking methods for small-object detection in remote-sensing imagery remains needed because object size critically affects detection tasks.

  • Method

    The paper constructs the public RSSOD dataset and proposes MCGR, a cyclic super-resolution model with residual feature aggregation, for benchmarking super-resolution-based detection.

  • Results

    The proposed dataset provides the highest spatial resolution among compared detection datasets, with ~1000×1000-pixel images emphasizing small objects, while the detector surpasses two-stage detectors in accuracy.

  • Takeaways & Limitations

    RSSOD supplies a high-resolution benchmark focused on small objects for evaluating remote-sensing detection methods.

  • Takeaways & Limitations

    Existing scene-classification datasets used for detection are limited by low spatial resolution and other reported drawbacks.

Abstract

from arXiv · show

For the past two decades, there have been significant efforts to develop methods for object detection in Remote Sensing (RS) images. In most cases, the datasets for small object detection in remote sensing images are inadequate. Many researchers used scene classification datasets for object detection, which has its limitations; for example, the large-sized objects outnumber the small objects in object categories. Thus, they lack diversity; this further affects the detection performance of small object detectors in RS images. This paper reviews current datasets and object detection methods (deep learning-based) for remote sensing images. We also propose a large-scale, publicly available benchmark Remote Sensing Super-resolution Object Detection (RSSOD) dataset. The RSSOD dataset consists of 1,759 hand-annotated images with 22,091 instances of very high resolution (VHR) images with a spatial resolution of ~0.05 m. There are five classes with varying frequencies of labels per class. The image patches are extracted from satellite images, including real image distortions such as tangential scale distortion and skew distortion. We also propose a novel Multi-class Cyclic super-resolution Generative adversarial network with Residual feature aggregation (MCGR) and auxiliary YOLOv5 detector to benchmark image super-resolution-based object detection and compare with the existing state-of-the-art methods based on image super-resolution (SR). The proposed MCGR achieved state-of-the-art performance for image SR with an improvement of 1.2dB PSNR compared to the current state-of-the-art NLSN method. MCGR achieved best object detection mAPs of 0.758, 0.881, 0.841, and 0.983, respectively, for five-class, four-class, two-class, and single classes, respectively surpassing the performance of the state-of-the-art object detectors YOLOv5, EfficientDet, Faster RCNN, SSD, and RetinaNet.

Introduction

Remote sensing object detection is difficult because images contain multiscale objects, many of which occupy only a few pixels, while deep learning methods depend on large, high-quality datasets. The paper addresses this need with the RSSOD benchmark and the MCGR super-resolution network for detection in low-resolution images.

  • RS object detection supports applications including geographic mapping, crop harvest analysis, disaster management, traffic planning, and navigation.
  • Small objects such as vehicles may occupy only a few pixels, making their identification and detection challenging in optical RS images.A 4×1.5 m2 vehicle at 0.5 m GSD occupies a grid of only 8×3 pixels.
  • The RSSOD dataset provides a large-scale public benchmark for small-object detection in urban environments, containing 22,091 hand-annotated instances across five classes.Its geographic information and annotation orientation make detection challenging.
  • The paper benchmarks RSSOD performance against current state-of-the-art methods and proposes MCGR for object detection in low-resolution images at scale factors 2 and 4.

2. A Review of Remote Sensing Object Detection Datasets and Methods

This section reviews remote-sensing object-detection datasets and methods, emphasizing challenges from diverse object scales and orientations. It contrasts generic end-to-end detectors with super-resolution-based approaches and highlights RSSOD’s high spatial resolution and small-object emphasis.

  • Dataset characteristics: Remote-sensing objects vary substantially in size and orientation because of top-view imaging and satellite sensor design.These variations make object detection challenging, particularly for small objects.
  • Dataset characteristics: Existing datasets span scene classification, object detection, and segmentation, including NWPU-RESISC45, NWPU VHR-10, VEDAI, DOTA, and DIOR.Their scales, classes, resolutions, and annotation formats differ across tasks.
  • RSSOD dataset: RSSOD has the highest spatial resolution among compared object-detection datasets, with approximately 1000×1000-pixel images emphasizing small objects.The comparison is presented in Table 1.
  • Detection methods: Object-detection methods are mainly generic end-to-end detectors or super-resolution-based methods that enhance image quality before detection.Detection architectures may be single-stage or dual-stage, with dual-stage methods generating region proposals before box classification.

3. The Proposed RSSOD Dataset

The proposed RSSOD dataset addresses the lack of high-resolution urban object-detection data by extracting VHR patches and manually annotating diverse, predominantly small objects. It uses five object classes, conventional train-validation-test splitting, and multiple class-group configurations for detector evaluation.

  • Data characteristics: The dataset contains predominantly small objects with random locations and orientations collected from multiple sensors.Object locations and sizes are illustrated through normalized frequency plots.
  • Class selection: The five final classes are vehicle, tree, airplane, ship, and low vegetation; buildings were omitted because they occupy most urban image space.Class instances and frequencies are reported in the dataset’s class-distribution materials.
  • Data split: 1232 training, 351 validation, and 176 test images were assigned using the conventional 70:20:10 split.The corresponding instance distributions are shown for the three subsets.
  • Annotation: All objects were manually labeled with horizontal YOLO-format bounding boxes and refined through three rounds of review.Annotations use the format <class identifier, x, y, w, h>, with (x, y) as the box center and (w, h) as width and height.
  • Annotation: The annotations were grouped into five-, four-, two-, and one-class sets to evaluate detectors across different class subsets.These configurations facilitate object detection on varying class combinations.

4. Small-object Detection using Multiclass Cyclic GAN with RFA

The section proposes a Multiclass Cyclic GAN with residual feature aggregation (RFA) for RSSOD object detection, combining image super-resolution and object detection. Its RFA blocks aggregate residual features to improve super-resolved image quality, while a cyclic SRGAN-based design uses HR and LR generators with discriminators.

  • 4. Small-object Detection using Multiclass Cyclic GAN with RFA: The proposed Multiclass Cyclic GAN with residual feature aggregation targets object detection on the RSSOD dataset through image SR and object detection tasks.The network is explicitly based on these two tasks.
  • 4. Small-object Detection using Multiclass Cyclic GAN with RFA: RFA-based blocks replace conventional SRResNet and EDSR residual blocks, aggregating features from all residual blocks with a 1×1 convolution layer.The feature concatenation enhanced network performance and increased SR image quality.
  • 4. Small-object Detection using Multiclass Cyclic GAN with RFA: The SR component modifies SRGAN with RFA-based residual blocks and a Wasserstein GAN-based cyclic network trained using L1 and L2 loss functions.The proposed cyclic GAN architecture is presented in Figure 7.
  • 4. Small-object Detection using Multiclass Cyclic GAN with RFA: The cyclic model uses HRGEN to generate an HR image and LRGEN to generate the LR image from the HRGEN output, with separate discriminators for both generators.The HR generator and discriminator are shown in the top part of Figure 7, while the LR generator and discriminator appear in the bottom part.

L GEN LR HR MSE GEN GEN HR HR

The section presents a sequence of abbreviated labels, including GEN, LR, HR, and MSE.

  • The listed labels include GEN, LR, HR, and MSE.
  • The sequence also contains repeated I markers between LR and HR labels.
  • The passage does not provide an interpretable methodological or quantitative result beyond these abbreviations.

L GEN HR LR MSE GEN GEN LR LR

MCGR combines cyclic HR/LR GANs with residual feature aggregation and an auxiliary YOLOv5 detector, jointly optimizing generator, discriminator, and detection losses. The SR output is then used for object classification and localization, evaluated with PSNR, SSIM, and mAP@0.5 IoU.

  • MCGR architecture: MCGR uses paired HR and LR generators in a cyclic framework so generated LR images resemble actual LR images and generated HR images resemble actual HR images.The cyclic design evaluates the output generated by each GAN against the other to minimize overall loss.
  • MCGR architecture: The architecture contains generator, discriminator, and YOLOv5 detector networks whose losses are combined as a weighted total loss.The detector performs object class identification and localization using the SR image generated by the generator.
  • Implementation: The RFA-based generator uses 48 blocks with block size 64 × 64 and a 3 × 3 kernel, while the discriminator uses a gradient penalty coefficient λ of 10.The stated total-loss coefficients are µ1 = 0.90, µ2 = 10, and µ3 = 0.10.
  • Evaluation: mAP@0.5 IoU is used to evaluate object detection across classes with YOLOv5 as the auxiliary detector.Because most objects are small, inference uses low IoU and confidence thresholds.
  • Implementation: The network was implemented in PyTorch on Ubuntu 20.04 with an Nvidia Titan XP graphics processor and trained for 100 epochs.During training, the total-loss coefficients were 0.90, 10, and 0.10, respectively.

5. Benchmark Results of RSSOD

MCGR recovers high-frequency textures more effectively than competing super-resolution methods and achieves the strongest reported IQA results. Its detection performance remains strong across class settings, scale factors, and external aerial data, although inference is slower than YOLOv5.

  • Image super-resolution: MCGR recovers high-frequency texture details that competing methods either oversmooth or blend with neighboring pixels.Real-ESRGAN, SwinIR-L, and BSRGAN oversmooth, DRN blends neighboring information, while MCGR recovers detailed textures from low-resolution images.
  • Image super-resolution: 34.68 dB PSNR and 0.93 SSIM were achieved by MCGR, the best reported values, alongside the best MSE of 27.98.NLSN, EDSR, and DRN achieved PSNR values of 33.48 dB, 33.13 dB, and 32.69 dB, respectively.
  • RSSOD detection training: 0.758, 0.881, 0.841, and 0.983 were the training mAPs for five-class, four-class, two-class, and one-class detection, respectively.MCGR converged in under 25 epochs for one and two classes and reached stable mAP after 40 epochs for four classes; five-class learning was harder because low vegetation resembles trees.
  • RSSOD object detection: 0.731 and 0.711 were MCGR’s detection mAPs at scale factors 2 and 4, while YOLOv5 achieved 0.76 and MCGR 0.746 on HR-test images.Generic detectors deteriorated as scale factor increased; MCGR achieved these results with 16.61ms inference time per image, but its inference time was four times higher than YOLOv5.

6. Conclusion and Future Directions

The paper introduces the RSSOD benchmark and an RFA-based MCGR network for small-object remote-sensing detection and image super-resolution. Experiments show that super-resolution improves detection, while tree and low-vegetation recognition requires further exploration.

  • The RSSOD dataset benchmarks remote-sensing object detectors in complex settings with substantial class overlap and an emphasis on small objects.
  • The RFA-based MCGR network achieves state-of-the-art performance for image super-resolution and object detection.
  • Detection accuracy is satisfactory for vehicle, airplane, and ship classes, but tree and low-vegetation recognition requires further exploration of their complex features.
  • Image super-resolution before detection improves object-detection mAP, with MCGR outperforming YOLOv5 by 5% and 13% for scale factors of 2 and 4, respectively.
  • The independent-dataset results demonstrate MCGR's flexibility for object detection on other datasets.

CRediT Authorship Contribution Statement

The authorship statement assigns contributions spanning conceptualization, supervision, dataset annotation, experiments, MCGR development, analysis, and manuscript preparation. Contributions are distributed across individual authors and author groups.

  • Yi Wang handled conceptualization, funding acquisition, formal analysis, methodology, supervision, and final-version review and editing.
  • Syed Muhammad Arsalan Bashir led conceptualization, analysis, investigation, methodology, experiments, dataset annotation, MCGR development, training, validation, and manuscript preparation.
  • Mahrukh Khan contributed to conceptualization, state-of-the-art experiments, dataset collection and annotation, MCGR development, training, and validation.
  • Qudrat Ullah, Rui Wang, and Yilin Song collected and annotated the dataset and ran experiments using current image super-resolution methods.
  • Zhe Guo and Yilong Niu assisted with analysis and provided overall supervision.
Loading 2111.03260v1…