PaperlayerView exact revision on arXiv

Source-linked AI summary

Small-Object Detection in Remote Sensing Images with End-to-End Edge-Enhanced GAN and Object Detector Network

Jakaria Rabbi, Nilanjan Ray, Matthias Schubert, Subir Chowdhury, Dennis Chao

arXiv:2003.09085v5cs.CVcs.LG

TL;DR

Small-object detection from noisy and low-resolution remote-sensing imagery is limited by weak features and missing high-frequency edges, while high-resolution imagery is costly. The paper introduces an edge-enhanced super-resolution GAN jointly trained with object detectors, and reports better detection than standalone state-of-the-art detectors on two datasets. The best reported configuration uses EESRGAN with Faster R-CNN, although SSD offers a faster lower-accuracy alternative.

  • Problem

    Small objects remain difficult to detect in noisy and low-resolution remote-sensing imagery, and high-resolution imagery is costly for large areas with frequent updates.

  • Method

    The paper jointly trains an EESRGAN-based super-resolution network with FRCNN or SSD, backpropagating detector loss into image generation while enhancing edge information.

  • Results

    The method outperformed standalone state-of-the-art detectors on the COWC and OGST datasets; EESRGAN with Faster R-CNN achieved the best results.

  • Takeaways & Limitations

    End-to-end edge-enhanced super-resolution can improve small-object detection from low-resolution satellite imagery within the evaluated datasets.

  • Takeaways & Limitations

    The OGST data came from one region and season, and the low-resolution images were artificially generated rather than true low-resolution observations.

Abstract

from arXiv · show

The detection performance of small objects in remote sensing images is not satisfactory compared to large objects, especially in low-resolution and noisy images. A generative adversarial network (GAN)-based model called enhanced super-resolution GAN (ESRGAN) shows remarkable image enhancement performance, but reconstructed images miss high-frequency edge information. Therefore, object detection performance degrades for small objects on recovered noisy and low-resolution remote sensing images. Inspired by the success of edge enhanced GAN (EEGAN) and ESRGAN, we apply a new edge-enhanced super-resolution GAN (EESRGAN) to improve the image quality of remote sensing images and use different detector networks in an end-to-end manner where detector loss is backpropagated into the EESRGAN to improve the detection performance. We propose an architecture with three components: ESRGAN, Edge Enhancement Network (EEN), and Detection network. We use residual-in-residual dense blocks (RRDB) for both the ESRGAN and EEN, and for the detector network, we use the faster region-based convolutional network (FRCNN) (two-stage detector) and single-shot multi-box detector (SSD) (one stage detector). Extensive experiments on a public (car overhead with context) and a self-assembled (oil and gas storage tank) satellite dataset show superior performance of our method compared to the standalone state-of-the-art object detectors.

1. Introduction

Small-object detection in remote-sensing imagery remains difficult because low resolution, noise, and limited edge detail reduce useful input features, while high-resolution imagery is costly. The paper proposes jointly enhancing imagery and detecting objects with an edge-enhanced super-resolution network.

  • Problem Description and Motivation: Small objects are harder to detect than large objects, particularly in noisy and low-resolution remote-sensing images.The difficulty also persists in high-resolution imagery, where small objects perform worse than large ones.
  • Problem Description and Motivation: Satellite imagery often lacks sufficient small-object features, while frequent high-resolution coverage over large areas is expensive.Public Landsat-8 and Sentinel-2 imagery has high ground sampling distances, whereas commercial imagery is costly and may still be less accurate than higher-resolution data.
  • Problem Description and Motivation: Super-resolution methods can generate realistic high-resolution imagery, but reconstructed edges may be inconsistent with ground truth and harm downstream recognition.Because edge information is important for object detection, enhanced images need to preserve it accurately.
  • Contributions of Our Method: The proposed end-to-end architecture jointly performs super-resolution and object detection, addressing a gap in combining edge enhancement with detector optimization.Detector loss is backpropagated into the super-resolution network so the two tasks are optimized together.
  • Contributions of Our Method: The system combines EESRGAN components with FRCNN or SSD detectors and uses RRDB-based generator and edge-enhancement subnetworks.The architecture includes generator, discriminator, and edge-enhancement components, with detector networks operating on generated super-resolution images.
  • Contributions of Our Method: Experiments on COWC and OGST show that the method outperformed standalone state-of-the-art detectors when only low-resolution images were required for testing.The OGST dataset was self-assembled for oil and gas storage-tank detection, while COWC provided a public vehicle-detection benchmark.

2. Related Works

Prior work spans CNN- and GAN-based super-resolution, two-stage and one-stage object detectors, and specialized small-object methods. However, many approaches target natural imagery or use separate enhancement and detection stages, motivating joint super-resolution and detection for remote sensing.

  • Super-Resolution: CNN-based super-resolution evolved from SRCNN and VDSR toward residual, densely connected, and GAN-based architectures.These methods generate higher-resolution imagery from low-resolution inputs, commonly using paired high- and low-resolution data.
  • Super-Resolution: EEGAN introduced an edge-enhancement subnetwork for smoother edge details in GAN-generated satellite imagery.Its design addressed the need to preserve edge information during image enhancement.
  • Object Detection: Object detectors are commonly divided into two-stage R-CNN models and one-stage detectors such as SSD, YOLO, and RetinaNet.Two-stage detectors first determine regions of interest, whereas one-stage detectors perform detection directly in a single stage.
  • Object Detection: Remote-sensing detection research has used specialized CNNs, region proposals, hard-negative mining, tiling, and transfer learning for vehicles, plants, aircraft, and other small objects.These methods adapt detector architectures or input processing to object scale, orientation, or large-image constraints.
  • Small-Object Detection: Feature pyramid, receptive-field, and refinement modules improved small-object detection in general object-detection settings.Examples include feature pyramid networks for Faster R-CNN, receptive-field blocks for SSD, and RefineDet's anchor refinement module.
  • Joint Super-Resolution and Detection: Prior joint super-resolution and detection studies included natural-image methods and satellite-image approaches, but the paper targets edge-enhanced super-resolution with end-to-end detector training.The proposed setup requires high- and low-resolution image pairs during training.

3. Method

The method is an end-to-end pipeline combining a GAN-based super-resolution network, an edge-enhancement network, and an object detector. Detection gradients influence image generation, while enhanced edges are incorporated into the final super-resolution output before detection.

  • Method: The architecture contains a GAN-based super-resolution network and a detector network trained jointly from paired high- and low-resolution images.The detector loss is propagated into the generator during end-to-end training.
  • Method: The super-resolution network comprises generator G, discriminator DRa, and edge-enhancement network EEN components.The detector is treated as part of the discriminator side of the overall architecture.
  • Method: Figure 2 summarizes the overall generator-and-discriminator organization of the network.The detector functions within the discriminator-side training pathway through backpropagated detection loss.
  • Method: Generator G produces intermediate super-resolution images, while DRa distinguishes ground-truth high-resolution images from intermediate outputs.Inverted discriminator gradients are backpropagated into G to support images suitable for accurate detection.
  • Method: The final SR images are formed by adding enhanced edges to intermediate SR images before object detection.The method uses image and edge consistency losses, plus perceptual loss from VGG19 feature extraction.

3.1. Generator

The generator module combines an ESRGAN-style generator with an edge-enhancement network to reconstruct super-resolution images while preserving useful edge information.

  • Generator Network G: The generator uses ESRGAN’s architecture, removes batch-normalization layers, and builds G from RRDB blocks with convolutional and upsampling blocks.The paper links batch-normalization removal to reduced computational complexity and fewer artifacts when training and testing statistics differ.
  • Generator Network G: RRDBs use multi-level residual connections and dense connections to increase network capacity, while residual scaling helps stabilize training.PReLU is used in the dense blocks.
  • Generator Network G: The generator is trained with adversarial, perceptual, and content losses, with content loss measuring the 1-norm distance between generated and high-resolution images.Perceptual loss uses feature maps before activation layers in a fine-tuned VGG19 network.
  • Edge-Enhancement Network EEN: The EEN extracts edges with a Laplacian operator, processes them through convolutional, RRDB, and upsampling blocks, and adds enhanced edges back to the input.A sigmoid mask branch removes edge noise before the enhanced edges are combined with the edge-subtracted input.
  • Edge-Enhancement Network EEN: Compared with the prior edge-enhancement subnetwork, EEN replaces dense blocks with RRDBs and introduces a new loss term for edge reconstruction.Its consistency losses compare both images and extracted edges using the Charbonnier penalty.

3.2. Discriminator

The discriminator module combines a relativistic discriminator with Faster R-CNN or SSD detector networks, whose losses contribute to training the generator module.

  • Discriminator: The architecture uses an ESRGAN-style relativistic discriminator together with Faster R-CNN and SSD detector networks.The discriminator and detector jointly act as the discriminator for the generator module.
  • Faster R-CNN: Faster R-CNN is a two-stage detector whose region proposal network generates candidate regions before classification and bounding-box regression.The implementation uses ResNet-50-FPN as its backbone.
  • Faster R-CNN: Faster R-CNN uses classification and smooth L1 regression losses, with its classifier distinguishing background from the single object class in these datasets.The regression branch fits accurate bounding boxes around detected objects.
  • SSD: SSD performs classification and localization in one forward pass and predicts objects across convolutional feature layers with decreasing spatial sizes.VGG-16 is used as the feature extractor to support SSD’s speed objective.
  • SSD: SSD combines classification and regression losses, with smooth L1 used for localization and a separate classification-loss formulation.The loss-balancing parameter α is set to 1 empirically.

3.3. Training

The architecture supports separate or end-to-end training; end-to-end training propagates detector loss into the generator so it receives detector and discriminator feedback.

  • Training: The architecture can be trained either in separate stages or jointly in an end-to-end manner.The distinction is whether detector loss is backpropagated into the generator module.
  • Separate Training: In separate training, the detector is trained after the super-resolution network converges, and detector loss is prevented from updating the generator.The generator therefore receives feedback only from the relativistic discriminator.
  • End-to-End Training: In end-to-end training, detector loss is backpropagated into the generator, which receives gradients from both the detector and relativistic discriminator.The final discriminator loss combines relativistic discriminator loss with detector loss.

4. Experiments

Experiments train the separate and end-to-end variants with staged initialization and joint optimization, using specified optimization settings and hardware for the COWC evaluation.

  • Training Procedure: Separate training first optimizes the super-resolution network to convergence, then trains detectors on the resulting super-resolution images.End-to-end training uses the separately trained weights for initialization before propagating detector gradients into the generator.
  • Implementation Details: Training uses Adam with learning rate 0.0001, batch size 5, β1 = 0.9, β2 = 0.999, and learning-rate halving every 50k iterations.The generator uses 23 RRDB blocks and EEN uses 5 RRDB blocks, implemented in PyTorch on two NVIDIA Titan X GPUs.
  • Implementation Details: End-to-end training on COWC took 96 hours for 200

4.1. Datasets

The experiments use two single-class satellite datasets: COWC for cars and OGST for oil and gas storage tanks, with paired HR imagery and synthetically downscaled LR inputs.

  • COWC dataset: COWC provides 15 cm RGB satellite imagery from Toronto and Potsdam, with 12,651 cars in the selected regions.
  • COWC dataset: COWC experiments use 256-by-256 tiles, 4x bicubic downsampling to 64-by-64 LR images, and cars occupying 240 to 960 pixels.
  • COWC dataset: The COWC study includes one car class across 3,340 tiles, with 80%/20% train/test splitting and augmentation by flips and ninety-degree rotations.
  • OGST dataset: OGST contains 30 cm RGB imagery from Alberta’s Cold Lake Oil Sands region and 1,671 oil and gas storage tanks.
  • OGST dataset: OGST experiments use 512-by-512 tiles, 4x bicubic downsampling to 128-by-128 LR images, and bounding-box coordinate files for tanks.
  • OGST dataset: The OGST study uses one tank class across 760 tiles, with 90%/10% train/test and train/validation splits.

4.2. Evaluation Metrics for Detection

Detection is evaluated from predicted bounding boxes using precision, recall, IoU, and average precision, with AP computed across IoU thresholds from 0.5 to 0.95.

  • Average precision is the primary metric, calculated from IoU, precision, and recall for the single-class datasets.
  • Precision measures the proportion of predicted objects that are true positives among all predicted objects.
  • Recall measures the proportion of dataset objects detected as true positives, with undetected objects counted as false negatives.
  • IoU measures overlap between detected and ground-truth boxes, while detections meeting IoU ≥τ are treated as true positives at threshold τ.
  • AP at IoU=0.5:0.95 combines ten precision values obtained by varying IoU from 0.5 to 0.95 in steps of 0.05.

4.3. Results

Across two satellite datasets, super-resolution improves small-object detection over low-resolution baselines, and end-to-end EESRGAN training performs better than separate training while approaching HR detection performance.

  • Detection without Super-Resolution: 64% AP was obtained for Faster R-CNN when training and testing on LR images, while HR training and testing reached up to 98% AP.
  • Separate Training with Super-Resolution: EESRGAN produced the best detection results among EESRGAN, EEGAN, and ESRGAN, approaching HR-only detection rates and remaining effective when applied directly to LR imagery.
  • End-to-End Training: End-to-end training backpropagates detector loss into the super-resolution network so enhancement is optimized using detection feedback.
  • End-to-End Training: The proposed end-to-end architecture delivered superior results compared with ESRGAN and EEGAN combined with detectors.
  • IoU Analysis: End-to-end EESRGAN-FRCNN outperformed separately trained EESRGAN-FRCNN, especially at higher IoUs on COWC.
  • Training Dataset Size: With the full training sets, end-to-end AP reached 95.5% for COWC and 83.2% for OGST; reducing training data lowered AP, especially below half-scale training.
  • Edge Consistency Loss: Adding edge consistency loss produced noise-free enhanced edges, whereas using only image consistency loss yielded noisy edges, blurrier SR images, and lower detection accuracy.

5. Discussion

The end-to-end SR-detector network improved detection accuracy, with EESRGAN outperforming other tested SR methods and faster R-CNN achieving the best detector result. Performance varied by dataset, training-data diversity, and detector speed.

  • Detection performance: EESRGAN combined with object detectors outperformed EEGAN and ESRGAN, while edge enhancement improved detection accuracy.The AP improvement was larger at higher IoU values.
  • Dataset differences: The OGST dataset showed smaller AP improvements than COWC because tanks were larger and less diverse in size and color than cars.
  • Training data: AP values improved slightly with more training data, while reduced training data produced similar accuracy on both datasets.
  • Detector trade-offs: Faster R-CNN achieved the best result but required more time than SSD.
  • Detector trade-offs: SSD was presented as the practical choice for broad-area detection when sacrificing some accuracy is acceptable.
  • Limitations: OGST robustness remained limited because its data came from one area and season, requiring more diverse regions and seasons.
  • Limitations: The experiments covered single-class datasets with limited variation, motivating evaluation across more object types and landscapes.
  • Limitations: Training used artificially generated LR-HR pairs because suitable public satellite data with real pairs and detection boxes were unavailable.The authors propose creating more accurate LR images in future work.

6. Conclusions

The paper presents an end-to-end architecture that converts LR satellite imagery into object-detection results. Its SR network with faster R-CNN performed best, while broader training data and more realistic LR images remain future needs.

  • The proposed architecture combines an SR network and detector network to produce object-detection results from LR satellite imagery.
  • The proposed SR network with faster R-CNN yielded the best results for small objects on satellite imagery.
  • More diverse OGST training data are needed to make oil-and-gas storage-tank detection more robust.
  • Future work should explore diverse datasets and techniques for creating more realistic LR images.
  • The method combines multiple strategies to improve small-object detection on LR imagery.
Loading 2003.09085v5…