PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery

Jiaqing Zhang, Jie Lei, Weiying Xie, Zhenman Fang, Yunsong Li, Qian Du

arXiv:2209.13351v2cs.CV

TL;DR

Small remote-sensing objects are difficult to detect because they occupy few pixels amid broad, complex backgrounds, while high-accuracy solutions can be computationally heavy. SuperYOLO combines compact pixel-level multimodal fusion with training-time assisted super-resolution learning and removes the SR branch for inference. On VEDAI, it reaches 75.09% mAP50 and reports a favorable accuracy-speed trade-off with substantially fewer resources than YOLOv5x.

  • Problem

    Small, multiscale remote-sensing objects are difficult to distinguish from broad backgrounds, while existing accurate solutions can impose heavy computation.

  • Method

    SuperYOLO uses compact symmetric pixel-level multimodal fusion and an assisted SR branch that learns high-resolution features during training and is removed during inference.

  • Results

    75.09% mAP50 is reported on the VEDAI RS dataset, with SuperYOLO showing a favorable accuracy-speed trade-off against state-of-the-art models.

  • Takeaways & Limitations

    The proposed model provides real-time multimodal remote-sensing detection with improved small-object performance while avoiding inference computation from the SR branch.

Abstract

from arXiv · show

Accurately and timely detecting multiscale small objects that contain tens of pixels from remote sensing images (RSI) remains challenging. Most of the existing solutions primarily design complex deep neural networks to learn strong feature representations for objects separated from the background, which often results in a heavy computation burden. In this article, we propose an accurate yet fast object detection method for RSI, named SuperYOLO, which fuses multimodal data and performs high-resolution (HR) object detection on multiscale objects by utilizing the assisted super resolution (SR) learning and considering both the detection accuracy and computation cost. First, we utilize a symmetric compact multimodal fusion (MF) to extract supplementary information from various data for improving small object detection in RSI. Furthermore, we design a simple and flexible SR branch to learn HR feature representations that can discriminate small objects from vast backgrounds with low-resolution (LR) input, thus further improving the detection accuracy. Moreover, to avoid introducing additional computation, the SR branch is discarded in the inference stage, and the computation of the network model is reduced due to the LR input. Experimental results show that, on the widely used VEDAI RS dataset, SuperYOLO achieves an accuracy of 75.09% (in terms of mAP50 ), which is more than 10% higher than the SOTA large models, such as YOLOv5l, YOLOv5x, and RS designed YOLOrs. Meanwhile, the parameter size and GFLOPs of SuperYOLO are about 18 times and 3.8 times less than YOLOv5x. Our proposed model shows a favorable accuracy and speed tradeoff compared to the state-of-the-art models. The code will be open-sourced at https://github.com/icey-zhang/SuperYOLO.

I. INTRODUCTION

Remote sensing detection must handle small, diverse objects and limited labeled data while balancing accuracy with real-time computation. SuperYOLO addresses these challenges through resolution preservation, pixel-level multimodal fusion, and an auxiliary super-resolution branch removed during inference.

  • Remote sensing images contain relatively few labeled samples, objects spanning tens of pixels, broad backgrounds, and diverse object scales.
  • Single-modality detection lacks complementary information, motivating fusion of RGB and infrared imagery for improved recognition.
  • SuperYOLO removes YOLOv5s’s Focus module to preserve spatial resolution and better localize small dense objects.
  • The assisted SR branch learns high-resolution features during training, then is removed during inference to avoid additional computation.
  • Its symmetric compact pixel-level fusion combines modalities bidirectionally while reducing computation relative to feature-level fusion without sacrificing accuracy.
  • SuperYOLO reports improved real-time multimodal detection and a favorable accuracy-speed trade-off against state-of-the-art detectors.

II. RELATED WORK

Related work combines remote-sensing modalities to supply complementary information and uses pixel-level fusion to reduce computation. SuperYOLO extends this direction with an assisted SR branch that guides high-resolution representation learning during detection training.

  • Multimodal remote-sensing data include RGB, SAR, LiDAR, IR, PAN, and multispectral imagery with complementary characteristics for task improvement.
  • Infrared imagery can improve detection under difficult weather conditions by capturing longer thermal wavelengths.
  • Decision-level fusion may repeat calculations across modality branches, while feature-level fusion commonly uses multiple branches and can raise computation.
  • SuperYOLO uses pixel-level fusion to reduce unnecessary computation while extracting information across spatial and channel domains.

B. Super Resolution in Object Detection

Small-object detection methods often lose high-resolution texture and pattern information through downsampling or upsampling. SuperYOLO retains such information with a lightweight backbone design and an assisted SR pathway that is used during training but removed at inference.

  • Multi-scale and context-based methods improve representation across scales but may overlook preservation of high-resolution contextual information.
  • The YOLOv5s backbone uses CSP, CBS, and SPP structures to extract low-level texture and high-level semantic features for multiscale detection.
  • YOLO backbones halve feature dimensions through stride-2 convolutions; with 608-pixel input, the last detection layers have sizes 76, 38, and 19.
  • SuperYOLO removes the Focus module, fuses RGB and IR at pixel level, and adds an assisted SR module to guide backbone learning toward high-resolution features.
  • The pixel-level MF module is the architectural mechanism used to fuse multimodal inputs after Focus removal.

B. Multimodal Fusion

SuperYOLO uses pixel-level multimodal fusion to combine RGB and IR information while limiting computation. The fusion extracts complementary channel and spatial information before producing backbone features.

  • Pixel-level fusion is selected because decision-level fusion can consume substantial computation resources.
  • The multimodal fusion extracts shared and modality-specific information through a symmetric, compact, bidirectional design.
  • Channel-domain processing uses SE blocks to extract internal information from the RGB and IR modalities.
  • The method also models inner spatial information between the different modalities using 1 × 1 convolutions and element-wise multiplication.
  • The resulting features incorporate modality inputs and are fused before being passed to the backbone; inputs are subsampled to 1/n size to accelerate training.

ReLU

The SR structure is described as a simple Encode-Decoder that combines backbone features, while downsampling is performed with bilinear interpolation.

  • The SR structure can be regarded as a simple Encode-Decoder model.
  • Low-level and high-level backbone features are selected to combine local textures and semantic information, respectively.
  • The operation D(·) represents n times downsampling using bilinear interpolation.

C. Super Resolution

SuperYOLO adds an auxiliary SR branch to preserve high-resolution information for small-object detection without retaining that branch during inference. The branch combines multilevel backbone features and guides spatial learning in the main detector.

  • Feature upsampling alone can lose texture and pattern information needed to detect small targets requiring high-resolution preservation.
  • The auxiliary SR branch extracts high-resolution information during training while avoiding additional inference computation after removal at inference.
  • The SR module uses low-level and high-level backbone features to fuse local textures, patterns, and semantic information.
  • The decoder uses three deconvolutional layers, and SR transfers spatial-dimension learning to the main branch.
  • SuperYOLO produces clearer object structures with higher-resolution features than YOLOv5s and YOLOv5x in the feature visualization.
  • The overall network loss combines detection loss with SR construction loss, using L1 loss between input image X and SR result S.

V. EXPERIMENTAL RESULTS

The experiments use VEDAI multimodal RGB and IR imagery with ten-fold cross-validation and evaluate detection using recall, precision, and mAP, alongside complexity and SR image-quality metrics.

  • Dataset and protocol: VEDAI contains 1246 cropped images from AGRC, with RGB and IR modalities collected for the same scenes.AGRC images are approximately 16,000 × 16,000 pixels at about 12.5 cm × 12.5 cm per pixel.
  • Dataset and protocol: Ten-fold cross-validation uses 1089 images for training and 121 images for testing in each split.Ablations use the first fold, while comparisons with previous methods average results across all ten folds.
  • Evaluation metrics: The evaluation reports recall, precision, and mAP to measure detection agreement with reference masks.Precision and recall are linked to commission and omission errors, while mAP averages AP values across categories.
  • Evaluation metrics: mAP integrates the Precision–Recall curve area and averages AP values across the N categories.In the supplied formulation, p denotes Precision, r denotes Recall, and N is the number of categories.
  • Complexity and SR metrics: GFLOPs and parameter size measure model complexity and computation cost, while PSNR and SSIM assess SR image quality.Higher PSNR and SSIM values indicate better generated-image quality.

D. Ablation Study

The study begins with ablation experiments on the first fold of the validation set to assess the proposed method’s components.

  • Ablation protocol: Ablation experiments evaluating the proposed method are conducted on the first fold of the validation set.The experiments are introduced as the initial verification of the method’s effectiveness.

1) Validation of the Baseline Framework:

Removing Focus improves small-object detection by preserving input resolution, while pixel-level RGB–IR fusion provides the strongest reported fusion result with lower computation than deeper multistage fusion.

  • Impact of Removing Focus Module: YOLOv5s mAP50 rises from 62.2% to 69.5% after removing Focus, while YOLOv5m, YOLOv5l, and YOLOv5x also improve.The corresponding changes are 64.5%→72.2%, 63.7%→72.5%, and 64.0%→69.2%.
  • Impact of Removing Focus Module: Removing Focus increases YOLOv5s computation from 5.3 to 20.4 GFLOPs, but remains below YOLOv3, YOLOv4, and YOLOrs.The reported comparison is 20.4 GFLOPs versus 52.8, 38.2, and 46.4 GFLOPs, respectively.
  • Comparison of Different Fusion Methods: Pixel-level fusion with the MF module achieves 7.0897M parameters, 21.67 GFLOPs, and 70.3% mAP50, the best among compared fusion methods.The corresponding concatenation fusion result is 7.0705M parameters, 20.37 GFLOPs, and 69.5% mAP50.
  • Comparison of Different Fusion Methods: Multistage feature-level fusion reaches 59.3% mAP50 with 34.56 GFLOPs and 7.7545M parameters, lower accuracy and higher computation than pixel-level fusion.Figure 7 distinguishes feature-level fusion from multistage feature-level fusion by their fusion stages.

4) Impact of High Resolution:

Higher-resolution inputs improve detection but increase computation; assisted SR recovers near-HR accuracy from LR inputs while removing SR at inference and preserving LR computation.

  • Impact of High Resolution: Doubling YOLOv5s input size from 512 to 1024 raises mAP50 from 62.2% to 77.7%, a 15.5% increase.YOLOv5s-noFocus similarly improves from 69.5% to 79.3% mAP50.
  • Impact of High Resolution: High-resolution inputs increase computation: YOLOv5s rises from 5.3 to 21.3 GFLOPs, and YOLOv5s-noFocus from 20.4 to 81.5 GFLOPs.Mean recall and mean precision also increase with resolution in the reported comparisons.
  • Impact of High Resolution: Mismatched training and testing resolutions reduce mAP50, with reported paired scores including 10.6% versus 62.2% and 48.2% versus 77.7%.The authors attribute the reduction to inconsistent object scales between training and testing.
  • Impact of Super Resolution Branch: The SR network improves mAP50 by 1.8% over upsampling and supports feature extraction for detection.The ablation study reports SR as a learnable upsampling method with stronger reconstruction ability.
  • Impact of Super Resolution Branch: Using one detector reduces parameters from 7.0705M to 4.8259M and GFLOPs from 20.37 to 16.68 while increasing detection accuracy from 78.0% to 79.0%.The reported rationale is that VEDAI objects are small and can be detected with a small-scale detector.
  • Impact of Super Resolution Branch: Adding SR improves mAP50 by 9.2% for YOLOv3, 3.3% for YOLOv4, and 2.2% for YOLOv5s.The SR branch can be removed during inference, so it adds no inference parameters or computation.

E. Comparisons with Previous Methods

SuperYOLO improves multimodal remote-sensing detection over several YOLO-based methods, especially for small and easily confused objects, while maintaining a favorable accuracy–speed trade-off. It also generalizes evaluation across additional single-modality aerial-image datasets.

  • VEDAI visual comparison: SuperYOLO detects objects missed, misclassified, or uncertain in YOLOv4, YOLOv5s, and YOLOv5m, including confusing Pickup–Car and Van–Boat cases.The visual comparison uses red markers for false alarms, yellow markers for false positives, and blue markers for false negatives.
  • VEDAI quantitative comparison: Multimodal configurations generally achieve higher class-wise AP and overall mAP50 than RGB-only or IR-only configurations.The authors attribute this pattern to complementary information between modalities.
  • VEDAI quantitative comparison: 12.44% mAP50 is SuperYOLO’s improvement over YOLOv5x in multimodal mode, while its parameter size and GFLOPs are about 18x and 3.8x lower.YOLOFusion performs slightly better but uses pretrained MS COCO weights and approximately three times as many parameters.
  • VEDAI quantitative comparison: 18.30% mAP50 is SuperYOLO’s improvement over YOLOv5s, whose Focus module reduces GFLOPs but produces especially poor small-object detection.The reported comparison supports the claimed speed–accuracy trade-off against state-of-the-art models.
  • Additional datasets: SuperYOLO is also compared with one-stage, two-stage, lightweight, and distillation-based methods on DOTA, DIOR, and NWPU VHR-10.These experiments address evaluation beyond the multimodal VEDAI setting using single-modality data.

2) NWPU VHR-10:

The NWPU VHR-10 evaluation uses a defined train–test split and fixed image size within a broader comparison across remote-sensing detection methods. SuperYOLO is reported as achieving optimal detection results with substantially smaller computational requirements.

  • NWPU VHR-10 dataset: NWPU VHR-10 contains 800 images, including 650 images with objects, with 520 used for training and 130 for testing across 10 categories.Images are resized to 512 × 512 pixels.
  • Experimental setup: The evaluation trains NWPU VHR-10 models for 150 epochs with batch size 8 and compares SuperYOLO against 11 generic detection methods.The comparison includes one-stage, two-stage, lightweight, and distillation-based methods.
  • Results: 69.99%, 93.30%, and 71.82% mAP50 are the reported optimal detection results for SuperYOLO across the evaluated datasets.The corresponding model parameters are 7.70 M, 7.68 M, and 7.70 M, with GFLOPs of 20.89, 20.86, and 20.93.
  • Results: SuperYOLO’s PANet structure and three detectors are described as supporting small-, middle-, and large-scale target detection across the evaluated datasets.The design accounts for large objects such as playgrounds.
  • Conclusion: The paper concludes that SuperYOLO improves small-object RSI detection using a modified YOLOv5s baseline, multimodal fusion, and an SR branch removed during inference.On VEDAI, it reports 75.09% mAP50, 18.30% higher than YOLOv5s and more than 12.44% higher than YOLOv5x.
Loading 2209.13351v2…