PaperlayerView exact revision on arXiv

Source-linked AI summary

MTU-Net: Multi-level TransUNet for Space-based Infrared Tiny Ship Detection

Tianhao Wu, Boyang Li, Yihang Luo, Yingqian Wang, Chao Xiao, Ting Liu, Jungang Yang, Wei An, Yulan Guo

arXiv:2209.13756v1cs.CV

TL;DR

Space-based infrared tiny ship detection must handle extremely small, dim, and variable targets in very large, challenging scenes where existing SIRST methods are not well suited. The paper develops NUDT-SIRST-Sea and MTU-Net, combining multi-level Transformer-CNN features with CRRP augmentation and FocalIoU loss; on this dataset, MTU-Net outperforms traditional and deep-learning SIRST methods across Pd, Fa, and IoU.

  • Problem

    Space-based infrared images contain extremely small, dim, and variable targets, while existing short-distance infrared datasets and methods are not well suited to this surveillance task.

  • Method

    The paper develops NUDT-SIRST-Sea and MTU-Net with a multi-level ViT-CNN encoder, CRRP augmentation, and FocalIoU loss.

  • Results

    64.14% IoU, 85.44% Pd, and 11.72 × 10^-6 Fa are reported for MTU-Net on NUDT-SIRST-Sea.

  • Takeaways & Limitations

    MTU-Net outperforms traditional and existing deep-learning SIRST methods on Pd, Fa, and IoU for space-based infrared tiny ship detection.

Abstract

from arXiv · show

Space-based infrared tiny ship detection aims at separating tiny ships from the images captured by earth orbiting satellites. Due to the extremely large image coverage area (e.g., thousands square kilometers), candidate targets in these images are much smaller, dimer, more changeable than those targets observed by aerial-based and land-based imaging devices. Existing short imaging distance-based infrared datasets and target detection methods cannot be well adopted to the space-based surveillance task. To address these problems, we develop a space-based infrared tiny ship detection dataset (namely, NUDT-SIRST-Sea) with 48 space-based infrared images and 17598 pixel-level tiny ship annotations. Each image covers about 10000 square kilometers of area with 10000X10000 pixels. Considering the extreme characteristics (e.g., small, dim, changeable) of those tiny ships in such challenging scenes, we propose a multi-level TransUNet (MTU-Net) in this paper. Specifically, we design a Vision Transformer (ViT) Convolutional Neural Network (CNN) hybrid encoder to extract multi-level features. Local feature maps are first extracted by several convolution layers and then fed into the multi-level feature extraction module (MVTM) to capture long-distance dependency. We further propose a copy-rotate-resize-paste (CRRP) data augmentation approach to accelerate the training phase, which effectively alleviates the issue of sample imbalance between targets and background. Besides, we design a FocalIoU loss to achieve both target localization and shape description. Experimental results on the NUDT-SIRST-Sea dataset show that our MTU-Net outperforms traditional and existing deep learning based SIRST methods in terms of probability of detection, false alarm rate and intersection over union.

I. INTRODUCTION

Space-based infrared tiny ship detection involves smaller, dimmer, and more variable targets than conventional land- and aerial-based SIRST settings. The paper introduces NUDT-SIRST-Sea and MTU-Net to address these challenges through contextual feature modeling, augmentation, and specialized loss design.

  • Motivation: Space-based infrared images contain larger, more complex scenes and smaller, dimmer, more variable targets than land- and aerial-based SIRST images.Existing methods cannot be well adopted to these space-based challenges.
  • Dataset: NUDT-SIRST-Sea provides 17598 high-precision bounding boxes and pixel-level annotations for space-based infrared tiny ship detection.The dataset is presented as the largest manually annotated space-based infrared dataset described by the authors.
  • Method: MTU-Net combines a Transformer-CNN hybrid encoder with multi-level feature extraction and fusion to capture long-distance dependency.CNNs first extract multi-level features, which are refined by the multi-level ViT module before fusion.
  • Method: CRRP augmentation increases candidate-target frequency during training, while FocalIoU loss jointly supports target localization and shape description.These components address foreground-background imbalance and improve the training objective.
  • Results: MTU-Net achieves state-of-the-art probability of detection, false alarm rate, and intersection over union on NUDT-SIRST-Sea.The reported evaluation covers both traditional and existing deep-learning-based SIRST methods.

II. CHALLENGES

NUDT-SIRST-Sea is designed around unusually large images, complex backgrounds, confusing distractors, and highly variable ship scales. These properties make space-based infrared tiny ship detection computationally difficult and challenge contextual understanding and multi-scale detection.

  • Target Statistics: The dataset has a target-to-background ratio of 0.000029%, indicating extreme foreground-background imbalance.The paper contrasts this ratio with those of other SIRST datasets.

B. Statistical Properties of NUDT-SIRST-Sea

NUDT-SIRST-Sea contains space-based infrared scenes with unusually large coverage, complex backgrounds, diverse confusers, tiny and dim targets, and substantial scale variation.

  • Much Larger Image Size: Each 10000 × 10000-pixel image covers about 10000 square kilometers, includes varied scenes, and creates higher computational difficulty.
  • Much More Complex Background: Clouds, ports, land, sea, and urban inland rivers form complex backgrounds that challenge detection.
  • Multi-Type Suspicious Targets: Tiny clouds, port containers, reefs, and land bright spots resemble ships in shape and brightness, causing false alarms.
  • Much Smaller Targets: The average target size is 29 pixels, while the target-to-background ratio is 0.000029%.
  • Much Dimmer Targets: More than 20% of targets have brightness below 0.5, whereas other compared datasets contain fewer than 5%.
  • Multi-Scale Targets: Ship sizes range from 2 to 500 pixels, so multiple target scales often occur within the same scene.

B. Single-frame Infrared Small Target Detection

Traditional and CNN-based SIRST methods have progressed, but space-based infrared scenes demand stronger handling of tiny-target detail, complex confusers, and long-distance dependencies.

  • Traditional Methods: Handcrafted traditional methods use fixed features and hyperparameters, with detection degrading as backgrounds change.
  • CNN-Based Methods: CNN-based methods learn target characteristics from data and have achieved promising results on land-based and aerial-based SIRST tasks.
  • Space-Based Challenge: Space-based images contain smaller, dimmer, more changeable targets, while limited CNN receptive fields hinder high-performance detection.
  • Vision Transformers: Transformer-based methods include DETR, TransUNet, and ViT-based SIRST detection, but their original settings do not target space-based SIRST.
  • Proposed Direction: MTU-Net combines multi-level ViT and CNN features because space-based detection requires localization information and low-level shape description.

B. Multi-level ViT CNN Hybrid Encoder

The hybrid encoder first extracts multi-scale local CNN features, then applies multi-level ViT branches to refine them and capture long-distance dependencies.

  • CNN Feature Extraction: Images are cut into 1024 × 1024 patches before ResNet-18 extracts multi-scale local features for efficient processing of very large images.
  • Multi-Level Processing: MVTM uses k − 1 ViT branches with a shared structure to process multi-level feature representations.
  • Token Refinement: Feature maps are flattened into token sequences, position-embedded, and passed through multi-head self-attention and MLP processing.
  • Token Refinement: Each ViT branch reshapes its processed tokens into feature maps V_i with spatial dimensions H_k × W_k.

3) Multi-level Feature Fusion Module:

The multi-level feature fusion module concatenates ViT-refined features with a CNN feature and uses a 1 × 1 convolution to produce fused long-distance information; CRRP augments sparse targets.

  • Multi-level Feature Fusion Module: MFFM concatenates V_i features with CNN feature F_k and applies a 1 × 1 convolution to generate M_k.
  • Multi-level Feature Fusion Module: The fused representation incorporates long-distance dependencies from extracted high-level features before decoder processing.
  • Copy-Rotate-Resize-Paste Augmentation: CRRP collects target-neighborhood images, randomly copies, rotates, resizes, and pastes targets into background regions.

C. U-shape Decoder

The U-shaped decoder upsamples multi-level features, combines them with encoder features through skip connections, and produces confidence maps before clustering connected target pixels. CRRP augments sparse targets with rotated, resized targets and neighborhood context to reduce imbalance and accelerate training.

  • C. U-shape Decoder: The decoder upsamples multi-level feature Mk and concatenates encoder features through skip connections to generate decoder features Mi.
  • C. U-shape Decoder: An eight-connected neighborhood module groups adjacent pixels with equal values into target areas and calculates each target’s centroid.
  • E. Data Augmentation: CRRP copies target neighborhoods, randomly rotates and resizes targets, then pastes them into background regions during training.
  • E. Data Augmentation: CRRP relieves foreground-background imbalance and greatly reduces training time compared with simple augmentation methods such as rotation, translation, and color jitter.

F. FocalIoU Loss

FocalIoU combines Focal loss and SoftIoU loss to support target localization and shape description in imbalanced tiny-ship scenes. Its behavior shifts with IoU, emphasizing difficult samples at high IoU while prioritizing poor segmentation at low IoU.

  • F. FocalIoU Loss: FocalIoU combines Focal loss and SoftIoU loss to achieve target localization and shape description with low background response and attention to small targets.
  • F. FocalIoU Loss: SoftIoU is a convergent IoU formulation with a smooth factor that avoids infinity, while γ controls attention on hard samples.
  • F. FocalIoU Loss: The paper derives FocalIoU’s gradient with respect to the network output, using p = Sigmoid(x) to relate logits x to pixel probabilities p.
  • F. FocalIoU Loss: When IoU is small, FocalIoU sharply decreases from a high loss and focuses more on difficult large-scale samples, reducing Fa while increasing IoU.
  • F. FocalIoU Loss: When IoU is large, FocalIoU behaves like Focal loss and emphasizes difficult samples, which helps increase Pd.
  • Experiments: The experiments compare MTU-Net with state-of-the-art SIRST methods and use Pd, Fa, IoU, and ROC analysis to evaluate detection and shape description.

1) Probability of Detection:

The evaluation uses centroid-based detection criteria for Pd and Fa, while IoU measures overlap between predicted and labeled targets. ROC analysis examines detection effectiveness across prediction thresholds, and Table II reports IoU, Pd, and Fa across methods.

  • 1) Probability of Detection:: A target is counted as correctly predicted when its centroid deviation is below the threshold Dthresh, set to 3 in this paper.
  • 1) Probability of Detection:: False alarm rate Fa measures falsely predicted pixels relative to all image pixels, with centroid deviations above the threshold treated as false predictions.
  • 3) Intersection over Union:: IoU evaluates target description using the intersection-over-union area ratio between predicted targets and target labels.
  • 3) Intersection over Union:: Table II compares IoU, Pd, and Fa across traditional and CNN-based methods, where larger IoU and Pd and smaller Fa indicate better performance.
  • Evaluation: ROC curves vary the predicted pixel threshold τ to examine relationships among probability of detection Pd, false alarm rate Fa, and threshold.

B. Implementation Details

The study evaluates MTU-Net against traditional and CNN-based detectors on NUDT-SIRST-Sea, using thresholding and qualitative comparisons across challenging scenes. MTU-Net provides precise localization and shape segmentation with strong detection metrics and lower false-alarm performance.

  • Implementation Details: The evaluation compares MTU-Net with seven traditional and five CNN-based SIRST methods on NUDT-SIRST-Sea.All CNN-based methods were retrained on the dataset for fair comparison.
  • Qualitative Results: In dense, port, and dim-target scenes, MTU-Net produces more precise localization and shape segmentation than the compared methods with smaller Fa.Traditional methods lose dense and port targets or produce poor segmentation for dim targets; MTU-Net improves localization in port scenes and segmentation for dim targets.
  • Implementation Details: Adaptive thresholds remove low-response areas for traditional methods, while CNN methods use fixed thresholds from their original papers.The adaptive threshold uses the output maximum, standard deviation, and average.
  • Quantitative Results: MTU-Net achieves 64.14% IoU, 85.44% Pd, and 11.72 × 10−6 Fa on NUDT-SIRST-Sea.It exceeds other deep learning methods by more than 15% on IoU and Pd, while its Fa is only 3.82 × 10−6 higher than ResU-Net.

3) 3D ROC Analysis:

ROC analyses show MTU-Net has strong detection effectiveness, target detectability, and background suppression. Ablations further show that multi-level ViT features and CRRP augmentation improve IoU and Pd while reducing Fa.

  • 3D ROC Analysis: MTU-Net’s (Fa,Pd) ROC curve is closer to the upper-left corner than competing methods, indicating better detection effectiveness.Its (τ,Fa) curve also achieves higher Pd when τ is smaller than 0.6.
  • Multi-level ViT Module: Removing all ViT branches and MVTM decreases IoU by 11.20% and Pd by 7.06%, while increasing Fa by 32.04 × 10−6.Removing only level 1 ViT decreases IoU by 3.93% and Pd by 2.85%, and increases Fa by 0.86 × 10−6.
  • Data Augmentation: CRRP augmentation improves IoU by 5.17% and Pd by 5.05% while reducing Fa by 11.96 × 10−6 compared with no augmentation.Compared with copy-paste augmentation, CRRP improves IoU by 2.69% and Pd by 2.80%, while reducing Fa by 5.52 × 10−6.
  • Data Augmentation: CRRP addresses foreground-background imbalance by preserving long-range and contextual information from targets in complex scenes.Without augmentation, highlighted backgrounds and suspicious targets produce more false alarms.

3) FocalIoU Loss:

FocalIoU is designed to balance target localization and shape description by adapting emphasis across IoU levels. Its ablation results show better IoU and Fa than Focal loss, while visualization shows a shift toward small targets as IoU rises.

  • FocalIoU Loss: FocalIoU focuses more on low-IoU images and reduces difficult-sample weights relative to simple samples when IoU is small.The loss is intended to jointly support target localization and shape description.
  • FocalIoU Loss: Compared with Focal loss, FocalIoU improves IoU by 11.02% and reduces Fa by 22.21 × 10−6, while Focal loss increases Pd by 0.74%.Focal loss emphasizes difficult positive samples but produces higher background responses and more false alarms.
  • FocalIoU Loss: As IoU rises from 0.2 to 0.6, MTU-Net shifts its focus from large-scale targets toward small-scale targets.The visualization supports FocalIoU’s reported double-win in target localization and shape description.
  • Conclusion: The complete MTU-Net pipeline combines multi-level long-distance features, CRRP augmentation, and FocalIoU loss for space-based tiny ship detection.The conclusion reports that the resulting model outperforms traditional and existing deep learning SIRST methods across evaluation metrics.
Loading 2209.13756v1…