PaperlayerView exact revision on arXiv

Source-linked AI summary

UIU-Net: U-Net in U-Net for Infrared Small Object Detection

Xin Wu, Danfeng Hong, Jocelyn Chanussot

arXiv:2212.00968v1cs.CV

TL;DR

Infrared small-object detection is challenged by tiny-object loss and weak object-background distinguishability in classification-backbone networks. UIU-Net uses a trainable-from-scratch nested U-Net with resolution-maintaining deep supervision and interactive-cross attention. It reports strong results on SIRST and synthetic images, generalization to ATR video sequences, and robustness on reduced-data settings.

  • Problem

    Classification-backbone methods can lose tiny-object information as depth increases, while infrared targets require precise global and local contrast representation.

  • Method

    UIU-Net embeds a small U-Net within a larger U-Net, combining RM-DS for high-resolution multi-scale global features with IC-A for low-level and high-level local-context interaction.

  • Results

    UIU-Net shows superior detection performance on SIRST and synthetic datasets, generalizes to ATR video sequences, and remains optimal with small-scale data despite about a 4% IoU and nIoU decrease.

  • Takeaways & Limitations

    Resolution maintenance and cross-level feature interaction provide a practical framework for preserving infrared small-object representations without relying on a classification backbone.

Abstract

from arXiv · show

Learning-based infrared small object detection methods currently rely heavily on the classification backbone network. This tends to result in tiny object loss and feature distinguishability limitations as the network depth increases. Furthermore, small objects in infrared images are frequently emerged bright and dark, posing severe demands for obtaining precise object contrast information. For this reason, we in this paper propose a simple and effective ``U-Net in U-Net'' framework, UIU-Net for short, and detect small objects in infrared images. As the name suggests, UIU-Net embeds a tiny U-Net into a larger U-Net backbone, enabling the multi-level and multi-scale representation learning of objects. Moreover, UIU-Net can be trained from scratch, and the learned features can enhance global and local contrast information effectively. More specifically, the UIU-Net model is divided into two modules: the resolution-maintenance deep supervision (RM-DS) module and the interactive-cross attention (IC-A) module. RM-DS integrates Residual U-blocks into a deep supervision network to generate deep multi-scale resolution-maintenance features while learning global context information. Further, IC-A encodes the local context information between the low-level details and high-level semantic features. Extensive experiments conducted on two infrared single-frame image datasets, i.e., SIRST and Synthetic datasets, show the effectiveness and superiority of the proposed UIU-Net in comparison with several state-of-the-art infrared small object detection methods. The proposed UIU-Net also produces powerful generalization performance for video sequence infrared small object datasets, e.g., ATR ground/air video sequence dataset. The codes of this work are available openly at \url{https://github.com/danfenghong/IEEE_TIP_UIU-Net}.

I. INTRODUCTION

Infrared small objects are difficult to detect because they are tiny, low-contrast, and embedded in complex backgrounds. UIU-Net addresses these challenges with a resolution-maintaining nested U-Net and interactive-cross attention, and reports strong performance across image and video datasets.

  • Challenge: Infrared small objects often cover less than 30×30 pixels and lack color and texture information in complex backgrounds.These characteristics arise in long-distance applications such as maritime rescue and airborne-object detection.
  • Prior limitations: Model-driven methods are vulnerable to clutter, noise, hyperparameters, and inadequate local-contrast modeling, which limits robustness and generalization.
  • Prior limitations: Classification-backbone segmentation networks can lose tiny-object information during downsampling and overlook local and global contrast information.
  • Proposed approach: UIU-Net models infrared small-object detection as semantic segmentation and trains an interactive-cross attention nested U-Net from scratch.
  • Proposed approach: RM-DS integrates Residual U-blocks into deep supervision to learn high-resolution multi-scale features and improve global context representation.
  • Proposed approach: IC-A interactively cross-encodes low-level details and high-level semantic features to enhance local contextual contrast.
  • Evaluation: Experiments on SIRST and synthetic single-frame datasets report advantages over state-of-the-art methods, with additional generalization evaluation on ATR video sequences.

II. RELATED WORK

Prior infrared segmentation methods improve detection but commonly depend on classification backbones and downsampling that can discard contrast information. UIU-Net instead maintains resolution while combining multi-scale global features with interactive local-context encoding.

  • Object Segmentation: Semantic segmentation formulates infrared small-object detection as pixel-level classification with dense image prediction.
  • Object Segmentation: Existing infrared segmentation networks generally rely on classification backbones and typical downsampling, while overlooking local and global contrast information.
  • Object Segmentation: The proposed multi-scale depth supervision structure addresses the conflict between feature resolution and network depth while improving contextual representation.
  • Attention Mechanism: Interactive-cross attention encodes pixel-level local context from resolution-maintaining multi-scale features to enhance infrared-object detail information.
  • Attention Mechanism: UIU-Net first learns deep multi-scale features with RM-DS and then feeds them to IC-A for local-context encoding.

A. Network Overview

UIU-Net replaces a classification backbone with a nested U-Net architecture that maintains feature resolution while learning deep multi-scale and global-context representations. Its RM-DS module uses RSU blocks and intermediate-layer supervision, while IC-A addresses remaining localization limitations.

  • Network Overview: UIU-Net is an interactive-cross attention nested U-Net trained from scratch without a classic classification backbone.It is designed for infrared small object detection.
  • Network Overview: RM-DS uses multiple intermediate layers instead of only the last layer to obtain complete and distinguishable features.This design targets low object-background contrast in infrared images.
  • Network Overview: RSU blocks take intermediate feature maps as input and encode deep multiscale features using U-Net structures with dilated convolutions.Pooling reduces computation in standard RSU blocks, while additional RSU blocks omit pooling to reduce feature loss.
  • Network Overview: Shallow RSU layers use one final dilated convolution, whereas deep layers use dilated convolutions at varying rates across multiple layers.The deep-layer design maintains low memory consumption because deep feature maps are small.
  • Network Overview: Increasing depth within each stage does not affect feature resolution and enhances global-context representation, but RM-DS alone cannot precisely locate objects in changing backgrounds.The IC-A module is therefore used to address this localization limitation.

C. Interactive-cross Attention (IC-A) Module

IC-A replaces the original U-Net skip connection with cross-channel and interactive-cross spatial attention. It combines low-level detail and high-level semantic features to encode local context and improve object-background distinguishability.

  • Interactive-cross Attention: IC-A replaces the original U-Net skip connection and retains more decoder-layer context information.Its coding objects are low-level detail and high-level semantic features.
  • Interactive-cross Attention: Cross-channel attention and interactive-cross spatial attention are the two suboutputs of IC-A.The module connects low-level encoder details with high-level semantic features.
  • Interactive-cross Attention: High-level feature channels provide distinct object responses whose interdependencies help focus on infrared small objects.The channel features are defined after RSU processing in the deep encoder.
  • Interactive-cross Attention: Adaptive average pooling traverses high-level features, and excitation reshapes channels from C to C/r and back with r=4.The excitation operation is used to minimize network parameters.
  • Interactive-cross Attention: Interactive-cross spatial attention applies excitation to cross-attention features and aggregates them with average-pooling and max-pooling.This design targets local detail information needed to distinguish objects from backgrounds.

A. Dataset Description

The evaluation uses SIRST, synthetic single-frame infrared data, and ATR ground/air video-sequence data. These datasets cover dark or cluttered small targets, synthetic scenes, and shifting backgrounds with varied UAV conditions.

  • SIRST Dataset: SIRST contains 427 representative images from different real-world scenarios and supports both detection and segmentation tasks.The dataset includes short-, mid-, and 950nm wavelength imagery.
  • SIRST Dataset: Most SIRST objects are very dark and buried in complicated backgrounds, while only 35% belong to the brightest pixels.The dataset contains substantial clutter.
  • Synthetic Dataset: The experiment adds 80 representative synthetic multiple-scene images for verification and 20 true SIRST images for quantitative analysis.These images are used for model verification and quantitative evaluation, respectively.
  • Synthetic Dataset: The synthetic dataset overlays real infrared objects or two-dimensional Gaussian functions onto high-resolution natural-scene images.Its objects come from 11 real sequences and 100 individual infrared images.
  • ATR Ground/Air Dataset: ATR ground/air data contains 22 segments, 30 traces, 16,177 frames, and 16,944 labeled objects collected across sky, ground, and other scenes.Objects occur in complex, shifting backgrounds and vary in size, interference, storage, and mobility.

B. Evaluation Metrics

UIU-Net is evaluated as a semantic-segmentation detector using IoU, nIoU, and ROC, with ROC additionally measuring performance across sliding thresholds.

  • UIU-Net models infrared small object detection as semantic segmentation and evaluates it with IoU, nIoU, and ROC.
  • IoU divides the intersection of real and predicted pixels by their union.
  • nIoU averages per-sample IoU values to balance model-driven and data-driven evaluation.
  • ROC traces false positive rate against true positive rate under a sliding threshold, unlike IoU at a fixed threshold.

C. Ablation Study

Ablation results show that multi-scale residual backbones and interactive-cross attention improve UIU-Net’s infrared small-object detection, while broader comparisons favor the proposed method.

  • Backbone comparison: IoU increased from 0.6178 to 0.7825 on SIRST and from 0.4335 to 0.4773 on the synthetic dataset with the multi-scale residual backbone.
  • Backbone comparison: nIoU improved by ∼0.12 to 0.7515 on SIRST and by ∼0.07 to 0.4721 on the synthetic dataset.
  • Module contributions: Adding the incremental module increased IoU from 0.7330 to 0.7825 on SIRST and from 0.4100 to 0.4773 on the synthetic dataset.
  • Comparative evaluation: The proposed method achieved the best IoU and nIoU among the compared methods and performed best under dynamic thresholds.
  • Visual comparison: Visual comparisons showed optimal results for tightly connected multiple objects, whereas ACM and MDvsFA-cGAN exhibited adhesion or pixel loss.

E. Results and Analysis on the Synthetic Data

On synthetic data, UIU-Net remained strongest despite resolution and labeling challenges, retained top performance with fewer samples, and showed some missed detections in visual results.

  • Quantitative comparison: Synthetic data produced worse IoU and nIoU than SIRST because of lower resolution and inflated reference-label ranges.
  • Data volume comparison: With a 12:1 large-to-small training-sample ratio, UIU-Net’s IoU and nIoU decreased by about 4% but remained optimal.
  • ROC comparison: UIU-Net’s ROC curve remained best, while model-driven methods and an underfit MDvsFA-cGAN fell outside or performed poorly within the displayed range.
  • Visual comparison: UIU-Net had the highest similarity to the reference label in the highlighted example, but some images still contained missed detections.

F. Generalization Analysis

UIU-Net generalized from single-frame infrared training data to ATR video sequences, outperforming comparison methods across training-set conditions while remaining vulnerable to difficult backgrounds and scene changes.

  • Cross-dataset generalization: UIU-Net trained on 408 SIRST images exceeded the model trained on 500 synthetic images but remained below training with 6,200 synthetic images.
  • Cross-dataset generalization: The SIRST-trained UIU-Net exceeded other compared methods under both 6,200-image and 500-image synthetic training conditions.
  • Video evaluation: ATR video evaluation sampled data2, data6, and data7, displaying one detection result every 50 frames.
  • Video results: UIU-Net performed well for cross-flying UAVs against sky backgrounds, but ground clutter and flash elements caused more missed detections.
  • Generalization conclusion: The SIRST-trained model showed superior generalization across three ATR video segments, while unknown scenes and objects remained challenging.

V. CONCLUSION

UIU-Net improves infrared small object detection by increasing network depth without reducing object resolution and by encoding multi-scale features with global and local context. Experiments on SIRST, synthetic, and ATR datasets demonstrate its effectiveness and generalization, while future work targets multi-source fusion and complex video sequences.

  • Proposed framework: UIU-Net increases network depth without reducing object resolution and avoids reliance on a classification backbone.This design addresses information loss for small objects during downsampling.
  • Proposed framework: Interactive-cross attention encodes deep multi-scale features while improving global and local context representation.
  • Experimental findings: Quantitative experiments on SIRST and synthetic datasets, plus ATR ground/sir generalization studies, demonstrate UIU-Net's superiority and efficiency.
  • Future work: Future work will investigate multi-source visible-infrared data fusion and factors affecting adaptation to varied scenes.
  • Experimental findings: The synthetic-dataset ROC results for all model-driven methods and MDvdFA cGAN are poor enough to be displayed separately.
Loading 2212.00968v1…