PaperlayerView exact revision on arXiv

Source-linked AI summary

SCTransNet: Spatial-channel Cross Transformer Network for Infrared Small Target Detection

Shuai Yuan, Hanlin Qin, Xiang Yan, Naveed AKhtar, Ajmal Mian

arXiv:2401.15583v3cs.CV

TL;DR

IRSTD methods can struggle when small targets resemble their backgrounds because semantic interaction across levels and encoder-decoder feature enhancement remain limited. SCTransNet addresses this with spatial-channel cross transformer blocks using SSCA and CFN, and comprehensive evaluation on three public datasets reports superiority over existing IRSTD methods.

  • Problem

    Small infrared targets can resemble complex backgrounds, while existing methods inadequately model cross-level semantics and encoder-decoder feature differences.

  • Method

    SCTransNet places spatial-channel cross transformer blocks on long-range skip connections to mix encoder features and enhance target-background semantic differences.

  • Results

    SCTransNet outperforms other algorithms across IoU, nIoU, and F-measure on NUDT-SIRST, NUAA-SIRST, and IRSTD-1k.

  • Takeaways & Limitations

    The method strengthens semantic differences between targets and backgrounds for accurate infrared small-target detection.

Abstract

from arXiv · show

Infrared small target detection (IRSTD) has recently benefitted greatly from U-shaped neural models. However, largely overlooking effective global information modeling, existing techniques struggle when the target has high similarities with the background. We present a Spatial-channel Cross Transformer Network (SCTransNet) that leverages spatial-channel cross transformer blocks (SCTBs) on top of long-range skip connections to address the aforementioned challenge. In the proposed SCTBs, the outputs of all encoders are interacted with cross transformer to generate mixed features, which are redistributed to all decoders to effectively reinforce semantic differences between the target and clutter at full scales. Specifically, SCTB contains the following two key elements: (a) spatial-embedded single-head channel-cross attention (SSCA) for exchanging local spatial features and full-level global channel information to eliminate ambiguity among the encoders and facilitate high-level semantic associations of the images, and (b) a complementary feed-forward network (CFN) for enhancing the feature discriminability via a multi-scale strategy and cross-spatial-channel information interaction to promote beneficial information transfer. Our SCTransNet effectively encodes the semantic differences between targets and backgrounds to boost its internal representation for detecting small infrared targets accurately. Extensive experiments on three public datasets, NUDT-SIRST, NUAA-SIRST, and IRSTD-1k, demonstrate that the proposed SCTransNet outperforms existing IRSTD methods. Our code will be made public at https://github.com/xdFai.

I. Introduction

IRSTD methods must model both small-target saliency and background continuity because targets can closely resemble complex scenes. SCTransNet addresses these limitations by connecting encoder and decoder features through spatial-channel cross transformer blocks.

  • CNN-based approaches emphasize local target information but inadequately establish comprehensive global image semantics.
  • Downsampling loses spatial information, weakening semantic interaction across feature levels and global semantic encoding.
  • Simple skip connections and dense nested modules do not sufficiently bridge encoder-decoder semantic gaps for segmentation mapping.
  • Small infrared targets can resemble their backgrounds, requiring detectors to model both target saliency and background continuity.
  • SCTransNet adds SCTBs to skip connections, with SSCA modeling cross-level long-range context and CFN enhancing target-background semantic differences.
  • Its semantic modeling distinguishes targets from background clutter and is reported to reduce false alarms.

II. RELATED WORK

Prior IRSTD methods use CNN feature fusion, nested interactions, or transformer hybrids to recover target information and model context. Their remaining challenge is establishing sufficiently comprehensive long-range semantics across feature levels.

  • CNN-based IRSTD methods use cross-layer or densely nested feature fusion to preserve and enhance small-target representations.
  • CNNs’ inductive bias makes unambiguous long-range contextual modeling difficult, motivating transformer components in CNN backbones.
  • ViT-based methods establish long-distance dependence among image patches and have been applied to global image modeling in IRSTD.
  • Single-feature spatial attention does not establish contextual connections among all feature levels.
  • SCTransNet introduces channel-wise cross transformers on long-range skip connections to connect cross-channel semantic patterns across levels with acceptable computational overhead.

C. Channel-wise Cross Transformer on Image Processing

SCTransNet uses a U-shaped encoder-decoder pipeline that blends multilevel features on long-range skip connections before decoding and deeply supervised saliency fusion.

  • A. Overall pipeline: The network extracts high-level encoder features, embeds them at multiple resolutions, and feeds them into SCTB for full-level semantic blending.
  • A. Overall pipeline: SCTB outputs are restored through feature mapping and merged with encoder features using residual connections before decoder processing.
  • A. Overall pipeline: Channel-wise cross attention fuses high- and low-level features, followed by decoding with two CBL blocks.
  • A. Overall pipeline: Each decoder output produces a saliency map through 1 × 1 convolution and sigmoid activation.
  • A. Overall pipeline: Low-resolution saliency maps are upsampled and fused into a final map, with BCE and combined losses used for optimization.

B. Spatial-channel Cross Transformer Block

SCTB is a spatial-channel blending unit that mixes encoded features across levels. It uses SSCA for global semantic association and CFN for complementary feature enhancement.

  • SCTB mixes full-level encoded features to address vanilla channel transformers’ insufficient spatial neighborhood modeling.
  • Its five SSCA inputs comprise four level-specific tokens and one concatenated full-level token; SSCA outputs are passed onward within SCTB.
  • SSCA: SSCA compares local-spatial channel similarity between single-level and concatenated features to establish global semantics.

1) Spatial-embedded single-head channel-cross attention:

SSCA establishes cross-level global semantics by combining local spatial context with channel information from all feature levels. The design uses single-head attention and omits positional encoding as specific architectural choices.

  • SSCA design: SSCA uses four single-level feature tokens as queries and one concatenated full-level token as key and value to establish global semantics.The inputs are normalized, and 1 × 1 convolutions consolidate pixel-wise cross-channel context before 3 × 3 depth-wise convolutions capture local spatial context.
  • Information perspectives: The LSGC paradigm combines local spatial and global channel information, while the complementary GSLC paradigm combines global spatial and local channel information.The paper presents these as distinct information-enhancement perspectives for feature modeling.
  • Architectural choices: SSCA uses a single head and patches without positional encoding rather than the common channel-cross attention configuration.These design choices are evaluated in the ablation study.

2) Complementary Feed-forward Network:

CFN enhances features through complementary multi-scale spatial modeling and global-spatial/local-channel interaction. It is intended to improve target localization and preserve background continuity.

  • LSGC enhancement: CFN first increases channel dimension with a 1 × 1 convolution, splits features into two branches, and applies 3 × 3 and 5 × 5 depth-wise convolutions.The multi-scale features are concatenated along channels and restored to their original dimensions.
  • LSGC enhancement: Chunk(·) divides the feature vector into two equal parts along the channel dimension, supporting the two-branch multi-scale operation.The notation defines the channel split used in CFN.
  • GSLC enhancement: CFN constructs GSLC information with global average pooling over spatial dimensions and one-dimensional convolution to capture local channel information.Global average pooling approximates total spatial information without computationally intensive spatial MLPs.
  • Feature representation: By combining complementary spatial and channel information, CFN enriches feature representations for target localization and background global continuity.This cross-spatial-channel interaction is the stated contribution of the feed-forward design.

IV. Experiments and Analysis

The paper evaluates IRSTD models with pixel-level overlap, detection, false-alarm, and ROC-based metrics to assess accuracy across operating thresholds.

  • Evaluation metrics: IoU measures pixel-level overlap between predicted and ground-truth target regions.It is defined using intersection and union regions across samples.
  • Evaluation metrics: nIoU is a normalized version of IoU for evaluating target-region overlap.
  • Evaluation metrics: F-measure evaluates pixel-level miss detection and false alarms using precision and recall.Precision and recall are the component rates.
  • Evaluation metrics: Pd is the ratio of correctly predicted targets to all targets, with centroid deviation below 3 counted as correct.This target-level criterion operationalizes correct prediction.
  • Evaluation metrics: ROC curves describe how Pd changes with varying Fa, providing evaluation beyond fixed thresholds.

B. Experiment settings

Experiments use three public datasets, standardized splits, controlled retraining of learning-based baselines, and specified preprocessing and optimization settings.

  • Datasets: The experiments use NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, containing 427, 1327, and 1000 images, respectively.Standard partitioning procedures are adopted for all datasets.
  • Implementation details: SCTransNet uses a U-Net backbone with four residual blocks and four downsampling layers.
  • Implementation details: Training normalizes images, randomly crops 256×256 patches, and applies random flipping and rotation for augmentation.
  • Implementation details: The model uses BCE loss, Adam optimization, Kaiming initialization, and cosine annealing from 0.001 to 1 × 10−5.
  • Baselines: SCTransNet is compared with seven traditional and nine learning-based IRSTD methods on all three datasets.Learning-based methods are retrained using the same training datasets and their original fixed thresholds.

C. Quantitative Results

SCTransNet outperforms the compared IRSTD algorithms across the three datasets, combining strong contour metrics with reduced false alarms and favorable ROC behavior.

  • Quantitative comparison: SCTransNet leads all compared algorithms in IoU, nIoU, and F-measure across NUAA-SIRST, NUDT-SIRST, and IRSTD-1K.The authors associate this with retaining target contours and distinguishing target-background pixel differences.
  • Quantitative comparison: SCTransNet achieves a superior balance between false alarms and detection accuracy, reflected by its high F-measure.
  • ROC analysis: SCTransNet reaches the highest detection accuracy while maintaining the lowest false alarms on NUAA-SIRST and NUDT-SIRST at suitable thresholds.
  • ROC analysis: SCTransNet achieves optimal detection performance across the evaluated false-alarm rates and shows a more continuous, rounded ROC curve.The authors describe this as tunable adaptability.
  • Visual results: In visual comparisons, SCTransNet separates nearby targets, distinguishes a UAV from mountains, and avoids some false alarms on stones in grass.The qualitative results cover NUAA-SIRST, NUDT-SIRST, and IRSTD-1K.
  • Ablation study: SSCA improves baseline U-Net IoU, nIoU, and F-measure by 4.66%, 4.93%, and 2.87%, respectively.The ablation attributes these gains to full-level information modeling.

1) The Spatial-channel Cross Transformer Block:

The SCTB mixes full-stage encoder outputs and redistributes them through decoders, while SSCA and CFN designs strengthen spatial, channel, and multi-scale feature interactions.

  • SCTB design: SCTB uses four query levels and forms keys and values from concatenated features spanning all four encoder levels.This design mixes and redistributes full-stage features for contextual target-background prediction.
  • Full-level feature modeling: Full-level queries perform best, while complete channel information helps queries capture more accurate key features across three datasets.Higher, more comprehensive query levels contain richer image semantics.
  • SSCA design: SSCA outperforms MCA and the positional-encoding variant on the three datasets, whereas absolute positional encoding is unsuitable for variable-size IRSTD inputs.The authors link scaled position embeddings to inaccurate small-target position coding.
  • SSCA design: Replacing single-head attention with multi-head attention decreases IoU, nIoU, and F-measure by 1.15%, 1.52%, and 0.73% on SIRST-1K.The authors attribute this to a more complicated feature-mapping space for targets with limited features.
  • SSCA design: Local spatial embedding improves detection across the three datasets by capturing target details and background spatial correlations in deep layers.The authors report fewer missed detections and higher detection confidence.
  • CFN design: CFN is evaluated against representative feed-forward networks as a feature-enhancement component within SCTransNet.The comparison includes conventional, localized, gated-convolution, and multi-scale feed-forward designs.

3) The Complementary Feed-forward Network:

The complementary feed-forward network improves infrared small-target feature discrimination through local spatial aggregation and multi-scale spatial-channel interaction, while preserving nearly unchanged model size and complexity.

  • LeFF slightly improves metrics over FFN, indicating that local spatial information aggregation benefits infrared small-target detection.
  • GDFN has low detection accuracy because gated convolution can treat infrared small targets as noise and filter them out.
  • MSFN outperforms all evaluated feed-forward variants except CFN, highlighting the value of multi-scale structures for spatial interaction.
  • CFN improves target enhancement and clutter suppression in building and jungle backgrounds, leading to improved detection accuracy.
  • The complementary mechanism leaves the network’s parameters and computational complexity almost unchanged.

4) The Impact of CCA Block:

The analysis evaluates alternative cross-layer fusion modules and model hyperparameters, including robustness to real stripe noise. SCTransNet’s SCTB-based design supports improved feature encoding and accurate detection in the illustrated noisy images.

  • The Impact of CCA Block: Replacing SCTransNet’s CCA module with ACM, AGPC, or AFFPN produces comparison variants for evaluating alternative cross-layer feature fusion structures.
  • Core Hyper-parameter Analysis: Increasing residual-block depth slightly raises parameters and flops while significantly improving IRSTD performance, supporting the use of four residual blocks.
  • Core Hyper-parameter Analysis: Increasing SCTB count steadily improves performance, but six SCTBs are slightly better than four at excessive computational cost.
  • Core Hyper-parameter Analysis: The channel expansion factor η = 2.66 gives the best performance, while base width W=48 causes slight degradation.
  • Robustness to Real Stripe Noise: Real stripe noise destroys local target-neighborhood information, yet SCTransNet accurately detects two targets in the illustrated case while other methods do not.
  • The Impact of CCA Block: SCTBs associate encoder and decoder features to predict target-background context differences, while SSCA exchanges local spatial and global full-level channel information.
Loading 2401.15583v3…