PaperlayerView exact revision on arXiv

Source-linked AI summary

MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detection

Tianxiang Chen, Zi Ye, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Nenghai Yu, Jieping Ye

arXiv:2403.02148v4cs.CV

TL;DR

Infrared small target detection needs local detail and global context, while transformer-based context modeling is computationally expensive and direct Mamba transfer underexplores local features. The paper proposes MiM-ISTD, which nests Outer and Inner Mamba blocks over visual sentences and words. On NUAA-SIRST and IRSTD-1k, it reports superior accuracy and efficiency, including an 8× speedup and 62.2% lower GPU memory usage at 2048 × 2048 resolution.

  • Problem

    ISTD requires global context and local features, but hybrid methods inherit quadratic transformer complexity and direct visual Mamba underexplores local features.

  • Method

    MiM-ISTD nests Outer Mamba blocks over visual sentences with shared Inner Mamba blocks over visual words to extract global and local information efficiently.

  • Results

    8× speedup over the SOTA method and 62.2% lower GPU memory usage are reported for 2048 × 2048 images during inference.

  • Takeaways & Limitations

    MiM-ISTD provides a Mamba-based ISTD structure for efficient modeling of both local and global information.

Abstract

from arXiv · show

Recently, infrared small target detection (ISTD) has made significant progress, thanks to the development of basic models. Specifically, the models combining CNNs with transformers can successfully extract both local and global features. However, the disadvantage of the transformer is also inherited, i.e., the quadratic computational complexity to sequence length. Inspired by the recent basic model with linear complexity for long-distance modeling, Mamba, we explore the potential of this state space model for ISTD task in terms of effectiveness and efficiency in the paper. However, directly applying Mamba achieves suboptimal performances due to the insufficient harnessing of local features, which are imperative for detecting small targets. Instead, we tailor a nested structure, Mamba-in-Mamba (MiM-ISTD), for efficient ISTD. It consists of Outer and Inner Mamba blocks to adeptly capture both global and local features. Specifically, we treat the local patches as "visual sentences" and use the Outer Mamba to explore the global information. We then decompose each visual sentence into sub-patches as "visual words" and use the Inner Mamba to further explore the local information among words in the visual sentence with negligible computational costs. By aggregating the visual word and visual sentence features, our MiM-ISTD can effectively explore both global and local information. Experiments on NUAA-SIRST and IRSTD-1k show the superior accuracy and efficiency of our method. Specifically, MiM-ISTD is $8 \times$ faster than the SOTA method and reduces GPU memory usage by 62.2$\%$ when testing on $2048 \times 2048$ images, overcoming the computation and memory constraints on high-resolution infrared images.

I. INTRODUCTION

Infrared small target detection requires both local details and global context, but existing approaches trade accuracy against computational efficiency. MiM-ISTD addresses this tension with nested Mamba blocks and reports strong accuracy and efficiency on public datasets.

  • ISTD is a binary segmentation task whose tiny targets are easily missed or confused with background disturbances.
  • CNN-based methods emphasize local features, whereas CNN–ViT hybrids capture global context but inherit ViT’s quadratic computational complexity.
  • Directly transferring visual Mamba to ISTD preserves efficiency but underexplores local features needed for small-target detection.
  • MiM-ISTD uses Outer Mamba blocks for visual sentences and shared Inner Mamba blocks for visual words, adding local modeling with negligible parameters and FLOPs.
  • 8× speedup over the SOTA method and 62.2% lower GPU memory usage are reported for each 2048 × 2048 image during inference.

II. RELATED WORK

Related ISTD work spans handcrafted, CNN-based, hybrid, and emerging Mamba-based networks. These approaches motivate MiM-ISTD by exposing the need to combine local detail extraction, global context modeling, and efficient long-sequence processing.

  • CNN-based networks focus on local extraction, while hybrid CNN–ViT methods add global context to address targets resembling their backgrounds.
  • MiM-ISTD combines visual sentences and visual words through nested Mamba blocks for efficient local and global feature extraction.
  • Mamba and SSM-based vision models offer linear sequence-length scaling and long-range dependency modeling compared with transformers.
  • Direct visual Mamba application to ISTD remains inaccurate because its local-feature modeling is insufficient for very small targets.

III. PROPOSED METHOD

The proposed method uses a Mamba-based hierarchical encoder to model image information efficiently. Its design combines state-space processing with two-dimensional scanning and a convolutional stem–encoder–decoder architecture.

  • SSMs transform one-dimensional inputs through implicit states into outputs using state-transition, projection, and skip-connection parameters.
  • Continuous SSM equations are discretized for sampled inputs using a timescale parameter and the zeroth-order hold rule.
  • The discretized formulation uses B and C in R^(D×N) and a timescale parameter Δ in R^D, with B refined by a first-order Taylor approximation.
  • MiM-ISTD is organized around a convolutional stem, a pure Mamba-based hierarchical encoder, and a plain decoder.

3) 2D Selective Scan.:

The method adapts Mamba to 2D images with quad-directional scanning, while organizing image patches as visual sentences and sub-patches as visual words for local and global modeling.

  • 2D Selective Scan.: SS2D arranges image patches into four directional sequences to provide a global receptive field without increasing linear computational complexity.Each feature-map element integrates information from all other locations through different scan directions.
  • Motivation: Directly applying visual Mamba to ISTD can yield unimpressive accuracy because local features important for small-target detection are insufficiently explored.Hybrid CNN–ViT methods address local and global modeling jointly but inherit quadratic complexity from transformers.
  • Patch organization: MiM-ISTD divides an image into patches as visual sentences and further segments each patch into smaller visual words.The visual words form sequences within their corresponding visual sentences.

1) Convolutional Stem.:

The hierarchical MiM encoder processes visual-word and visual-sentence streams with shared Inner Mamba blocks for local relations and Outer Mamba blocks for global relations.

  • Block structure: The MiM block contains shared Inner Mamba processing within a layer and an Outer Mamba path, with the architecture illustrated in Fig. 4.The figure indicates that the dashed path bypasses the Outer Mamba block.
  • Hierarchical design: The encoder uses four stages, each containing multiple MiM blocks that process both word-level and sentence-level features.The default block counts are L1, L2, L3, L4 = 2, 2, 2, 2.
  • MiM Hierarchical Encoder.: Visual words are vectorized and linearly projected into word embeddings that form one sequence for each visual sentence.The embedding dimension is c, and each collection of word embeddings is denoted W_i.
  • MiM blocks: The Inner Mamba block models relationships among visual words, while the Outer Mamba block models relationships among visual sentences.A convolutional feed-forward network augments both streams with finer local details.
  • Feature interaction: Word-level features are projected into sentence-level embeddings, integrated with sentence representations, and transformed by the Outer Mamba block.The MiM block maps word and sentence streams across hierarchical encoder layers.

3) Decoder Structure.:

The decoder upsamples hierarchical encoder outputs, fuses them with decoder features, and produces the final infrared target mask.

  • Feature fusion: Each decoder stage integrates upsampled encoder outputs with expanded decoder features and sends them to basic ResNet blocks.Patch expanding is used for decoder up-sampling instead of encoder patch merging.
  • Prediction head: The final features pass through a fully connected network head and interpolation operation to generate the mask prediction.
  • Efficiency context: 128nd is the stated SSM computation complexity for a visual sequence, whereas self-attention has complexity 4nd^2 + 2n^2d.The comparison identifies quadratic scaling with sequence length for self-attention and linear scaling for SSM.

4) Complexity Analysis.:

The complexity analysis shows that MiM retains linear complexity and can make its FLOP ratio negligible relative to a standard transformer when visual words and their embedding dimension are much smaller.

  • Complexity Analysis.: MiM block complexity is 128mnc + 128nd + 3mnc^2 + 3nd^2, combining Inner and Outer Mamba costs.Here m is the number of visual words per sentence, c is word-embedding dimension, n is sentence count, and d is sentence dimension.
  • Complexity comparison: MiM maintains linear complexity, while a standard transformer block has FLOPs 2nd(6d+n).
  • High-resolution setting: When c ≪ d and m ≪ n in high-resolution infrared images, the ratio of MiM to standard-transformer FLOPs approaches 0.Under these conditions, the MiM block introduces a trivial FLOP increase.

IV. EXPERIMENTS

The experiments evaluate MiM-ISTD on two public infrared small target detection datasets using pixel-level and object-level metrics, with centroid distance defining correct predictions.

  • Datasets: NUAA-SIRST contains 427 infrared images, while IRSTD-1k contains 1,000 real 512 × 512 infrared images.NUAA-SIRST images are resized to 512 × 512; each dataset uses 80% for training and 20% for testing.
  • Evaluation metrics: The evaluation uses IoU and nIoU for pixel-level segmentation, alongside Pd and Fa for object-level detection.IoU and nIoU assess pixel overlap, while Pd and Fa measure detection and false alarms at the object level.
  • Evaluation metrics: nIoU normalizes IoU to better balance structural similarity and pixel accuracy for infrared small targets.IoU measures the accuracy of detecting the corresponding object in a dataset.
  • Evaluation metrics: IoU and nIoU use intersection and union areas, true-positive pixels, ground-truth positives, predicted positives, and the total sample count.Ai and Au denote intersection and union areas; N is the total number of samples.
  • Evaluation metrics: Pd is the proportion of correctly predicted targets, whereas Fa is the ratio of falsely predicted target pixels to all image pixels.A prediction is correct when the centroid distance between the predicted target and ground truth is less than 3 pixels.

3) Optimization.:

MiM-ISTD generally achieves the strongest accuracy across both datasets while maintaining an efficient model, with Inner Mamba improving local-feature modeling at negligible added complexity.

  • Accuracy Comparison: MiM-ISTD generally achieves the best pixel-level and object-level performance on both NUAA-SIRST and IRSTD-1k.The comparison includes selected state-of-the-art ISTD methods.
  • Accuracy Comparison: Inner Mamba blocks improve IoU and nIoU by preserving distinguishable local details that Outer Mamba may overlook.The added local-feature modeling is reported to promote detection accuracy.
  • Accuracy Comparison: Pd reaches 100% on the NUAA-SIRST test set, indicating that MiM-ISTD detects all small targets there.The authors also report generally strong object-level results while balancing Pd against Fa.
  • Efficiency Comparison: MiM-ISTD generally offers the best efficiency-accuracy balance, with fewer computational resources than most compared methods except ACM.Its efficiency is attributed to the Mamba structure and shared visual-word relation network; removing Inner Mamba slightly reduces complexity but notably lowers average accuracy.

C. Visualization

Visualization and ROC analyses show refined target shapes, stronger preservation of local information, and robust detection across false-positive-rate ranges.

  • Visualization: MiM-ISTD more completely detects small-target shapes than other state-of-the-art methods in visual mask comparisons.The figure amplifies targets in yellow boxes and highlights false alarms and missed detections.
  • Feature visualization: MiM-ISTD preserves local information better in deeper layers and shows higher feature consistency across channels than TCI-Former.Feature maps are shown for stages 1–4, while T-SNE visualizes third-stage outputs across 64 channels.
  • ROC results: ROC curves compare NUAA-SIRST and IRSTD-1k performance across multiple thresholds rather than at one fixed segmentation threshold.The figure contains one ROC plot for each dataset.
  • ROC results: MiM-ISTD maintains a relatively high true positive rate at higher false positive rates, indicating robust overall detection.The reported advantage is especially notable in low-false-positive-rate intervals, where its true positive rate rises quickly.

E. Ablation study

The ablations evaluate the Inner Mamba replacement and the granularity of visual words and visual sentences. The reported best setting uses the present MiM configuration, while overly fine or coarse sentence division is less ideal.

  • Module ablation: The present MiM-ISTD setting achieves the best ablation results, outperforming the variant that replaces Inner Mamba with convolution, batch normalization, and activation operations.The baseline uses a plain visual Mamba encoder without Inner Mamba; the convolutional replacement tests whether Inner Mamba contributes beyond standard local operations.
  • Patch-division granularity: Visual-word granularity is varied by assigning each word to 1 × 1, 2 × 2, or 4 × 4 pixel regions in the original image.The reported present setting uses a 2 × 2 receptive field for each visual word.
  • Patch-division granularity: Too large or too small visual-sentence division granularity cannot deliver the most ideal performance.The ablation compares initial visual-sentence shapes H 4 × W 4, H 8 × W 8, and H 16 × W 16.
  • Patch-division granularity: The final configuration adopts the granularity that demonstrates the best performance in the present setting.This selection follows the observed sensitivity to visual-sentence division granularity.
Loading 2403.02148v4…