PaperlayerView exact revision on arXiv

Source-linked AI summary

MaIR: A Locality- and Continuity-Preserving Mamba for Image Restoration

Boyun Li, Haiyu Zhao, Wenxin Wang, Peng Hu, Yuanbiao Gou, Xi Peng

arXiv:2412.20066v2cs.CV

TL;DR

Mamba-based image restoration methods can overlook image locality and continuity and differences among independently processed sequences. MaIR combines structure-preserving scanning with attention-based sequence aggregation, and reports state-of-the-art results across four restoration tasks and 14 benchmarks against 40 baselines.

  • Problem

    Mamba restoration methods typically process flattened image sequences independently and aggregate them, while overlooking image locality, continuity, and differences among sequences.

    Source: §1 · p. 2

  • Method

    MaIR combines stripe-based Nested S-shaped Scanning to preserve locality and continuity with Sequence Shuffle Attention to aggregate sequences through corresponding-channel attention.

    Source: §1 · p. 2

  • Results

    MaIR reports state-of-the-art performance across four restoration tasks on 14 benchmarks compared with 40 baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across four restoration tasks and 14 benchmarks, experiments validate MaIR’s robustness and effectiveness while improving image quality without extra computations.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent advancements in Mamba have shown promising results in image restoration. These methods typically flatten 2D images into multiple distinct 1D sequences along rows and columns, process each sequence independently using selective scan operation, and recombine them to form the outputs. However, such a paradigm overlooks two vital aspects: i) the local relationships and spatial continuity inherent in natural images, and ii) the discrepancies among sequences unfolded through totally different ways. To overcome the drawbacks, we explore two problems in Mamba-based restoration methods: i) how to design a scanning strategy preserving both locality and continuity while facilitating restoration, and ii) how to aggregate the distinct sequences unfolded in totally different ways. To address these problems, we propose a novel Mamba-based Image Restoration model (MaIR), which consists of Nested S-shaped Scanning strategy (NSS) and Sequence Shuffle Attention block (SSA). Specifically, NSS preserves locality and continuity of the input images through the stripe-based scanning region and the S-shaped scanning path, respectively. SSA aggregates sequences through calculating attention weights within the corresponding channels of different sequences. Thanks to NSS and SSA, MaIR surpasses 40 baselines across 14 challenging datasets, achieving state-of-the-art performance on the tasks of image super-resolution, denoising, deblurring and dehazing. The code is available at https://github.com/XLearning-SCU/2025-CVPR-MaIR.

1. Introduction

MaIR addresses the conflict between long-range modeling and preservation of image structure in Mamba-based restoration, while also targeting the aggregation of differently unfolded sequences. It combines locality- and continuity-preserving scanning with sequence attention and reports state-of-the-art performance across four restoration tasks.

1. Introduction

  • The motivating challenge is to model long-range dependencies while preserving the locality and continuity that CNNs and Transformers capture in natural images.The introduction describes CNNs and Transformers as limited to localized input regions for modeling relationships across larger image areas.Source: §1 · pp. 1–2
  • Existing Mamba restoration methods flatten images into row- and column-based sequences, process each independently, and aggregate them, disrupting image structure and overlooking sequence-specific contexts.The described aggregation commonly uses pixel-wise summation.Source: §1 · p. 2
  • NSS uses stripe-based regions and S-shaped paths to preserve locality and continuity, while SSA aggregates sequences using attention across corresponding channels.MaIR also captures dependencies across distinct sequences, including forward and reversed rows and columns.Source: Figure 1 · p. 1, §1 · p. 2
  • MaIR reports state-of-the-art performance on four tasks across 14 benchmarks compared with 40 baselines.The tasks are super-resolution, denoising, deblurring, and dehazing.Source: §1 · p. 2

2. Related Works

The related-work section organizes restoration approaches around CNNs, Transformers, and Mamba, with the latter discussed in detail. It contrasts their handling of image structure and long-range dependencies with MaIR’s cost-free approach to preserving both locality and continuity.

2. Related Works

  • The section classifies image-restoration methods as CNN-, Transformer-, or Mamba-based and previews its discussion of the first two categories before Vision Mamba.Figure 2 illustrates MaIR’s architecture, including Residual Mamba Groups, Residual Mamba Blocks, and Visual Mamba Modules.Source: §2.1 · p. 2, Figure 2 · p. 3

2.1. Image Restoration

  • CNNs capture local structure but have limited long-range dependencies, while window-based Transformers preserve structural coherence but may also miss long-range relationships.The cited Transformer methods use local windows to avoid impractical quadratic complexity on images.Source: §2.1 · p. 2

2.2. Vision Mamba

  • Earlier Mamba restoration methods often disrupt structural coherence or add computationally costly modules, whereas MaIR offers a cost-free solution for preserving both locality and continuity.Some prior scanning strategies preserve only one of the two properties.Source: §2.2 · pp. 2–3

3. Methods

MaIR uses a three-stage restoration network with a MaIR Module built around nested scanning and sequence aggregation. Its module processes image features as multiple sequences and combines them through attention.

3.1. Overall Architecture

  • MaIR’s three stages extract shallow features, build deep features with Residual Mamba Groups, and reconstruct the restored image from shallow and deep features.Super-resolution uses pixel shuffle and convolution; denoising, deblurring, and dehazing use convolution with a residual connection.Source: §3.1 · p. 3
  • For super-resolution, MaIR is optimized with L1 loss; for denoising, deblurring, and dehazing, it uses Charbonnier loss.The supplied passages identify the target image for these losses.Source: §3.1 · p. 3

3.2. MaIR Module

  • The module’s scan and aggregation pipeline flattens features into four sequences, applies selective scan, then aggregates the processed sequences.Nested stripe partitions use a shift-stripe operation across successive modules so boundary regions are covered by a single stripe.Source: §3.2 · pp. 3–4
  • SSA pools and concatenates sequence features, shuffles their channels, derives attention weights with group convolution, then combines the sequences by weighted summation.The output feature is computed from the input features using the attention weights.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate MaIR across four image-restoration tasks, with quantitative results and visual comparisons reported for super-resolution.

4. Experiments

  • The evaluation covers image super-resolution, denoising, deblurring, and dehazing.Source: §4 · p. 4
  • Table 1 reports quantitative results on classic image super-resolution.Source: Table 1 · p. 5
  • Figure 5 visually compares ×4 super-resolution results on Manga109 and describes MaIR as preserving fine details and textures.Source: Figure 5 · p. 5
  • Table 2 reports quantitative results on lightweight image super-resolution.Source: Table 2 · p. 6

4.1. Results on Image Super-Resolution

The section evaluates classic and lightweight image super-resolution using standard training and test datasets. MaIR is compared with CNN, transformer, and Mamba baselines, with results favoring MaIR across most reported comparisons.

4.1. Results on Image Super-Resolution

  • Classic super-resolution uses DF2K for training, while lightweight super-resolution uses DIV2K; evaluation covers five test datasets.The test sets are Set5, Set14, B100, Urban100, and Manga109, with low-resolution images generated by bicubic downsampling.Source: §4.1 · p. 6
  • The comparisons include 15 competitive baselines spanning CNN, transformer, and Mamba methods, with separate classic and lightweight super-resolution sets.MaIR is available as MaIR-Tiny and MaIR-Small for lightweight super-resolution.Source: §4.1 · p. 6
  • MaIR achieves the best result in almost all classic super-resolution comparisons, while its lightweight variants also outperform cited baselines on selected ×4 comparisons.On Urban100, MaIR exceeds MambaIR by 0.03–0.12 dB in PSNR; MaIR-Small exceeds MambaIR-Small by 0.08 dB on Manga109, and MaIR-Tiny exceeds MambaIR-Tiny and SwinIR by 0.08 dB and 0.12 dB on Urban100, respectively.Source: §4.1 · p. 6

4.2. Results on Image Denoising

The denoising experiments cover synthetic Gaussian noise and real-world noise, evaluating MaIR against established restoration baselines. The reported results favor MaIR on both settings, with a stated Urban100 PSNR advantage over MambaIR.

4.2. Results on Image Denoising

  • Synthetic-noise training uses DFWB, with evaluation on BSD68, Kodak24, McMaster, and Urban100 at noise levels σ = 15, 25, and 50.Source: §4.2 · pp. 6–7
  • For real-world denoising, MaIR is trained and tested on SIDD-Medium, which provides 320 noisy-clean training pairs and 40 test pairs.Source: §4.2 · p. 7
  • The denoising comparisons include CNN, transformer, and Mamba baselines for both synthetic and real-world settings.Source: §4.2 · p. 7
  • MaIR outperforms denoising baselines on synthetic and real-world noise, including a 0.21 dB PSNR advantage over MambaIR on Urban100.The reported visual comparison also describes more detailed restored textures that more closely match ground truth.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.3. Results on Image Deblurring

The deblurring experiments use GoPro for training and evaluate on GoPro and HIDE. MaIR surpasses the reported baselines by PSNR on both datasets, with visual results highlighting blur removal and detail restoration.

4.3. Results on Image Deblurring

  • Training uses 2,103 blurry-clean GoPro pairs, and evaluation uses 1,111 GoPro test pairs and 2,025 HIDE pairs.Source: §4.3 · p. 8
  • The 11 comparison methods include CNN, transformer, RNN, and Mamba-based deblurring approaches.Source: §4.3 · p. 8
  • MaIR surpasses other deblurring baselines by PSNR on both GoPro and HIDE.It exceeds Restormer by 0.77 dB on GoPro and 0.35 dB on HIDE; on HIDE, it also exceeds NAFNet by 0.25 dB.Source: §4.3 · p. 8, Table 5 · p. 8
  • The GoPro visual comparison describes effective blur removal and detail restoration around heavily degraded wheel areas.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4. Results on Image Dehazing

MaIR is evaluated for image dehazing on RESIDE training and test sets against eight CNN-, transformer-, and Mamba-based baselines. It surpasses most baselines, with strong outdoor-scene results and lower model cost than UVM-Net.

4.4. Results on Image Dehazing

  • Evaluation uses RESIDE training sets and SOTS indoor and outdoor test sets, plus RESIDE-6K training with SOTS-mix testing.ITS has 13,990 hazy-clean pairs, OTS has 313,950 pairs, and each SOTS indoor/outdoor test set has 500 pairs or images.Source: §4.4 · p. 8
  • MaIR is compared with eight baselines: five CNN methods, two transformer methods, and one Mamba method.The Mamba baseline is UVM-Net.Source: §4.4 · p. 8
  • MaIR surpasses DehazeFormer and UVM-Net by 2.67dB and 2.04dB in PSNR, respectively, on outdoor scenes.UVM-Net is slightly higher in indoor-scene PSNR, while MaIR uses 0.3% of its parameters and 4.8% of its MACs.Source: §4.4 · p. 8

4.5. Analysis Experiments

The analysis tests NSS and SSA through ablations, visual comparisons of scanning strategies, and changes to stripe width. The reported findings favor NSS and SSA, show visually appealing results from the proposed scanning strategy, and indicate robustness across stripe widths.

4.5. Analysis Experiments

  • The analysis section covers NSS and SSA ablations, scanning-strategy observations, and stripe-width effects.The stripe-width analysis is conducted with a lightweight super-resolution model.Source: §4.5 · pp. 8–9

4.5.1. Ablation Studies

  • NSS is reported important to MaIR’s performance after comparisons with Z-shaped, LocalMamba, ZigMa, and Peano-Hilbert scanning variants.The ablation also removes the shift stripe; Table 7 evaluates lightweight ×2 super-resolution on Urban100.Source: §4.5.1 · p. 9, Table 7 · p. 9
  • SSA is reported more effective than sequence-wise addition, SSM, sequence-wise gating, channel-wise gating, and pixel-wise gating alternatives.The comparisons keep model sizes similar and use Urban100 lightweight super-resolution at ×2 scale.Source: §4.5.1 · p. 9, Table 8 · p. 9

4.5.2. Verification of Observations

  • Visual comparisons report that MaIR avoids the wrong textures of window scanning, local distortion of S-shaped scanning, and both problems of Z-shaped scanning.The comparison illustrates the scarf’s layer relationships and texture in Figure 9.Source: §4.5.2 · pp. 9–10, Figure 9 · p. 10

4.5.3. Results on different stripe width

  • MaIR’s PSNR and SSIM remain quite similar across stripe widths 2, 4, 8, 16, and 32, except at the smallest and largest widths.The lightweight ×2 super-resolution experiments use Urban100; the authors report robustness across a range of stripe widths.Source: §4.5.3 · p. 10, Table 9 · p. 10

5. Conclusion

MaIR combines NSS, which extracts locality- and continuity-preserving image sequences, with SSA, which adaptively aggregates them. Across four restoration tasks and 14 benchmarks, the authors report superiority over 40 baselines without extra computations.

5. Conclusion

  • Across four tasks and 14 benchmarks, MaIR’s experiments against 40 baselines validate its reported effectiveness and robustness in image restoration.The tasks are super-resolution, denoising, deblurring, and dehazing.Source: §5 · p. 10
  • NSS extracts locality- and continuity-preserving sequences, while SSA adaptively aggregates those sequences.The authors state that their cooperation improves image quality without introducing extra computations.Source: §5 · p. 10
  • MaIR is presented as a state space model for image restoration that preserves local dependencies and spatial continuity.Source: §5 · p. 10

Related papers

Loading 2412.20066v2…