PaperlayerView exact revision on arXiv

Source-linked AI summary

Frequency-Assisted Mamba for Remote Sensing Image Super-Resolution

Yi Xiao, Qiangqiang Yuan, Kui Jiang, Yuzeng Chen, Qiang Zhang, Chia-Wen Lin

arXiv:2405.04964v2cs.CV

TL;DR

Existing remote sensing image super-resolution methods face limited receptive fields or quadratic complexity, motivating efficient global modeling and high-frequency reconstruction. FMSR integrates Mamba with frequency analysis, local modeling, and adaptive cross-level feature fusion; experiments across AID, DOTA, and DIOR demonstrate superior performance.

  • Problem

    Existing super-resolution methods suffer from limited receptive fields or quadratic complexity, while large-scale remote sensing images require efficient modeling of global and heterogeneous representations.

  • Method

    FMSR combines the Vision State Space Module and Frequency Selection Module for global spatial-frequency modeling, Hybrid Gate Module local modeling, and learnable adaptors for cross-level feature fusion.

  • Results

    Experiments across AID, DOTA, and DIOR demonstrate superior quantitative and qualitative performance, including sharper edges, richer textures, and improved recovery of high-frequency details.

  • Takeaways & Limitations

    FMSR provides an efficient paradigm for remote sensing image super-resolution that captures global and local dependencies across spatial and frequency domains.

Abstract

from arXiv · show

Recent progress in remote sensing image (RSI) super-resolution (SR) has exhibited remarkable performance using deep neural networks, e.g., Convolutional Neural Networks and Transformers. However, existing SR methods often suffer from either a limited receptive field or quadratic computational overhead, resulting in sub-optimal global representation and unacceptable computational costs in large-scale RSI. To alleviate these issues, we develop the first attempt to integrate the Vision State Space Model (Mamba) for RSI-SR, which specializes in processing large-scale RSI by capturing long-range dependency with linear complexity. To achieve better SR reconstruction, building upon Mamba, we devise a Frequency-assisted Mamba framework, dubbed FMSR, to explore the spatial and frequent correlations. In particular, our FMSR features a multi-level fusion architecture equipped with the Frequency Selection Module (FSM), Vision State Space Module (VSSM), and Hybrid Gate Module (HGM) to grasp their merits for effective spatial-frequency fusion. Considering that global and local dependencies are complementary and both beneficial for SR, we further recalibrate these multi-level features for accurate feature fusion via learnable scaling adaptors. Extensive experiments on AID, DOTA, and DIOR benchmarks demonstrate that our FMSR outperforms state-of-the-art Transformer-based methods HAT-L in terms of PSNR by 0.11 dB on average, while consuming only 28.05% and 19.08% of its memory consumption and complexity, respectively. Code will be available at https://github.com/XY-boy/FreMamba

I. INTRODUCTION

Remote sensing super-resolution must recover high-resolution detail from degraded observations while modeling broad scene context efficiently. FMSR addresses this challenge by combining Mamba-based global modeling with frequency selection, local feature processing, and adaptive cross-level fusion.

  • Motivation: Severe imaging degradation, compression, and downsampling impede high-resolution remote sensing image acquisition, making reconstruction from low-resolution observations important.The motivation includes scattering, tremor, transmission instability, and other acquisition constraints.
  • Problem: Transformer-based methods improve receptive fields through global interaction but incur quadratic complexity with token size, challenging large-scale remote sensing images.Lightweight attention variants may also sacrifice global modeling accuracy while requiring additional blocks.
  • Approach: The Frequency Selection Module adaptively identifies informative frequency cues, while the Hybrid Gate Module combines convolutional local bias with spatially varying representation.These components complement Mamba’s long-range modeling with high-frequency and local feature processing.
  • Feature Fusion: A learnable adaptor rescales cross-level representations to improve integration of global and local features.The adaptor addresses misalignment between different feature levels.
  • Approach: FMSR introduces a state space model for remote sensing image super-resolution to capture long-range dependencies with efficient global modeling.The paper presents this as the first state space model application to remote sensing image super-resolution.

A. Remote Sensing Image Super-resolution

Remote sensing image super-resolution has progressed through CNN-, Transformer-, and Mamba-based approaches, but large-scale imagery still requires efficient global modeling and preservation of high-frequency information. The paper positions frequency-assisted Mamba as a response to these combined demands.

  • CNN-based Methods: CNN-based methods expand local receptive fields with residual, dense, attention, or non-local structures but can incur substantial computational overhead.Non-local exploration is particularly costly for large-scale remote sensing imagery.
  • Transformer-based Methods: Transformer-based methods provide strong long-range modeling through self-attention but remain inefficient for high-resolution images because complexity is quadratic in token size.Efficient attention variants can reduce global modeling capability without removing the underlying quadratic issue.
  • Mamba-based Methods: Mamba offers linear-complexity global modeling, but its potential for remote sensing image super-resolution had remained unexplored.Remote sensing imagery also contains complex content and comparatively less prominent texture information.
  • Research Need: The paper argues for jointly modeling heterogeneous representations and critical high-frequency components in an efficient super-resolution scheme.This motivates extending Mamba with frequency analysis for remote sensing image super-resolution.
  • Proposed Direction: FMSR explores the spatial-frequency dual domain rather than relying only on Vision State Space Modules for spatial-wise long-range modeling.The approach incorporates latent high-frequency cues into global representation.

III. METHODOLOGY

FMSR uses a convolutional backbone, cascaded Frequency-assisted Mamba Groups, and pixel-shuffle reconstruction to process low-resolution inputs into super-resolved images. Its modules combine spatial, frequency, global, and local processing with adaptive feature integration.

  • FMSR Overview: FMSR contains shallow feature extraction, deep feature acquisition through Frequency-assisted Mamba Groups, and high-resolution reconstruction.The reconstruction path uses convolution, pixel shuffle, and a terminal convolution.
  • FMSR Overview: Multiple Frequency-assisted Mamba Groups process features sequentially, with convolutional refinement and a global skip connection preparing reconstruction features.The skip connection combines refined deep features with the initial feature F0.
  • Frequency-assisted Mamba Block: Each Frequency-assisted Mamba Block performs global and local modeling in the frequency-spatial dual domain using VSSM, FSM, and learnable feature aggregation.VSSM captures spatial long-term information, FSM introduces high-frequency cues, and α_l supports dynamic aggregation.
  • Hybrid Gate Module: The Hybrid Gate Module splits channels into attention and pixel-wise projection branches, multiplies their outputs elementwise, and applies a 1 × 1 convolution.This design treats local convolution features as coordinates and produces the output tensor Y.
  • Frequency Selection Module: The Frequency Selection Module uses 2D FFT-based operations to explore frequency information within the spatial-frequency representation.The overview identifies three FSM variants for frequency selection.

B. Frequency-assisted Mamba Block

The Frequency-assisted Mamba Block combines global and local modeling in the frequency-spatial dual domain. It uses VSSM, FSM, HGM, and learnable scaling factors to adaptively fuse representations.

  • Each Frequency-assisted Mamba Group contains cascaded Frequency-assisted Mamba Blocks, a convolution layer, and a residual connection.
  • The block applies Layer Norm, VSSM, FSM, and a learnable scaling factor to capture spatial long-term information and high-frequency cues.
  • A subsequent HGM stage models spatial locality, while FSM assists frequency learning during local modeling.
  • A second learnable scaling factor adaptively integrates the resulting global and local representations.

C. Vision State Space Module

The Vision State Space Module introduces VSSM to remote sensing image super-resolution for long-term modeling and aggregation with linear complexity. Its two branches process normalized features before combining their outputs.

  • VSSM is introduced to remote sensing image super-resolution for long-term modeling and aggregation with linear complexity.
  • The first branch expands normalized features by factor λ, then applies depth-wise convolution, SiLU, 2D-selective scanning, and Layer Norm.
  • A second branch uses another linear layer and SiLU function to transform the normalized feature.
  • The final VSSM output combines the two branch outputs with a Hadamard product followed by a linear layer.

D. Hybrid Gate Module

The Hybrid Gate Module unifies local convolutional bias with spatially varying pixel-wise gating. It is used within FMSR alongside frequency selection and multi-level feature integration.

  • The module is presented as a unified alternative to separately adding MLP or convolutional locality after long-range exploration.
  • HGM selectively amplifies or attenuates local pixel-domain features while preserving channel-specific features.
  • HGM expands the input from c to 2c channels, splits it into two branches, and processes the branches separately.
  • One branch applies convolutional processing, while the other uses pixel-wise linear projection and GELU to generate gate weights.

E. Frequency Selection Module

The Frequency Selection Module uses FFT-based operations to select informative frequency cues for frequency-spatial representation. The chosen design uses lightweight convolution and GELU-based selection.

  • FSM explores frequency-spatial dual-domain representations at both global and local levels using FFT-based frequency selection operations.
  • The three candidate operations include no selection, ReLU-based dynamic selection, and stacked 1 × 1 convolution with GELU.
  • Scheme (c) is selected because 1 × 1 convolution provides flexible frequency thresholds with a lightweight design.
  • A complexity comparison increases input resolution from 32 × 32 to 88 × 88 after roughly matching parameters and FLOPs against standard multihead self-attention.

IV. EXPERIMENT

The experiments evaluate ×4 super-resolution on AID, DOTA, and DIOR using standard image-quality metrics and efficiency measurements. Comparisons include CNN-, Transformer-, and Mamba-based methods under common training conditions.

  • Evaluation Setup: ×4 super-resolution is evaluated on AID, DOTA, and DIOR using PSNR, SSIM, and LPIPS.PSNR and SSIM are computed on the luminance channel of YCbCr space.
  • Datasets: AID provides 3000 training and 900 non-overlapping test images, while 900 DOTA and 1000 DIOR images are used for evaluation.AID images have size 640 × 640.
  • Model Configuration: FMSR is configured with 6 FMGs, 6 FMBs per FMG, internal dimension c = 96, and ×4 reconstruction.HGM and FSM use 1 × 1 kernels, while other convolutional kernels are 3 × 3.
  • Training Protocol: All compared methods are retrained on AID with L1 loss and ADAM, using the same training framework and hardware conditions.Training uses a single NVIDIA RTX 3090 GPU and runs for up to 500 epochs.
  • Efficiency Metrics: Efficiency is assessed through memory consumption, inference time, parameters, and FLOPs, with inference times measured on 100 images.FLOPs are calculated using an input tensor of size 1 × 3 × 128 × 128.

D. Ablation Study

Ablation studies isolate the effects of VSSM, HGM, FSM, and adaptive feature integration under ×4 AID super-resolution. The results associate these components with improvements in reconstruction quality and feature fusion.

  • Effect of VSSM: VSSM improves baseline PSNR by 0.089 dB, demonstrating its contribution to global modeling.The baseline Model-1 obtains 27.751 dB.
  • Effect of VSSM: VSSM maintains efficiency as input resolution increases, with roughly matched parameters of 0.1279M versus 0.1308M and FLOPs of 0.1312G versus 0.1534G.The comparison uses VSSM against standard multihead self-attention with adjusted dimensions.
  • Effect of HGM: Using VSSM and MLP improves performance by 0.258 dB, while channel attention performs similarly to the VSSM-based comparison.The reported PSNRs are 28.088 dB and 28.104 dB for the compared variants.
  • Effect of FSM: FSM improves PSNR by 0.056 dB and improves high-frequency bridge textures compared with removing frequency selection.ReLU-based selection gains 0.018 dB, while 1 × 1 convolution with GELU achieves the best reported variant performance.
  • Effect of HAI: Removing HAI reduces performance by 0.13 dB, while the proposed adaptive scaling achieves a 0.36 dB improvement over a residual connection.The adaptive scaling factor α adjusts global- and local-range features for feature integration.

3) Model Efficiency:

FMSR targets a favorable performance–complexity trade-off through its chosen depth and Mamba-based global modeling. It achieves strong quality with reduced memory relative to competing large models, while inference time involves explicit trade-offs.

  • Comparison Scope: The efficiency comparison covers CNN-, Transformer-, and Mamba-based SR methods using parameters, FLOPs, inference time, and memory consumption.The study includes models such as EDSR, NLSN, HAT-L, RGT, MambaIR, and FMSR.
  • Network Depth: Increasing FMBs raises PSNR from 27.744 to 28.181 dB while increasing parameters from 4.55 to 15.37M, motivating the choice of 6 FMBs.The selected depth balances reconstruction quality and model size.
  • Memory Consumption: FMSR achieves 0.18 dB higher performance than NLSN while reducing memory by 160 MB.FMSR also delivers better performance than HAT-L using only 28% of its memory.
  • Inference Times: FMSR improves PSNR over RGT by 0.12 dB but requires about 90.6 ms more inference time for 100 images.The comparison illustrates a quality–latency trade-off.

2) Quantitative Evaluations:

Across remote-sensing benchmarks and scene categories, FMSR reports strong quantitative and qualitative performance while combining global modeling with frequency-aware reconstruction. The study focuses on ×4 super-resolution and identifies this scale as its evaluation scope.

  • Quantitative Results: FMSR and FMSR++ achieve the second-highest and highest average performance across PSNR, SSIM, and LPIPS on AID, DOTA, and DIOR.On AID, FMSR exceeds HAT-L by 0.12 dB PSNR with 29% of its parameters and 19% of its FLOPs.
  • Quantitative Results: FMSR obtains 0.18 dB higher PSNR than GRL-L on DOTA with 60% lower complexity.The reported comparison supports performance gains alongside reduced computational complexity.
  • Generalization: Across 30 AID scene categories, FMSR shows stronger generalization and achieves the best performance on almost all remote-sensing scenarios.On the Industrial scene, FMSR exceeds MambaIR by 0.18 dB PSNR and 0.039 SSIM.
  • Qualitative Results: FMSR recovers sharper edges and richer textures than competing methods in visual comparisons on AID, DOTA, and DIOR at scale ×4.It reconstructs runway lines and severely damaged ground lines that HAT-L and MambaIR fail to recover accurately in cited examples.
  • Qualitative Results: On a DIOR image with large-scale global information, FMSR preserves more high-frequency contextual information than CNN-, Transformer-, and Mamba-based alternatives.The comparison attributes this result to VSSM together with spatial-frequency dual-domain exploration.
  • Attribution Analysis: FMSR surpasses HAT-L by 8.637 in diffusion index, indicating wider-range pixel utilization during super-resolution.The LAM comparison links this behavior to the model’s spatial-frequency dual-domain representation.
  • Conclusion: The study introduces state space modeling for RSI super-resolution and combines linear-complexity global modeling with frequency selection and adaptive global-local feature adjustment.The reported conclusion is based on quantitative and qualitative experiments across AID, DOTA, and DIOR.
  • Scope: The evaluation focuses solely on ×4 super-resolution, limiting flexibility in assessing other scaling factors.The authors plan to extend FMSR to more scaling factors in future work.
Loading 2405.04964v2…