Source-linked AI summary
FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining
Zou Zhen, Yu Hu, Zhao Feng
TL;DR
Image deraining must recover frequency information while modeling both global degradation and local detail, but prior Mamba-based approaches have rarely incorporated frequency analysis. FreqMamba combines spatial, frequency-band, and Fourier modeling with multiscale degradation-prior attention, and experiments report strong visual and quantitative performance, including superiority over the evaluated baselines.
Problem
Rain degradation removes important frequency information and damages local details, while Mamba-based deraining has rarely been extended with frequency analysis for global degradation modeling.
Method
FreqMamba combines spatial Mamba, wavelet-packet frequency-band Mamba, Fourier global modeling, and multiscale degradation-prior attention for image deraining.
Results
FreqMamba achieves the best performance among the evaluated baseline algorithms across four deraining datasets and preserves scene integrity and detail.
Takeaways & Limitations
The complementary spatial-frequency architecture extends beyond rain removal to low-light enhancement, real-world dehazing, and other image-restoration tasks.
Abstract
from arXiv · showhide
Images corrupted by rain streaks often lose vital frequency information for perception, and image deraining aims to solve this issue which relies on global and local degradation modeling. Recent studies have witnessed the effectiveness and efficiency of Mamba for perceiving global and local information based on its exploiting local correlation among patches, however, rarely attempts have been explored to extend it with frequency analysis for image deraining, limiting its ability to perceive global degradation that is relevant to frequency modeling (e.g. Fourier transform). In this paper, we propose FreqMamba, an effective and efficient paradigm that leverages the complementary between Mamba and frequency analysis for image deraining. The core of our method lies in extending Mamba with frequency analysis from two perspectives: extending it with frequency-band for exploiting frequency correlation, and connecting it with Fourier transform for global degradation modeling. Specifically, FreqMamba introduces complementary triple interaction structures including spatial Mamba, frequency band Mamba, and Fourier global modeling. Frequency band Mamba decomposes the image into sub-bands of different frequencies to allow 2D scanning from the frequency dimension. Furthermore, leveraging Mamba's unique data-dependent properties, we use rainy images at different scales to provide degradation priors to the network, thereby facilitating efficient training. Extensive experiments show that our method outperforms state-of-the-art methods both visually and quantitatively.
1 INTRODUCTION
Image deraining must recover both local details and lost frequency information under complex rain degradation. FreqMamba combines spatial Mamba with frequency-band and Fourier analysis to model these complementary aspects.
- Motivation: Rain degrades object details, contrast, and frequency information, complicating recovery because raindrops are coupled with the background.This degradation can also harm downstream classification and detection.
- Frequency analysis: Fourier transform provides global modeling and degradation separation, while wavelet packet transform supplies a spatial-frequency intermediate representation.The wavelet domain decomposes features into spatial sub-bands with varying frequencies, smoothing transitions between spatial and Fourier domains.
- FreqMamba: FreqMamba uses a complementary three-branch structure combining spatial Mamba, frequency-band Mamba, and Fourier global modeling.The design pairs Mamba’s regional spatial correlations with frequency analysis for global degradation perception.
- FreqMamba: Frequency-band Mamba applies wavelet packet decomposition and scans reordered sub-bands from low to high frequency and in reverse.This frequency-dimension scanning bridges spatial and Fourier-domain analysis while providing a new perspective for 2D modeling.
- Training strategy: Mamba-derived attention maps from rainy images at multiple scales provide degradation priors that assist effective training.The maps dynamically emphasize different image regions according to degradation distribution.
2 RELATED WORK
Related work progresses from model-based rain separation toward deep learning, frequency-domain restoration, and scalable state-space modeling. Fourier and wavelet methods provide complementary frequency views, while Mamba offers data-dependent long-range modeling with linear scalability.
- Image deraining: Early deraining methods use handcrafted features and physical priors, but struggle with complex rain patterns and diverse real-world scenarios.These limitations reduce their practical applicability and performance.
- Frequency analysis: Fourier transform converts signals into a domain where global statistical properties are easier to analyze, supporting global modeling in vision and restoration.Prior restoration work uses Fourier amplitudes to isolate global lightness or haze information.
- Frequency analysis: Wavelet transform complements Fourier analysis by providing multi-resolution analysis for signals and images.Its perspective differs from Fourier transform’s emphasis on frequency information.
- State-space models: State-space models capture long-range dependencies with linear scalability in sequence length, and Mamba adds data-dependent selection with efficient hardware design.Mamba has shown strong performance while retaining linear scalability.
3 METHOD
The method section introduces the motivation, frequency-analysis and state-space-model preliminaries, and overall FreqMamba framework.
- The method section first presents motivation, then reviews frequency analysis and state-space models, and finally outlines FreqMamba.
3.1 Motivation
Rain degradation combines global corruption with local detail destruction, motivating complementary frequency modeling and spatial Mamba. FreqMamba uses a three-branch FreqSSM block and multiscale degradation-prior attention for single-image deraining.
- Motivation: Rainy images contain both global degradation and local detail destruction, while frequency operations capture global effects but miss spatial local dependencies.Mamba’s selective scanning models interactions among image regions with linear complexity.
- Motivation: FreqMamba introduces a three-branch FreqSSM block to transition from global-degradation correction to local-content refinement.The design combines the complementary strengths of frequency-based methods and Mamba.
- Motivation: Multiscale degraded images are used to derive attention maps added in the encoder, enhancing training efficiency for single-image deraining.
3.2 Preliminaries
This section introduces Fourier and wavelet-based frequency representations alongside state space models, establishing the frequency and sequence-processing foundations for FreqMamba.
- Frequency Analysis in Digital Imaging: The Fourier transform converts spatial features into complex frequency representations that can be decomposed into amplitude and phase spectra.
- Frequency Analysis in Digital Imaging: The wavelet packet transform extends sub-band analysis by recursively decomposing all frequency sub-bands while retaining spatial information.
- State Space Models: State space models map one-dimensional inputs to outputs through latent states, while Mamba adaptively scans input data using selective scanning.
- State Space Models: FreqMamba’s three-branch FreqSSM blocks form the basic units of a U-Net architecture for global and local modeling.
3.3 Architecture
FreqMamba uses a multi-scale U-Net with Fourier, spatial Mamba, and frequency-band branches, while degradation priors guide attention across image scales and regions.
- Frequency-SSM Block: FreqMamba uses a multi-scale U-Net whose core blocks combine Fourier modeling, spatial Mamba, and frequency-band processing.The Fourier branch processes frequency-domain features and returns them to the spatial domain; the spatial branch uses LayerNorm and Mamba scanning.
- Frequency-SSM Block: The spatial branch broadly recognizes raindrops with blurry boundaries, the Fourier branch emphasizes larger streaks with high contrast, and the frequency-band branch lies between them.
- Frequency-SSM Block: The frequency-band branch applies wavelet packet decomposition, frequency-ordered scanning, and inverse transformation to connect local detail restoration with global degradation processing.
- Frequency-SSM Block: The three branch outputs are fused through residual addition, concatenation, and a final 1×1 convolution to synthesize harmonized features.
- Data-dependent Degradation Prior Attention Map: Restormer shows larger errors in mountainous regions with dense rain and complex backgrounds, whereas FreqMamba’s error map contains fewer huge error values.
- Data-dependent Degradation Prior Attention Map: Multi-scale rainy inputs generate degradation priors that highlight likely rain locations at different granularities, enabling attention to vary across image regions.
3.4 Loss Function
The loss design supervises restoration in both spatial and frequency domains, while the supplied evaluation passages define benchmark comparison scope without reporting individual table values.
- Loss Function: FreqMamba combines spatial-domain, phase-spectrum, and amplitude-spectrum losses to optimize restoration across spatial and frequency representations.
- Loss Function: The spatial loss uses L1 distance between the final output and ground truth, while separate L1 terms supervise amplitude and phase spectra.
- Quantitative Evaluation: Table 1 compares PSNR and SSIM across five image-deraining benchmark datasets, marking the highest and second-highest performances.
- Loss Function: The loss weights α and β are both empirically set to 0.05 in the implementation.
4 EXPERIMENTS
FreqMamba is evaluated against state-of-the-art deraining methods across four datasets, with qualitative, quantitative, ablation, and cross-task extension studies. It achieves the best reported baseline performance and benefits from all major architectural components.
- 4.2 Comparison with State-of-the-art Methods: FreqMamba achieves the best performance among all baseline algorithms across Rain100L, Rain100H, Test1200, and Test2800, with strong visual recovery under severe rain.The evaluation uses PSNR and SSIM on the Y channel; qualitative comparisons report reduced rain-streak residue and improved detail restoration.
- 4.3 Ablation Studies: Removing the Fourier branch lowers performance by 0.33 dB, while removing the frequency-band branch lowers it by 0.1 dB.These ablations support the roles of global Fourier-domain modeling and the introduced frequency dimension.
- 4.3 Ablation Studies: Replacing spatial Mamba with standard convolutions significantly degrades performance because their limited receptive field weakens modeling capability.The comparison keeps the other components unchanged.
- 4.3 Ablation Studies: The degradation-prior attention map is evaluated through ablation, and its use is associated with reduced restoration errors.The strategy adaptively learns degradation distributions from rainy images at different scales.
- 4.4 Extensions on Other Tasks: The three-branch FreqMamba structure is also extended to low-light image enhancement and real-world image dehazing.Experiments use LOL-V1 and LOL-V2-Synthetic for enhancement, and Dense-Haze and NH-HAZE for dehazing, with quantitative results reported in Tables 4 and 5.
5 CONCLUSION
The conclusion presents FreqMamba as a deraining network that combines spatial sequence modeling with frequency-domain global modeling and multiscale degradation-prior attention. The reported experiments support effective rain removal while preserving scene detail and applicability to other restoration tasks.
- 5 CONCLUSION: FreqMamba integrates spatial-domain sequence modeling, frequency-domain global modeling, and multiscale attention maps derived from rainy images.Its core uses a frequency SSM block and Mamba’s input dependency properties to model degradation across scales.
- 5 CONCLUSION: The model removes rain while preserving the integrity and detail of the underlying scene.The conclusion emphasizes this balance for applications where final image quality is critical.
- 5 CONCLUSION: FreqMamba is reported to extend beyond rain removal to a variety of image restoration tasks.The conclusion characterizes this adaptability as evidence of the architecture’s broader potential.