Source-linked AI summary
Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining
Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha
TL;DR
Single-image deraining must recover background scenes despite visibility loss and complex, overlapping rain streaks, while existing approaches may neglect broad context and stage correlations. RESCAN combines dilated context aggregation, squeeze-and-excitation weighting, and recurrent multi-stage processing. The authors report superior performance against state-of-the-art methods on varied datasets and improved visual deraining of synthetic and real-world images.
Problem
Single-image deraining is difficult because complex rain streaks obscure scenes, while existing methods can neglect large-region context and correlations between removal stages.
Method
RESCAN uses dilated context aggregation, squeeze-and-excitation weighting of rain-streak channels, and recurrent processing across multiple rain-removal stages.
Results
The authors report superior performance against state-of-the-art methods across various datasets, with visual results showing substantial streak removal while maintaining background details.
Takeaways & Limitations
The proposed architecture is presented as suitable for recovering images containing complex rain streaks, especially in heavy rain and real-world conditions.
Takeaways & Limitations
The method addresses rain modeled as multiple irregular streak layers rather than relying on a single sparse layer with similar directions and shapes.
Abstract
from arXiv · showhide
Rain streaks can severely degrade the visibility, which causes many current computer vision algorithms fail to work. So it is necessary to remove the rain from images. We propose a novel deep network architecture based on deep convolutional and recurrent neural networks for single image deraining. As contextual information is very important for rain removal, we first adopt the dilated convolutional neural network to acquire large receptive field. To better fit the rain removal task, we also modify the network. In heavy rain, rain streaks have various directions and shapes, which can be regarded as the accumulation of multiple rain streak layers. We assign different alpha-values to various rain streak layers according to the intensity and transparency by incorporating the squeeze-and-excitation block. Since rain streak layers overlap with each other, it is not easy to remove the rain in one stage. So we further decompose the rain removal into multiple stages. Recurrent neural network is incorporated to preserve the useful information in previous stages and benefit the rain removal in later stages. We conduct extensive experiments on both synthetic and real-world datasets. Our proposed method outperforms the state-of-the-art approaches under all evaluation metrics. Codes and supplementary material are available at our project webpage: https://xialipku.github.io/RESCAN .
1 Introduction
Single-image deraining is challenging because heavy rain obscures scenes and existing methods often miss large-region context and correlations across iterative removal stages. RESCAN addresses these issues with contextual dilated processing, squeeze-and-excitation weighting, and recurrent stage-to-stage guidance.
- Heavy rain streaks from varied directions can obscure background scenes and impair computer vision systems, motivating single-image rain removal.
- Single-image deraining is more challenging than video deraining because images lack relationships between frames.
- Existing approaches often neglect large-region spatial context and treat iterative rain-removal stages independently, overlooking their correlations.
- RESCAN removes rain stage by stage using contextual full-convolutional layers with exponentially increasing dilations for a large receptive field.
- Squeeze-and-excitation blocks assign channel-specific alpha-values to rain-streak types, while recurrent units pass useful information from earlier stages to later ones.
- The authors report superior performance against state-of-the-art methods across various datasets, including suitability for complex streaks in heavy rain.
2 Related Works
Prior deraining work spans video-based, traditional single-image, and deep-learning approaches. The paper situates its method within efforts to model complex streak structure and remove rain at multiple scales or stages.
- Video deraining methods exploit temporal relationships between frames, making rain removal relatively easier than in single images.
- Traditional single-image methods include dictionary learning, Gaussian mixture models, and low-rank representation.
- Deep-learning methods decompose rain images, map high-frequency content to streak layers, use adversarial and perceptual losses, or model multiple scales.
3 Rain Models
The paper models rainy images as combinations of a clean background and multiple rain-related layers, extending the model to include atmospheric effects and layer-specific brightness. This reflects the irregular shapes, depths, attenuation, scattering, and haze found especially in heavy rain.
- Basic rain model: A common rain model decomposes the observed image O into a rain-free background B and a rain streak layer R.Removing R from O yields the rain-free scene B.
- Multiple rain layers: A single rain layer cannot adequately model rain streaks with varied appearances, depths, directions, and shapes.The irregular distribution of streaks makes one layer insufficient for complex rainy scenes.
- Multiple rain layers: The model therefore groups streaks with similar shape or depth into separate layers combined with an unpolluted background.Each R_i represents one rain streak layer, and n is the total number of different layers.
- Atmospheric effects: Heavy rain can introduce attenuation, scattering, haze, and blur, further polluting the observed image.Scattering affects brightness and can produce haze or fog-like effects, while distant streaks may become blurry because they occupy only part of a pixel.
- Atmospheric effects: The generalized model includes global atmospheric light A, scene transmission α0, and brightness coefficients αi for rain or haze layers.The αi coefficients represent the brightness of individual rain streak or haze layers.
4 Deraining Method
RESCAN directly maps rainy images to rain streak layers and removes them iteratively using contextual dilated convolutions, SE weighting, and recurrent feature connections across stages.
- Deraining Method: RESCAN learns to map the observed rain image directly to rain streak layers, then subtracts the estimated rain to recover the background.The formulation treats rain streaks as sparser and simpler in texture than the clean background.
- Deraining Method: SCAN is the non-recurrent base model, extending Context Aggregation Net with Squeeze-and-Excitation blocks.SCAN provides the forward architecture underlying RESCAN.
- Deraining Method: Dilated convolutions provide broad contextual information, with exponentially increasing dilation in intermediate layers and no batch normalization.Removing batch normalization also saves approximately 40% of training memory and can improve performance.
- Deraining Method: SE blocks update channel weights for each image so feature-map channels can represent rain streak layers with different properties.The model assigns different alpha-values to rain streak-layer embeddings rather than using fixed weights.
- Deraining Method: The recurrent design decomposes rain removal into stages, where each stage outputs an intermediate rain-free image and later stages use previous-stage information.For the s-th stage, S denotes the number of stages and R_s denotes its rain output.
- Deraining Method: RNN memory connections link features across stages, allowing information from earlier deraining stages to guide feature learning in later stages.The paper evaluates ConvRNN, ConvGRU, and ConvLSTM variants; ConvGRU is described in detail.
- Deraining Method: The network supports additive prediction of residual rain or full prediction of the entire rain streak layer at each stage.Both frameworks use the observed image and previous states as inputs but produce different stage outputs.
5 Experiments
Experiments evaluate SCAN and RESCAN against state-of-the-art deraining methods on synthetic and real-world datasets, using separate objective and visual evaluation settings.
- Experiments: The experiments compare SCAN and RESCAN with state-of-the-art methods on both synthetic and real-world datasets.The study presents experimental settings and quality measures for evaluating the proposed models.
- Experiments: Synthetic rainy datasets are used for training because large paired real-world clean/rainy datasets are difficult to obtain.Rain800 contains 800 synthesized rain images, while Rain100H is selected as the most difficult dataset among several synthesized datasets.
Real-world Dataset
Real-world rain datasets provide diverse image content and rain intensity and orientation, but lack rain-free ground truth, so evaluation relies on visual inspection.
- Real-world Dataset: Real-world datasets contain diverse image content and variation in rain-streak intensity and orientation.These datasets are used for objective evaluation in the supplied passage.
- Real-world Dataset: Training uses randomly generated 64 × 64 patch pairs from training images and optimizes the network with ADAM.The supplied settings use 100 patch pairs per training image pair and a batch size of 64.
- Real-world Dataset: Real-world deraining performance is evaluated visually because rain-free ground-truth images are unavailable.PSNR and SSIM are used for synthetic image pairs instead.
5.2 Results on Synthetic Data
On synthetic datasets, RESCAN delivers the strongest quantitative deraining performance and preserves background details on difficult examples. Its recurrent refinement further improves over the non-recurrent SCAN model.
- RESCAN considerably outperforms other methods on both PSNR and SSIM across the Rain800 and Rain100H datasets.SCAN also surpasses JORDER and DetailsNet and is slightly superior to JORDER-R.
- Recurrent refinement further improves performance by gradually recovering the full rain streak layer R.
- On difficult synthetic images, the proposed method removes most rain streaks while maintaining background-scene details.Other state-of-the-art methods leave rain streaks or blur the image.
5.3 Results on Real-world Dataset
The paper evaluates deraining on real-world rainy images using visual comparisons and a user study because clean ground-truth images are unavailable. The proposed method removes most streaks while retaining substantial background texture.
- On real-world rainy images, the proposed method removes most rain streaks and maintains much of the background texture.LP removes rain inefficiently, whereas DetailsNet tends to add artifacts.
- Real-world performance is evaluated visually because clean rain-free ground-truth images are unavailable.The authors also conduct a user study, with details provided in supplementary material.
5.4 Analysis on SCAN
SCAN is analyzed as the base architecture through controlled comparisons with alternative network designs and normalization variants. The results support contextual architecture choices and the contribution of squeeze-and-excitation blocks.
- SCAN is compared with Plain, ResNet, and Encoder-Decoder architectures using matched depth, width, parameter scale, and computation order.
- The comparison between SCAN and CAN indicates that the SE block contributes substantially to the base model.SE explicitly learns an alpha-value for each independent rain streak layer.
- Removing batch normalization is reported as beneficial for deraining because rain-streak layers are independent.
5.5 Analysis on RESCAN
RESCAN ablations compare recurrent units and prediction frameworks, emphasizing the value of preserving information across deraining stages. Full Prediction performs better among the recurrent frameworks, while ConvGRU and ConvLSTM outperform ConvRNN.
- The experiments cover ConvRNN, ConvLSTM, and ConvGRU combined with Additive or Full Prediction, plus the Iter setting.
- Iter cannot compete with the recurrent structures or SCAN because it discards information from previous stages.
- ConvGRU and ConvLSTM outperform ConvRNN, while their relative performance is difficult to distinguish because they perform similarly.The paper attributes their advantage to maintaining more parameters and requiring more computation.
- Full Prediction performs better than the Additive Prediction framework.
6 Conclusions
The paper proposes a recurrent squeeze-and-excitation context aggregation network for single-image deraining, combining staged removal with recurrent guidance across stages. Experiments on synthetic and real-world datasets show that it outperforms state-of-the-art approaches under all evaluation metrics.
- Architecture: The network divides rain removal into multiple stages and uses a context aggregation network at each stage.The method uses an adapted CAN with exponentially increasing dilation and removes the batch-normalization layer.
- Architecture: Squeeze-and-excitation blocks assign different alpha-values to rain-streak layers according to their properties.The design targets differences in rain-streak intensities and characterizes dependencies among channels.
- Architecture: An RNN uses useful information from previous stages to guide learning in later stages.The paper also evaluates different network architectures and recurrent units.
- Evaluation: The proposed method outperforms state-of-the-art approaches on synthetic and real-world datasets under all evaluation metrics.