Source-linked AI summary
Spatial Attentive Single-Image Deraining with a High Quality Real Rain Dataset
Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang Zhang, Rynson Lau
TL;DR
Single-image deraining lacks realistic training data and quantitative real-rain benchmarks, limiting objective evaluation. The paper constructs a semi-automatic real-rain dataset and proposes SPANet, which removes diverse rain streaks through local-to-global spatial attention. Experiments report improved derainer performance and favorable comparisons with state-of-the-art methods, while haze-like heavy rain remains a limitation.
Problem
Single-image deraining lacks realistic real-rain training data and high-quality benchmarks for quantitative evaluation.
Method
The paper uses temporal rain properties and human supervision to construct a real-rain dataset, then applies SPANet for local-to-global attentive deraining.
Results
The dataset contains ∼29.5K rain/clean image pairs, and experiments show improved state-of-the-art derainer performance and favorable SPANet comparisons.
Takeaways & Limitations
Real-rain data and spatially attentive modeling provide the paper's supported approach to evaluating and removing diverse rain streaks.
Takeaways & Limitations
The method fails on haze-like heavy rain because the dataset-generation process cannot select clean pixels from misty video frames.
Abstract
from arXiv · showhide
Removing rain streaks from a single image has been drawing considerable attention as rain streaks can severely degrade the image quality and affect the performance of existing outdoor vision tasks. While recent CNN-based derainers have reported promising performances, deraining remains an open problem for two reasons. First, existing synthesized rain datasets have only limited realism, in terms of modeling real rain characteristics such as rain shape, direction and intensity. Second, there are no public benchmarks for quantitative comparisons on real rain images, which makes the current evaluation less objective. The core challenge is that real world rain/clean image pairs cannot be captured at the same time. In this paper, we address the single image rain removal problem in two ways. First, we propose a semi-automatic method that incorporates temporal priors and human supervision to generate a high-quality clean image from each input sequence of real rain images. Using this method, we construct a large-scale dataset of $\sim$$29.5K$ rain/rain-free image pairs that covers a wide range of natural rain scenes. Second, to better cover the stochastic distribution of real rain streaks, we propose a novel SPatial Attentive Network (SPANet) to remove rain streaks in a local-to-global manner. Extensive experiments demonstrate that our network performs favorably against the state-of-the-art deraining methods.
1. Introduction
Single-image deraining remains difficult because synthetic data inadequately models real rain and quantitative real-rain benchmarks are lacking. The paper addresses these gaps with a real-rain dataset and SPANet, a local-to-global attentive derainer.
- Rain reduces visibility and can significantly affect pedestrian detection, visual tracking, and road sign recognition.
- Existing synthesized datasets lack the ability to model a large range of real-world rain streaks, including varied density and appearance.
- High-quality real deraining benchmarks are still needed for quantitative evaluation, which currently relies mainly on visual comparisons.
- The proposed semi-automatic method combines temporal rain properties and human supervision to generate clean images from sequences of real rain images.
- SPANet learns discriminative deraining features locally and globally to handle diverse real rain streak appearances within a single image.
- The dataset contains ∼29.5K high-resolution rain/clean image pairs covering a wide range of natural rain scenes and improving state-of-the-art derainers on real rain images.
2. Related works
Prior single-image methods model rain using handcrafted or learned representations, while multi-image methods exploit temporal information. The paper argues that existing multi-image approaches cannot reliably generate high-quality rain-free images under violated assumptions.
- Single-image rain removal: Single-image deraining is ill-posed because the input rain image, rain streak layer, and derained background must be separated.
- Single-image rain removal: Earlier single-image methods use frequency decomposition, filtering, sparse coding, Gaussian mixtures, affine transformations, and deep networks to remove rain streaks.
- Multi-image rain removal: Multi-image methods use temporal information and photometric or chromatic rain properties to estimate backgrounds and detect rain streaks.
- Multi-image rain removal: Existing multi-image methods can over- or under-derain entire sequences when assumptions such as low-rank structure are violated, burying the true background radiance.
- Generating the ground truth from real noisy images: Capturing rain-free images through exposure changes fails because longer exposure enlarges fast-falling rain streaks rather than removing them.
3. Real Rain Image Dataset
The dataset pipeline estimates clean backgrounds from real-rain image sequences using temporal pixel statistics, percentile ranges, spatial smoothness, and human evaluation. It produces high-quality rain/clean pairs at scale, while comparisons assess the method against video deraining approaches.
- Temporal background estimation: Pixel intensities fluctuate less when a pixel is uncovered than when rain covers it, supporting temporal background estimation.Users label pixel coverage across frames, revealing distinct intensity behavior for rain-covered and uncovered states.
- Clean image generation: The pipeline computes each pixel’s mode and percentile range, then selects a percentile crossing the most ranges to globally smooth the estimated background.This spatially informed selection reduces noise that occurs when modes are used independently, especially under dense rain.
- Qualitative comparison: The mode-based result can be noisy, whereas the proposed globally smoothed method produces a cleaner image in the demonstrated example.Figure 4 compares the best result from each method on a synthetic 100-frame rain video.
- Sequence selection: The method assumes every output pixel has some input frames without rain coverage, so the sequence length N is adjusted for rain density.Initial values are N={20,100,200} for sparse, normal, and dense rain, with additional frames added after human evaluation when needed.
- Quality control: Human judgments are used to evaluate generated images because people can detect both rain streaks and artifacts such as noise.The authors contrast this supervision with averaging or selecting outputs from video deraining methods, which may leave rain streaks incompletely removed.
- Dataset construction: 29,500 high-quality rain/clean image pairs are generated from 170 real rain videos, with 28,500 training pairs and 1,000 testing pairs.The videos include captured and online scenes spanning urban, suburban, and outdoor environments, with varied exposure and ISO settings.
4. Proposed Model
SPANet removes rain streaks through a local-to-global spatially attentive architecture that progressively identifies rain regions and reconstructs the clean background. Its training combines reconstruction, structural-similarity, and explicitly supervised attention losses.
- Architecture: SPANet uses residual blocks for feature extraction and background reconstruction, with four spatial attentive blocks progressively identifying rain streaks.Each spatial attentive block contains spatial attentive residual blocks and a spatial attentive module; dilation convolutions are used in the residual components.
- Attention: SAM explicitly generates an attention map to highlight rain spatial distributions and guide the deraining process.The attention map identifies rain-affected regions despite substantial variation in streak appearance.
- Global context: The two-round four-directional IRNN gathers neighborhood information first and global contextual information second.The architecture propagates information in the up, left, down, and right directions before repeating the process to cover the entire input feature map.
- Removal-via-detection: Given a rain image, SPANet uses SAM-guided spatial attentive residual blocks to remove rain through learned negative residuals.The spatial attentive block is repeated four times, with shared SAM weights across the four blocks, before final reconstruction.
- Training: SPANet is trained with total loss Ltotal = L1 + LSSIM + LAtt.L1 measures per-pixel reconstruction accuracy, LSSIM constrains structural similarity, and LAtt supervises the attention map against a rain-streak binary map.
- Training: Training uses Adam with batch size 16 for 40K iterations, with scaling and cropping for rain-streak augmentation.The learning rate starts at 0.005 and is divided by 10 after 30K iterations.
5. Experiments
Experiments evaluate the proposed real-rain dataset, compare SPANet with state-of-the-art derainers, and analyze its spatial-attention components. The dataset improves existing CNN derainers on real images, while SPANet produces cleaner results and handles varied rain appearances.
- Experimental setup: The experiments first assess the proposed dataset on CNN derainers, then compare SPANet with state-of-the-art single-image deraining methods.Internal analysis studies the contributions of individual SPANet components.
- Dataset evaluation: The real dataset significantly improves CNN-based derainers on real images because synthesized datasets poorly represent highly varying rain streaks.Visual comparisons show cleaner outputs after retraining on the proposed dataset.
- Dataset evaluation: State-of-the-art derainers can over-derain, causing input rain images to achieve similar or higher average PSNR and SSIM than derained results.Difference maps indicate that brighter regions correspond to larger differences and that several methods corrupt background content.
- SPANet evaluation: SPANet achieves superior deraining performance compared with state-of-the-art derainers on the proposed test set.Visual comparisons show competing methods either leave rain streaks or corrupt the background, whereas SPANet produces cleaner results.
- SPANet evaluation: On real images with diverse rain appearances, SPANet removes streaks while preserving more background details than existing derainers.Examples include rain images collected from previous deraining papers and the Internet.
- Internal analysis: All tested SAM variants improve performance, while the shared-weight four-SAM variant performs best in the internal analysis.The attention branch leverages directional local context, and explicit attention supervision improves attention-map generation.
- Limitations: The method fails to remove extremely dense rain streaks.This failure case defines a practical boundary of the reported deraining capability.
6. Conclusion and Future Work
The paper combines temporal priors and human supervision to build a large real-rain dataset and introduces SPANet for local-to-global rain removal. Experiments report improved derainer performance and superiority over state-of-the-art methods, while identifying failures on haze-like heavy rain and reliance on human judgments.
- ∼29.5K rain/clean image pairs cover a wide range of natural rain scenes and significantly improve state-of-the-art CNN-based derainers on real rain images.
- SPANet identifies and removes rain streaks in a local-to-global spatial attentive manner.
- Extensive evaluations demonstrate the superiority of SPANet over state-of-the-art derainers.
- The method fails on haze-like heavy rain because dataset generation cannot select clean pixels from misty video frames, producing haze-like results.
- Dataset generation relies on human judgments because no existing metrics assess generated rain-free images without clean reference images.