Source-linked AI summary
Multi-Scale Progressive Fusion Network for Single Image Deraining
Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, Junjun Jiang
TL;DR
Single-image deraining must handle rain streaks whose appearance varies with scale, while prior methods often overlook cross-scale correlations. MSPFN builds a Gaussian-pyramid framework with recurrent coarse fusion and attention-guided fine fusion to collaboratively represent streaks. It reports superior deraining results across synthetic and real-world data and evaluates downstream detection and segmentation tasks.
Problem
Single-scale deraining methods struggle to capture inherent correlations among rain streaks across scales, despite similar patterns carrying complementary information for representation.
Method
MSPFN progressively fuses Gaussian-pyramid rain information using recurrent coarse fusion, attention-guided fine fusion, and reconstruction modules within a unified framework.
Results
MSPFN achieves state-of-the-art deraining performance on synthetic and real-world datasets and is evaluated jointly with object detection and segmentation.
Takeaways & Limitations
Collaborative cross-scale rain-streak representation supports deraining evaluation across restoration and downstream vision tasks.
Abstract
from arXiv · showhide
Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of this correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task-driven image deraining. The source code is available at \url{https://github.com/kuihua/MSPFN}.
1. Introduction
Single-scale deraining methods struggle to capture correlations among rain streaks across scales. MSPFN addresses this by progressively fusing complementary multi-scale information through pyramid representations and dedicated fusion modules.
- Rain degradation substantially harms image content, motivating specialized deraining for tasks such as object detection and semantic segmentation.
- Traditional methods are not robust to rain streaks with varying directions, densities, and sizes, while single-scale deep models miss cross-scale correlations.
- Rain streaks at different positions and image scales provide complementary or redundant patterns that can characterize target streaks.
- Prior pyramid deraining methods either solve restoration separately at each resolution or progressively optimize density-specific stages without direct inter-level feature communication.
- MSPFN unifies input-scale and deep-feature representations, using recurrent coarse fusion and attention-guided fine fusion to integrate correlated information progressively.
- The framework combines initial feature extraction, coarse fusion, fine fusion, and rain streak reconstruction to regress the residual rain image.
2. Related Work
Single-image deraining remains challenging under complex rain conditions, and existing approaches only partially exploit multi-scale rain information. MSPFN progressively aggregates Gaussian-pyramid features to collaboratively represent rain streaks across scales.
- Single-image deraining is challenging because complex rain conditions can defeat traditional methods and degrade image contents.
- Existing methods include deep-learning restoration models and pyramid-based approaches, but their use of multi-scale correlations remains limited.
- Some prior pyramid methods decompose restoration into separate scale-specific subproblems, while others progressively remove rain without direct inter-level feature communication.
- MSPFN progressively aggregates multi-scale rain information extracted from Gaussian-pyramid images along pyramid layers and network stages.
- The resulting collaborative representation is intended to produce a more accurate predicted rain streak distribution through cross-scale information.
3. Proposed Method
MSPFN derains images by progressively extracting and fusing rain information across pyramid scales and hierarchical features. Its modules combine recurrent multi-scale fusion, attention-guided refinement, reconstruction, and losses targeting both content and high-frequency detail.
- Overall Pipeline: MSPFN processes Gaussian-pyramid rain images through parallel shallow features before multi-scale deep extraction and fusion.The pyramid includes down-sampled inputs such as 1/2 and 1/4 scales.
- Coarse Fusion: The coarse-fusion module uses residual recurrent units to extract and fuse multi-scale rain information.The method presents the overall MSPFN pipeline as exploiting correlations across scales, with CFM performing deep extraction and fusion.
- Fine Fusion: The fine-fusion module refines cross-scale information with channel attention focused on informative scale-specific knowledge.Strided convolution reduces feature spatial dimensions before later upsampling and fusion operations.
- Reconstruction: The reconstruction module combines low- and high-level features from CFM and FFM to learn the final residual rain image.It concatenates CFM outputs with the last FFM outputs and applies convolution to model channel interdependence and rescale feature values.
- Loss Function: Training uses a Charbonnier content loss and an additional edge loss to preserve high-frequency details in the predicted rain-free image.The Charbonnier penalty is used for successive approximation to the real rain-streak distribution, while edge loss constrains Laplacian high-frequency components.
4. Experiments and Discussions
Experiments evaluate MSPFN on synthetic and real-world deraining benchmarks, ablations, and downstream detection and segmentation tasks. The results show strong restoration quality, benefits from multi-scale fusion, and improved downstream performance, including for a lightweight variant.
- Experimental Setup: The experiments use unified retraining on about 13,700 clean/rain pairs and evaluate restoration with PSNR, FSIM, and SSIM.The study covers synthetic and real-world datasets and six competing deraining methods.
- Ablation Studies: MSPFN improves over incomplete variants in ablations, while multi-scale progressive fusion provides a 0.54dB gain over Model4 with acceptable complexity.Model6 also exceeds the single-scale Model1 by 0.32dB at similar computation complexity.
- Ablation Studies: Performance declines when FFM depth is reduced, whereas increasing CAU count yields only a 0.13dB improvement with 30% more parameters.The deeper MSPFNM30N1 achieves the best scores across all metrics, while MSPFNM5N1 is designed as a lightweight model.
- Synthetic Data: On Test1200, MSPFN surpasses DerainNet and DIDMDN by 9.01dB and 2.74dB in PSNR, respectively.The comparison covers five synthetic datasets and six top-performing deraining methods.
- Real-world Data: On real-world data, MSPFN obtains the best average NIQE/SSEQ scores on 200 samples and preserves realistic details while removing major rain streaks.Lower NIQE and SSEQ indicate better perceptual quality and clearer contents.
- Other Applications: MSPFN improves detection precision over rainy inputs by nearly 10% and achieves 52.96% mPA and 35.90% mIoU for semantic segmentation on BDD150.The lightweight MSPFN* remains competitive while targeting computation-efficient, real-time applications.
5. Conclusion
MSPFN uses a pyramid-based multi-scale progressive fusion mechanism to cooperatively represent rain streaks across scales. Experiments show superiority over other top-performing methods across deraining datasets, real-world scenarios, and downstream vision tasks.
- MSPFN cooperatively represents rain streaks across different scales through a pyramid framework and multi-scale progressive fusion.The framework includes CFM, FFM, and RM modules to explore correlations among similar rain patterns.
- Experimental results demonstrate MSPFN’s superiority on synthetic deraining datasets, real-world scenarios, object detection, and segmentation.