Source-linked AI summary
Density-aware Single Image De-raining using a Multi-stream Dense Network
He Zhang, Vishal M. Patel
TL;DR
Non-uniform rain density makes single-image rain-streak removal difficult, while existing approaches can over- or under-de-rain and density-specific models require known labels. DID-MDN jointly estimates rain density and removes streaks with a residual-aware classifier and multi-stream dense network, achieving significant improvements over recent state-of-the-art methods on synthetic and real-world datasets.
Problem
Non-uniform rain densities, shapes, and scales make single-image de-raining difficult, and existing approaches may over- or under-de-rain images.
Method
DID-MDN automatically estimates heavy, medium, or light rain density with a residual-aware classifier and guides multi-stream densely connected de-raining using multi-scale features.
Results
DID-MDN significantly outperforms many recent state-of-the-art methods on two synthetic and one real-world dataset.
Takeaways & Limitations
The method jointly estimates density and de-rains images, while its density-labeled dataset supports training and the network generalizes well to real-world rainy images.
Takeaways & Limitations
The main-paper presentation shows only the most recent state-of-the-art comparisons, with additional method results deferred to supplementary material.
Abstract
from arXiv · showhide
Single image rain streak removal is an extremely challenging problem due to the presence of non-uniform rain densities in images. We present a novel density-aware multi-stream densely connected convolutional neural network-based algorithm, called DID-MDN, for joint rain density estimation and de-raining. The proposed method enables the network itself to automatically determine the rain-density information and then efficiently remove the corresponding rain-streaks guided by the estimated rain-density label. To better characterize rain-streaks with different scales and shapes, a multi-stream densely connected de-raining network is proposed which efficiently leverages features from different scales. Furthermore, a new dataset containing images with rain-density labels is created and used to train the proposed density-aware network. Extensive experiments on synthetic and real datasets demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods. In addition, an ablation study is performed to demonstrate the improvements obtained by different modules in the proposed method. Code can be found at: https://github.com/hezhangsprinter
1. Introduction
Single-image de-raining must handle rain with varying densities, shapes, and scales without damaging image content. DID-MDN addresses this by estimating density automatically, guiding multi-stream rain removal, and evaluating the approach across synthetic and real datasets.
- Single-image de-raining is important because rain and other artifacts degrade computer-vision systems in applications such as autonomous driving and drone surveillance.
- Existing methods may overlook rain-drop shapes, scales, and densities, causing over-de-raining that removes image content or under-de-raining that leaves rain streaks.Fu et al.'s method can remove important content, while Li et al.'s method can leave residual streaks.
- A single network trained across many rain conditions may fail to learn all variations, whereas density-specific models require the density label before selecting a network.
- DID-MDN automatically classifies rain as heavy, medium, or light, then fuses the estimated density with a multi-stream densely connected network for rain-streak removal.The classifier uses residual information, while the removal network accounts for distinct rain-streak scales and shapes.
- The authors synthesize 12,000 labeled training images and report experiments on two synthetic and one real-world dataset, including comparisons and ablation studies.The dataset includes rain-density labels for heavy, medium, and light conditions.
2. Background and Related Work
Single-image de-raining is ill-posed because a rainy image combines clean background and rain components, motivating learned and multi-scale feature representations. DID-MDN combines density classification with multi-stream dense feature processing to remove rain streaks using estimated density information.
- 2.1. Single Image De-raining: A rainy image is modeled as y = x + r, so recovering the clean image x from y is an ill-posed single-image de-raining problem.
- 2.1. Single Image De-raining: Prior-based methods include sparse coding, low-rank representation, and GMM approaches, but some can over-smooth image details.
- 2.1. Single Image De-raining: CNN-based methods learn mappings from rainy inputs to corresponding ground truths for image de-raining.
- 2.2. Multi-scale Feature Aggregation: Combining convolutional features across scales improves representation of objects and context, motivating skip connections and multi-resolution feature aggregation.Examples include FCN and U-Net-style architectures.
- 2.2. Multi-scale Feature Aggregation: DID-MDN uses a residual-aware density classifier and a multi-stream densely connected de-raining network guided by estimated rain density.Its streams capture rain-streak components with different scales and shapes using dense blocks and different receptive fields.
3. Proposed Method
DID-MDN combines residual-aware rain-density classification with a multi-stream densely connected de-raining network. Estimated density labels guide rain-streak removal, while streams with different receptive fields capture varied streak scales and shapes.
- Residual-aware rain-density classifier: DID-MDN uses residual-aware classification to estimate whether the input contains heavy, medium, or light rain.The classifier first estimates a residual component and then uses it for rain-density classification, with the two stages jointly optimized.
- Motivation: A single network can over- or under-remove rain because it may not learn the different rain densities encountered in practice.The method incorporates density-level information to address this limitation and improve adaptation across rain conditions.
- Multi-stream dense network: The de-raining network uses three densely connected streams with 7 × 7, 5 × 5, and 3 × 3 kernels to capture different rain-streak scales and shapes.Smaller streaks are associated with smaller receptive fields, whereas longer streaks are associated with larger receptive fields.
- Multi-stream dense network: Features from all dense blocks and streams are concatenated, creating short paths that strengthen multi-scale feature aggregation for rain-streak estimation.Each stream contains six dense blocks, and the outputs are concatenated before estimating the rain-streak components.
- Label-guided de-raining: The up-sampled density label map is concatenated with rain-streak features, after which the estimated residual is subtracted from the rainy input and refined by two convolutional layers.The training objective combines residual estimation, de-raining reconstruction, and a weighted CNN feature-based loss.
- Inference: At test time, the classifier estimates the rain-density label before the multi-stream network produces the final de-rained image.The estimated label map and input image are jointly fed into the de-raining network.
4. Experimental Results
Experiments on synthetic and real-world images evaluate DID-MDN against recent methods and baseline configurations, while ablations examine density classification, label fusion, and multi-stream design. DID-MDN preserves details while removing rain streaks across varied densities, shapes, and scales, with about 0.3 seconds per 512 × 512 image.
- Experimental Setup: The study evaluates DID-MDN on synthetic and real-world datasets using PSNR and SSIM for synthetic images and visual comparisons for real-world images.Comparisons include recent state-of-the-art methods.
- Synthetic Dataset: 12,000 labeled training images and 1,200 test images span light, medium, and heavy rain with different orientations and scales.The synthetic dataset uses roughly 4,000 images per density level.
- Ablation Study: The proposed label-fused multi-stream network removes rain streaks while preserving background details, unlike baselines that over-de-rain or leave residual streaks.The comparison is supported by both qualitative observations and average PSNR and SSIM results on Test1.
- Synthetic Results: DID-MDN achieves superior quantitative performance on Test1 and Test2 and selectively demonstrates performance across all three synthetic rain-density conditions.Figure 7 presents examples from Test1 and Test2 alongside recent methods.
- Real-World Results: On real-world images, DID-MDN removes rain streaks while preserving details across heavy and medium rain and streaks with different shapes and scales.Examples include small round streaks and long-thin streaks.
- Running Time Comparisons: DID-MDN takes about 0.3s to de-rain a 512 × 512 image, with testing time comparable to DDN.The running-time comparison averages results over images of the stated size.
5. Conclusion
DID-MDN jointly estimates rain density and removes rain streaks using a density-aware multi-stream densely connected network. Experiments on two synthetic and one real-world dataset show significant improvements over many recent state-of-the-art methods, while baseline comparisons quantify gains from each module.
- DID-MDN jointly performs rain-density estimation and single-image de-raining.The method uses estimated rain-density labels to guide synthesis of the de-rained image.
- The method uses a residual-aware rain-density classifier and a multi-stream densely connected de-raining network.The classifier predicts density labels, which guide the multi-stream network's rain-streak removal.
- Experiments on two synthetic and one real-world dataset show significant improvements over many recent state-of-the-art methods.
- Baseline configurations demonstrate the performance gains obtained by each proposed module.