Source-linked AI summary
Uncertainty Guided Multi-Scale Residual Learning-using a Cycle Spinning CNN for Single Image De-Raining
Rajeev Yasarla, Vishal M. Patel
TL;DR
Single-image de-raining must handle rain streaks that vary in size, direction, and density, while prior approaches may not account for rain-drop locations. UMRL learns multi-scale residual and confidence maps and incorporates cycle spinning during training and testing. Experiments on synthetic and real images report improved de-raining performance, including PSNR 24.59 and SSIM 0.87 versus RESCAN's 24.37 and 0.84 on Rain800.
Problem
Single-image de-raining is difficult because rain streaks vary in size, direction, and density, and prior methods may not consider rain-drop location information.
Method
UMRL learns residual maps and confidence or uncertainty maps at multiple scales, then uses cycle spinning in training and testing to guide estimation and improve de-raining.
Results
24.59 PSNR and 0.87 SSIM were achieved by UMRL versus 24.37 PSNR and 0.84 SSIM for RESCAN on the Rain800 dataset.
Takeaways & Limitations
UMRL was reported to handle different levels of rain content for both synthetic and real-world rainy images.
Takeaways & Limitations
Compared methods may introduce artifacts or remove important image content when their priors overlook the locations of rain drops.
Abstract
from arXiv · showhide
Single image de-raining is an extremely challenging problem since the rainy image may contain rain streaks which may vary in size, direction and density. Previous approaches have attempted to address this problem by leveraging some prior information to remove rain streaks from a single image. One of the major limitations of these approaches is that they do not consider the location information of rain drops in the image. The proposed Uncertainty guided Multi-scale Residual Learning (UMRL) network attempts to address this issue by learning the rain content at different scales and using them to estimate the final de-rained output. In addition, we introduce a technique which guides the network to learn the network weights based on the confidence measure about the estimate. Furthermore, we introduce a new training and testing procedure based on the notion of cycle spinning to improve the final de-raining performance. Extensive experiments on synthetic and real datasets to demonstrate that the proposed method achieves significant improvements over the recent state-of-the-art methods. Code is available at: https://github.com/rajeevyasarla/UMRL--using-Cycle-Spinning
1. Introduction
Single-image de-raining is difficult because rain streaks vary in size, direction, and density, while prior methods may overlook their locations. UMRL addresses this with multi-scale residual and uncertainty learning, combined with cycle spinning and evaluations on synthetic and real images.
- Single-image de-raining is challenging because rain streaks vary in size, direction, and density.
- Prior approaches using image-level priors may overlook rain-drop locations and introduce artifacts or remove important image content.
- UMRL estimates rain streak content and uncertainty maps at multiple image locations and scales to guide subsequent network layers.
- Cycle spinning is incorporated during training and testing to improve de-raining performance and reduce edge artifacts.
- Sample comparisons report that UMRL removes noise artifacts and provides better results than DID-MDN and Fu et al.
- UMRL is evaluated against recent state-of-the-art methods on synthetic and real rainy images, with an ablation study of its components.
2. Background and Related Work
The paper models a rainy image as a clean image plus a rain residual and reviews prior dictionary, GMM, low-rank, and deep-learning approaches. UMRL estimates residuals and confidence maps to produce the de-rained image.
- A rainy observation is modeled as y = x + r, where x is the clean image and r is the rain component.
- Image de-raining can estimate the rain residual first and subtract it from the observed image to recover the clean image.
- Prior methods include dictionary learning, Gaussian mixture models, low-rank representations, and deep-learning approaches.
- UMRL uses residual and confidence maps to encode rain information and guide later layers of the network.
3. Proposed Method
UMRL estimates rain residuals and confidence maps across multiple scales, then combines these signals to guide de-raining. Confidence-guided training and cycle spinning are used to reduce estimation and edge artifacts.
- UMRL first estimates the rain-streak residual map ˆr and then computes the de-rained image as ˆx = y −ˆr.
- Residual and confidence information are estimated at three scales and passed to subsequent higher-scale layers to preserve location-based rain information.The scales are original size, 0.5 input size, and 0.25 input size.
- Residual maps at scales 1.0, 0.5, and 0.25 retain the same rain-streak direction and density at each image location.
- The confidence map estimates pixelwise certainty about residual values, while elementwise residual-confidence products are upsampled into subsequent network layers.
- Loss for UMRL: Confidence-guided training combines an L1 loss, a confidence regularizer, and a VGG-16 perceptual loss to avoid the trivial zero-confidence solution and improve visual quality.The overall loss is L = Ll − λ1Lc + λ2Lp.
- Cycle Spinning: Cycle spinning cyclically shifts images, applies UMRL, unshifts the outputs, and averages them to remove artifacts introduced by the original network.The procedure is illustrated for shifted inputs and corresponding de-rained outputs.
4. Experimental Results
Experiments evaluate UMRL with cycle spinning on synthetic and real rainy images, using quantitative metrics for synthetic data and visual inspection for real data. The method is compared with recent approaches, ablated by component, and assessed for runtime and robustness across rain conditions.
- Evaluation setup: PSNR and SSIM are used to compare methods on synthetic images, while real-image performance is visually inspected because clean ground truth is unavailable.The evaluation includes Test-1 and Test-2 synthetic datasets and real-world rainy images.
- Ablation study: Ablation experiments add network components progressively to a Unet-based base network to assess their contributions to the final de-rained image.The study compares BN, BN+RN, and UMRL configurations and separately evaluates cycle spinning.
- Quantitative results: UMRL with cycle spinning is quantitatively compared against state-of-the-art methods, including RESCAN on the Rain800 dataset.RESCAN achieves PSNR 24.37 and SSIM 0.84, whereas UMRL achieves PSNR 24.59 and SSIM 0.87.
- Quantitative results: UMRL achieves higher PSNR and SSIM than RESCAN on Rain800.The reported values are PSNR 24.59 and SSIM 0.87 for UMRL, compared with PSNR 24.37 and SSIM 0.84 for RESCAN.
- Synthetic test images: Qualitative synthetic results show UMRL avoiding over- and under-de-raining while preserving wall texture, clear skies, and sharp building and water-tank edges.Competing methods are described as removing texture, blurring edges, or leaving residual streaks.
- Runtime: For a 512×512 image, UMRL takes about 0.05 seconds on average, while UMRL with cycle spinning takes about 5.1 seconds.Cycle spinning substantially increases the reported processing time.
- Real-world rainy images: Real-image experiments show UMRL removing rain streaks while preserving object details across low, medium, and high rain levels and different shapes and scales.Examples include clearer faces, clothing details, trees, plants, and backgrounds.
5. Conclusion
The conclusion presents UMRL with cycle spinning as a single-image de-raining method that combines multi-scale residual and confidence estimation. Experiments report robustness across rain levels on synthetic and real-world images.
- 5. Conclusion: UMRL estimates residual maps and corresponding confidence maps at multiple scales, feeding them back to guide subsequent network layers.The method uses uncertainty-guided residual learning within a cycle-spinning framework.
- 5. Conclusion: Experiments report that UMRL is robust to different rain levels in both synthetic and real-world rainy images.The conclusion summarizes the method's reported evaluation scope.