PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Image Deraining: From Model-Based to Data-Driven and Beyond

Wenhan Yang, Robby T. Tan, Shiqi Wang, Yuming Fang, Jiaying Liu

arXiv:1912.07150v2eess.IVcs.CV

TL;DR

Single-image deraining seeks to recover rain-free backgrounds despite occlusion, veiling, and the resulting ill-posed inverse problem. This survey synthesizes rain appearance models and model-based and data-driven methods, then reviews their development, evaluation, limitations, and future directions. It reports that data-driven methods generally perform better than model-based methods, while current evaluation metrics remain poorly aligned with human perception.

  • Problem

    Single-image deraining must estimate background scenes from rain-degraded images whose rain and background information are fused and can be completely occluded.

  • Method

    The paper surveys rain appearance models and organizes deraining literature into model-based and data-driven approaches, covering methods, architectures, constraints, losses, datasets, and comparisons.

  • Results

    Data-driven methods generally perform better than model-based methods in the survey’s comparisons.

  • Takeaways & Limitations

    The survey identifies physics and real-rain integration, more accurate rain models, evaluation methodology, and real applications as open problems.

  • Takeaways & Limitations

    Current rain models cover limited streak diversity, and methods tend to fail when captured-rain distributions differ from synthetic training images.

Abstract

from arXiv · show

The goal of single-image deraining is to restore the rain-free background scenes of an image degraded by rain streaks and rain accumulation. The early single-image deraining methods employ a cost function, where various priors are developed to represent the properties of rain and background layers. Since 2017, single-image deraining methods step into a deep-learning era, and exploit various types of networks, i.e. convolutional neural networks, recurrent neural networks, generative adversarial networks, etc., demonstrating impressive performance. Given the current rapid development, in this paper, we provide a comprehensive survey of deraining methods over the last decade. We summarize the rain appearance models, and discuss two categories of deraining approaches: model-based and data-driven approaches. For the former, we organize the literature based on their basic models and priors. For the latter, we discuss developed ideas related to architectures, constraints, loss functions, and training datasets. We present milestones of single-image deraining methods, review a broad selection of previous works in different categories, and provide insights on the historical development route from the model-based to data-driven methods. We also summarize performance comparisons quantitatively and qualitatively. Beyond discussing the technicality of deraining methods, we also discuss the future directions.

1 INTRODUCTION

Single-image deraining restores rain-free backgrounds from visibility degradation caused by rain streaks and accumulation. The field progressed from model-based priors and optimization to data-driven deep learning, while paired-data limitations motivate unsupervised approaches.

  • Motivation: Rain streaks occlude backgrounds, while rain accumulation forms a veil that reduces scene contrast and visibility.These degradations impair human vision and computer-vision algorithms designed for clear weather.
  • Historical development: Kang et al. pioneered single-image deraining in 2012 by separating high-frequency rain and non-rain components with dictionary learning and sparse coding.
  • Historical development: Before 2017, model-based methods used image decomposition, sparse coding, Gaussian mixture models, handcrafted priors, cost functions, and optimization.These priors represent rain streaks and background scenes statistically.
  • Historical development: Since 2017, deep-learning methods using architectures such as convolutional, recurrent, recursive, and generative-adversarial networks have become dominant.More than 30 deep-learning deraining papers appeared during 2017–2019, exceeding the number published before 2017.
  • Open challenge: Fully supervised training is limited by the intractability of paired rain and rain-free images, while synthetic rain introduces a domain gap from real rain.Unsupervised and semi-supervised methods therefore exploit real rain images.
  • Survey scope: The survey organizes single-image deraining methods, emphasizing recent deep-learning architectures, constraints, losses, datasets, quantitative comparisons, qualitative comparisons, and future directions.

2 RAINDROP APPEARANCE MODELS

Raindrop appearance is modeled as environmental radiance transferred through refraction, specular reflection, and internal reflection. Moving drops form streaks whose intensity changes depend on drop brightness, background radiance, and camera exposure.

  • Raindrop appearance: A raindrop’s radiance is approximated by contributions from refracted, specularly reflected, and internally reflected rays.These rays direct environmental light toward the observer.
  • Raindrop appearance: The radiance transfer functions R, S, and P represent the fractions of incident environmental radiance reaching the camera after refraction, reflection, and internal reflection.
  • Rain streaks: For a moving drop, the resulting rain streak appearance depends on drop brightness, background-scene radiance, and camera exposure time.
  • Rain streaks: The rain-streak intensity change correlates linearly with background intensity, and in most real cases the alpha term dominates its appearance.The cited empirical bounds are 0 < beta < 0.039 and 0 < tau < 1.18.
  • Synthetic modeling: Because the dominant alpha contribution simplifies modeling, most synthetic rain models assume rain streaks are superimposed on the background image.

3 LITERATURE SURVEY

The survey traces rain synthesis and single-image deraining from heuristic degradation models and optimization-based methods toward data-driven approaches, while outlining persistent modeling, ill-posedness, and evaluation challenges.

  • 3.1 Synthetic Rain Models: The additive composite model superimposes rain streaks on the background, whereas the screen-blend model models signal-dependent interaction between the two layers.The screen-blend formulation is intended to capture visual properties such as internal reflections and generate more authentic rain images.
  • 3.1 Synthetic Rain Models: Synthetic rain models range from additive and screen-blend formulations to heavy-rain, occlusion-aware, comprehensive, and depth-aware models.The models represent increasingly varied interactions among rain streaks, accumulation, atmospheric effects, depth, and occlusion.
  • 3.1 Synthetic Rain Models: Heavy-rain and depth-aware models cover the most comprehensive degradation for single-image rain synthesis, although the reviewed models are heuristic and not fully physically verified.Despite this limitation, the literature reports that these models are effective to some extent for image deraining.
  • 3.2 Deraining Challenges: Single-image deraining is ill-posed because observed intensities fuse rain and background information, and dense rain can totally occlude background content.Rain and background properties may overlap in feature space, making suitable priors difficult to find and causing background textures to be mistaken for rain.
  • 3.2 Deraining Challenges: Deep-learning methods face limited real paired ground truths, so synthetic training and evaluation data remain separated from real rain by significant domain gaps.Unsupervised and semi-supervised methods are introduced to exploit real rain images and address this problem.
  • 3.3.2 Deep Learning Based Methods: GAN-based methods improve visual properties by removing rain accumulation more thoroughly and producing a more balanced luminance distribution, but they do not properly model diversified real streak details.The survey attributes the detail limitation to GAN-based methods' difficulty capturing fine-grained signals.

4 TECHNICAL DEVELOPMENT REVIEW

The survey reviews deep-learning deraining through architectures, assumptions, basic blocks, loss functions, and image-processing-inspired designs. Recent methods increasingly use recurrent, multi-path, recursive, attention-based, and multi-scale structures to progressively separate rain from background.

  • Overview: Deep-learning deraining development is organized around network architectures, basic blocks, loss functions, and datasets that influence learning capacity and performance.The survey also frames architectures as relying on assumptions, constraints, and general image-processing knowledge.
  • Assumptions/Constraints: Rain- and background-related assumptions, including detail/base decomposition, rain detection, density estimation, scale diversity, depth, and confidence maps, guide feature learning and layer separation.Confidence maps weight wrongly estimated pixels during back-propagation, directing training toward difficult regions.
  • Image Processing Knowledge: Image-processing ideas such as wavelets, Laplacian pyramids, and multi-scale structures support scale-aware or lightweight deraining networks.PyramidDerain combines a Gaussian Laplacian pyramid with a deep network, reducing depth and parameters while achieving comparable state-of-the-art performance.
  • Recursive Architectures: Recursive and recurrent architectures process inputs and intermediate results progressively, forwarding information across stages or recurrences to generate clean outputs.Examples include recurrent units such as GRU and LSTM and stage-wise processing in PReNet.
  • Basic Blocks and Architectures: Newer methods build more complex basic blocks and stack them into networks using residual, dense, cascaded, recursive, attention, and U-Net designs.Multi-path architectures gather information through differing receptive fields, kernel sizes, dilation factors, or filter directions.
  • Loss Functions: Deraining training uses fidelity-driven losses such as MSE, MAE, and SSIM, alongside other proposed loss functions.The survey introduces SSIM as a fidelity-driven metric and notes that multiple loss functions regularize deraining-network training.

5 PERFORMANCE SUMMARY

The survey compares model-based and deep-learning deraining methods using full-reference, non-reference, visual, subjective, and computational evaluations. Deep-learning methods generally outperform model-based methods quantitatively, but metric and domain-gap issues complicate conclusions about real-image quality.

  • Evaluation Setup: The evaluation covers methods spanning image decomposition, sparse coding, Gaussian mixture priors, deep networks, and generative adversarial learning across several rain datasets.Training datasets include Rain100H, RealDataset, Rain800, NYU-Rain, and Outdoor-Rain.
  • Evaluation Setup: PSNR and SSIM provide full-reference evaluation, while non-reference metrics assess visual quality when clean ground truths are unavailable.Non-reference measures include NIQE, PIQE, BRISQUE, IL-NIQE, SSEQ, and related quality-assessment metrics.
  • Quantitative Evaluation: DDN exceeds model-based performance by more than 3dB on Rain100L, 7dB on Rain100H, and 0.7dB on Rain1400.The survey reports that most deep-learning methods achieve significantly superior performance to model-based methods.
  • Quantitative Evaluation: The best objective performance gradually converges, with RESCAN, PReNet, and JORDER-E showing considerably close performance.Methods in the objective comparison are ordered by publication year, and PSNR and SSIM are plotted separately.
  • Qualitative Evaluation: JORDER-E and PReNet handle large rain streaks well, while JORDER-E and HeavyRainRemoval better remove rain accumulation and enhance visibility in the shown examples.These observations come from visual comparisons containing large streaks and dense rain accumulation.
  • Qualitative Evaluation: 2019 methods are on average superior on the dataset, but qualitative superiority does not match quantitative superiority because synthesized and real rain differ.The survey attributes this disagreement to the domain gap between real rain images and synthesized data.
  • Qualitative Evaluation: Non-reference metrics disagree with MOS and qualitative judgments; the best SROCC, KROCC, and PLCC values are only 0.2216, 0.1473, and 0.1864.The survey concludes that existing metrics are not suitable for measuring deraining performance and identifies evaluation as a future opportunity.
  • Computational Complexity: PReNet offers strong quantitative and qualitative performance while remaining lightweight, whereas JORDER-E, HeavyRainRemoval, and URML use many more parameters.Runtime comparisons use CPU implementations for sparse-coding methods and GPU acceleration for other methods, so implementation settings differ.

6 FUTURE DIRECTIONS

Future progress in single-image deraining depends on closing gaps between synthetic training and real rain, improving evaluation, and meeting real-application demands.

  • 6.2 Rain Modeling: Combining physics-based constraints with unpaired real rain images is a proposed way to train without paired clean backgrounds.A cited approach combines a physics model and a generative adversarial network to accept unpaired ground-truths.
  • 6.2 Rain Modeling: Synthetic rain models cover limited streak scales, shapes, and directions, creating gaps between training distributions and captured rain.Methods tend to fail when real and synthetic rain distributions differ.
  • 6.3 Evaluation Methodology: Existing quality assessments remain far from capturing human visual perception, motivating metrics for human preferences and task-driven machine-vision evaluation.MPID provides a preliminary task-driven evaluation set for traffic detection.
  • 6.4 More Related Tasks and Real Applications: Real applications require faster processing, robustness to compression distortion, and adaptive handling of composite degradations such as night rain and mixed raindrop-rain-streak scenes.Current methods are far from the 30 fps real-time requirement.

7 CONCLUDING REMARKS

The survey organizes single-image deraining into model-based and data-driven approaches and identifies unresolved challenges for future development.

  • The survey categorizes model-based methods into layer decomposition, sparse coding, and GMMs, while grouping data-driven methods into deep CNNs, GANs, and semi/unsupervised learning.
  • Data-driven methods generally perform better than model-based methods, but important open problems remain.
  • Future work is needed on combining physics models with real-rain images, improving rain models, evaluating deraining methods, and supporting real applications.
Loading 1912.07150v2…