Source-linked AI summary
Deep Joint Rain Detection and Removal from a Single Image
Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan
TL;DR
The paper addresses single-image rain removal under rain accumulation and heavy rain, where existing models inadequately represent important real-rain effects. It introduces region-dependent rain models and joint, recurrent detection-and-removal networks, with experiments reporting significant outperformance of state-of-the-art methods on synthesized and real data, especially some heavy-rain images.
Problem
Single-image rain removal must handle visibility degradation from rain accumulation, atmospheric veils, and varied streak shapes and directions that existing rain models inadequately represent.
Method
The paper models rain regions, accumulation, and overlapping streaks, then jointly detects and removes rain with multi-task and recurrent networks that progressively restore heavily rain-affected images.
Results
The method significantly outperforms state-of-the-art methods on synthesized and real data, with considerably good results on some heavy-rain images.
Takeaways & Limitations
Modeling rain accumulation and heavy-rain structure, while supplying detected rain regions, supports effective restoration of complex single-image rain scenes.
Takeaways & Limitations
Existing rain models inadequately cover atmospheric veils from rain accumulation and different streak shapes or directions, while streak-background overlap can cause texture removal and over-smoothing in non-rain regions.
Abstract
from arXiv · showhide
In this paper, we address a rain removal problem from a single image, even in the presence of heavy rain and rain streak accumulation. Our core ideas lie in the new rain image models and a novel deep learning architecture. We first modify an existing model comprising a rain streak layer and a background layer, by adding a binary map that locates rain streak regions. Second, we create a new model consisting of a component representing rain streak accumulation (where individual streaks cannot be seen, and thus visually similar to mist or fog), and another component representing various shapes and directions of overlapping rain streaks, which usually happen in heavy rain. Based on the first model, we develop a multi-task deep learning architecture that learns the binary rain streak map, the appearance of rain streaks, and the clean background, which is our ultimate output. The additional binary map is critically beneficial, since its loss function can provide additional strong information to the network. To handle rain streak accumulation (again, a phenomenon visually similar to mist or fog) and various shapes and directions of overlapping rain streaks, we propose a recurrent rain detection and removal network that removes rain streaks and clears up the rain accumulation iteratively and progressively. In each recurrence of our method, a new contextualized dilated network is developed to exploit regional contextual information and outputs better representation for rain detection. The evaluation on real images, particularly on heavy rain, shows the effectiveness of our novel models and architecture, outperforming the state-of-the-art methods significantly. Our codes and data sets will be publicly available.
1. Introduction
Single-image rain removal must address visibility loss from accumulated veils, overlapping streaks, and texture interference. The paper introduces region-dependent rain models and joint, recurrent networks designed to detect and progressively remove these effects.
- Motivation: Rain accumulation creates mist- or fog-like atmospheric veiling, while nearby streaks vary in shape and direction, especially in heavy rain.These effects scatter light, occlude backgrounds, and severely degrade visibility.
- Limitations of prior work: Existing methods can over-smooth non-rain regions because rain streaks overlap intrinsically with background texture patterns.Prior models also omit accumulation veils and varied streak shapes or directions.
- Proposed models: The paper models binary rain-streak regions, accumulated veils, and overlapping streak shapes and directions to better represent real heavy-rain conditions.The binary map marks pixels containing individually visible streaks and supports training-data synthesis.
- Proposed architecture: A joint network detects rain regions and removes rain, enabling adaptive processing of rain and non-rain areas while preserving richer details.The architecture uses multiple losses for rain detection and removal tasks.
- Proposed architecture: A contextualized dilated network aggregates parallel dilated-convolution features across recurrences to obtain broader context while preserving local details.The paper identifies limited receptive fields as a weakness of many existing algorithms.
- Results: The recurrent rain detection and removal network progressively removes rain streaks and addresses accumulation and varied directions in synthesized and real images.Experiments report significant outperformance of state-of-the-art methods, particularly on some heavy-rain images.
2. Related Work
Prior single-image deraining methods treat restoration as signal or layer separation, smoothing, or related image-processing problems. Their limited modeling capacity can produce over-smoothed backgrounds.
- Single-image deraining: Single-image rain removal is more ill-posed than video-based deraining because it lacks temporal information.Some methods formulate the task as separating rain streaks from background layers.
- Existing approaches: Existing approaches include sparse coding with HOG-learned dictionaries, morphological component analysis, layer separation, learned dictionaries, and nonlocal mean smoothing.These approaches represent several distinct formulations for recovering rain-degraded images.
- Limitations: Limited capacity in morphological component analysis, layer separation, and learned dictionaries usually causes background over-smoothness.The related-work discussion places deraining alongside broader deep-learning image-processing applications and bad-weather restoration.
3. Region-Dependent Rain Image Model
The paper extends rain-image formation models to distinguish individually visible rain regions and to represent rain accumulation and overlapping streaks in heavy rain. These models support synthesis and progressive restoration of more realistic rain conditions.
- Region-dependent rain image formation: Existing two-layer modeling struggles with heterogeneous rain density and can over-smooth rain-free regions when localization and removal are performed jointly.The conventional model represents the observation as background plus a rain-streak layer, whose location and intensity are difficult to estimate together.
- Region-dependent rain image formation: The generalized model adds a binary region variable R to indicate where individually visible rain streaks occur.Separating region location from streak appearance provides extra learning information and enables different processing of rain and non-rain regions.
- Rain accumulation and heavy rain: Rain accumulation is modeled as an atmospheric veiling effect similar to mist or fog, while heavy rain also includes overlapping streaks with varied shapes and directions.Accumulation can obscure individual streaks and reduce visibility, especially in distant scenes.
- Rain accumulation and heavy rain: The proposed accumulation model combines direction- and shape-consistent visible streaks with atmospheric effects to synthesize heavy-rain training images closer to natural rain.Its formulation represents overlapping streaks and applies atmospheric veiling to the integrated rain-contaminated image.
- Rain accumulation and heavy rain: The formulation separates rain accumulation handling from rain-streak removal, facilitating training for images containing both degradation types.The paper then uses these models to develop convolutional and recurrent restoration networks.
4. Convolutional Joint Rain Detection and Removal
The paper formulates rain restoration as a convolutional multi-task problem that predicts rain regions, streak appearance, and background jointly. It enriches feature extraction with recurrent multi-scale dilated context and compares alternative prediction structures empirically.
- Convolutional multi-task network: JORDER detects rain regions before constraining rain removal and uses a contextualized dilated network to extract rain-discriminative features.The architecture solves the region-dependent inverse problem through end-to-end learning.
- Network training: Joint training uses ground truths for rain regions, streak maps, and backgrounds, with a combined loss balancing the estimation tasks.The network parameters are optimized by error back-propagation, with λ1 and λ2 controlling relative task importance.
- Convolutional multi-task network: The network estimates R, S, and B sequentially from shared features, corresponding to rain detection, streak estimation, and background recovery.The outputs are predicted in order from contextualized features rather than independently.
- Architecture comparison: Alternative output orders and parallel prediction are evaluated against the proposed sequential architecture in supplementary experiments.The paper reports empirical comparison rather than deriving the ordering from the model formulation.
- Contextualized dilated network: The contextualized dilated network aggregates multi-scale context through recurrent refinement and convolution paths with different dilation factors.This enlarges receptive fields while retaining resolution for rain-feature extraction.
5. Removing Rains from Real Rain Images
The recurrent network progressively removes rain streaks and accumulation, while a derain-dehaze-derain sequence handles both visible streaks and atmospheric veils.
- 5.1. Recurrent Rain Detection and Removal: The recurrent JORDAR model cascades joint rain detection and removal networks to progressively recover visibility.
- 5.1. Recurrent Rain Detection and Removal: Each iteration accumulates predicted residues and updates intermediate estimates, while rain-mask and streak losses provide side information for learning.
- 5.1. Recurrent Rain Detection and Removal: With τ iterations, the network removes rain progressively, reducing per-iteration complexity and enabling better estimation in heavy rain.
- 5.2. Joint Derain and Dehaze: Rain accumulation forms an atmospheric veil similar to fog, so restoring visibility requires a dehazing component.
- 5.2. Joint Derain and Dehaze: The proposed pipeline derains first, dehazes second, and derains again because preprocessing dehazing boosts all streaks, including already visible ones.
- 5.2. Joint Derain and Dehaze: The dehazing network uses a contextualized dilated structure, and the sequential process generally proves effective.
6. Experimental Results
Experiments use synthesized datasets with varied streak patterns and compare the method against established baselines on synthetic and real images. Results report strong restoration, heavy-rain gains, real-image comparisons, efficiency, and the benefit of the derain-dehaze-derain order.
- Datasets: Rain100H synthesizes five streak directions, while Rain12 and Rain100L contain a single streak type for benchmark evaluation.
- Datasets: The synthesized datasets use BSD200 backgrounds, while training uses BSD300 and two rain-streak synthesis procedures.
- Experimental Setup: The study compares four JORDER variants with image decomposition, CNN, DSC, LP, and SRCNN baselines using PSNR and SSIM on luminance.
- Quantitative Evaluation: The method considerably outperforms other methods on Rain12 in both PSNR and SSIM.
- Quantitative Evaluation: More than 1 dB PSNR gain over JORDER on Rain100H demonstrates stronger performance from recurrent detection and removal on synthesized heavy rain.
- Qualitative Evaluation: On real images, JORDER-R removes most rain streaks and significantly outperforms the compared DSC and LP methods.
- Overall Results: The authors report promising removal of most streaks, improved visibility, detail preservation, and efficient GPU execution, with JORDER shortest among CPU approaches.
- Joint Derain and Dehaze: The derain-dehaze-derain order shows significant superiority over other tested joint deraining and dehazing combinations.
7. Conclusion and Future Works
The paper presents a deep learning method for jointly detecting and removing rain from single images, including rain accumulation and heavy rain. Its recurrent architecture progressively removes streaks and atmospheric veils, and real-image evaluations show significant improvements over state-of-the-art methods.
- The method jointly detects and removes rain from a single image, including rain streak accumulation and heavy rain.
- A region-dependent rain image model supports additional rain detection and simulates rain accumulation and heavy rain.
- A fully convolutional network jointly detects and removes rain, using detected rain regions as additional information for restoration.
- A recurrent rain detection and removal network progressively removes rain streaks and embeds a dehazing network to remove atmospheric veils.
- Evaluations on real images demonstrated that the method significantly outperforms state-of-the-art methods.