Source-linked AI summary
Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning
Ruotent Li, Loong Fah Cheong, Robby T. Tan
TL;DR
Heavy-rain restoration remains difficult because streaks, accumulation, veiling, blur, and model-unrepresented damage occur together. The paper combines a physics-based restoration stage with streak-aware decomposition and a model-free conditional-GAN refinement, and reports superiority over baselines on synthetic and real rain data.
Problem
Heavy rain contains veiling, blur, and entangled streak-and-accumulation effects that existing physics models and methods do not fully represent.
Method
A two-stage network estimates rain physics and reconstructs an intermediate image, then uses a model-free conditional GAN to refine it using atmospheric-light information.
Results
The method outperforms baselines on both synthetic and real rain data, with qualitative and quantitative superiority reported.
Takeaways & Limitations
The approach provides a combined physics-constrained and model-free framework for restoring heavy-rain images with rain streaks and accumulation.
Takeaways & Limitations
The decomposition assumes atmospheric light is spatially constant and low-frequency rain streaks are negligible, treating that component as accumulation.
Abstract
from arXiv · showhide
Most deraining works focus on rain streaks removal but they cannot deal adequately with heavy rain images. In heavy rain, streaks are strongly visible, dense rain accumulation or rain veiling effect significantly washes out the image, further scenes are relatively more blurry, etc. In this paper, we propose a novel method to address these problems. We put forth a 2-stage network: a physics-based backbone followed by a depth-guided GAN refinement. The first stage estimates the rain streaks, the transmission, and the atmospheric light governed by the underlying physics. To tease out these components more reliably, a guided filtering framework is used to decompose the image into its low- and high-frequency components. This filtering is guided by a rain-free residue image --- its content is used to set the passbands for the two channels in a spatially-variant manner so that the background details do not get mixed up with the rain-streaks. For the second stage, the refinement stage, we put forth a depth-guided GAN to recover the background details failed to be retrieved by the first stage, as well as correcting artefacts introduced by that stage. We have evaluated our method against the state of the art methods. Extensive experiments show that our method outperforms them on real rain image data, recovering visually clean images with good details.
1. Introduction
Heavy rain combines visible streaks, veiling, blur, and entangled degradation that common models and sequential pipelines do not adequately represent. The paper addresses these issues with a two-stage network, streak-aware decomposition, depth-consistent synthesis, and refinement, reporting qualitative and quantitative superiority.
- Problem: Existing rain models ignore veiling from rain-streak accumulation, while real heavy rain also includes effects not fully represented by the model.Multiflux scattering produces scene blurriness, especially in farther regions, creating problems for model-dependent methods.
- Problem: Heavy rain is defined by severe visual degradation, including visible streaks, washed-out veiling, distant-scene blur, and entangled streaks and accumulation.The paper notes that rain density is a spectrum rather than a sharply divided light/heavy category.
- Problem: Sequential deraining and dehazing cannot properly solve heavy rain because streaks and accumulation become entangled and differ from existing training data.The paper specifically identifies dense veiling and intertwined rain effects as obstacles to simple staged processing.
- Method: The proposed integrated network combines a physics-based subnetwork with a model-free conditional GAN refinement subnetwork.Stage one estimates S, A, and T and reconstructs J under the rain model; stage two is strongly influenced by those outputs.
- Method: Streak-aware decomposition adaptively separates high-frequency streaks from low-frequency accumulation and uses the low-frequency component to improve atmospheric-light estimation.This decomposition targets the entangled appearance of streaks and accumulation.
- Method: The synthetic-data pipeline renders veiling effects using scene depth and adds Gaussian blur to transmission and background to simulate heavy-rain scattering.The paper introduces this pipeline to create more realistic heavy-rain training data.
- Results: The method outperforms state-of-the-art methods qualitatively and quantitatively.The reported superiority is presented as the outcome of the paper’s experiments.
2. Related Works
Existing image- and video-based deraining methods generally target rain streaks rather than heavy-rain scenes. Prior decomposition and dehazing approaches leave background details mixed with streaks or can amplify streak contrast.
- Scope: Most existing image- and video-based deraining methods are not designed for heavy-rain scenes.The paper distinguishes its scope from prior work on this basis.
- Prior decomposition: Bilateral-filter decomposition can place both rain streaks and high-frequency background details in the same layer.The cited sparse-coding approach cannot differentiate genuine object details from rain streaks.
- Prior methods: A prior method combines GMM-based layer decomposition with a preprocessing dehazing step to address rain accumulation.The dehazing step is intended to handle accumulation before subsequent deraining.
3. Network Design
The network combines a physics-based restoration stage with a model-free cGAN refinement stage. Guided decomposition separates rain streaks from accumulation-related components, while depth guidance focuses refinement on distant-scene details.
- 3. Network Design: The first stage estimates rain streaks S, atmospheric light A, and transmission T, then reconstructs background image J under the rain model.The second stage refines J into final clean image C.
- 3. Network Design: The two-stage design combines physics-derived constraints with a less constrained model-free network because the rain model incompletely represents heavy-rain degradation.The physics stage supplies guidance, while the refinement stage adapts more freely to image data.
- 3.1. Stage 1: Physics-based Restoration: The physics-based subnetworks learn streaks from the high-frequency component, atmospheric light from low-frequency inputs, and transmission with an autoencoder using skip connections.Instance normalization is used in the first two transmission-estimation convolutional layers.
- 3.1. Stage 1: Physics-based Restoration: The decomposition assumes atmospheric light is spatially constant and low-frequency rain streaks are negligible, modeling their low-frequency manifestation as rain accumulation.Under these assumptions, the frequency components support estimation of the physical rain parameters.
- 3.1. Stage 1: Physics-based Restoration: Residue-guided decomposition uses spatially variant filtering so the high-frequency channel emphasizes rain streaks while the low-frequency channel retains background details.The residue image is treated as rain-streak invariant and guides the smoothing passband.
- 3.2. Stage 2: Model-Free Refinement: The conditional GAN takes the rain image and estimated J as input, injects atmospheric-light features, and produces refined image C using adversarial, MSE, and perceptual losses.The generator is designed to remove rain effects and artefacts from the previous stage.
4. Implementation
Training uses a new depth-based synthetic dataset for heavy-rain effects, followed by staged training and end-to-end fine-tuning. Test 1 evaluates alternative architectures using estimated physical parameters and reconstructed images.
- 4. Implementation: NYU-Rain uses NYU-Depthv2 backgrounds to synthesize rain streaks and depth-consistent rain accumulation, including veiling effects.The dataset addresses the absence of rain accumulation in existing large-scale synthetic deraining datasets.
- 4. Implementation: The network is trained stage-wise before end-to-end fine-tuning, with the physics stage trained first, the refinement stage trained with it fixed, and both later unfrozen.The physics stage is trained on NYU-Rain under physical-parameter supervision.
5. Experimental Results
Experiments on synthetic and real rain data compare the proposed method with baseline pipelines and analyze its decomposition and refinement stages. The results show improved component estimation, restoration quality, visual clarity, and object recognition.
- Evaluation setup: Test 1 evaluates the method against deraining, dehazing, and GAN-based baselines using PSNR and SSIM metrics.The comparison includes ordered derain/dehaze pipelines and several rain-streak removal methods.
- Ablation study: The decomposition module improves transmission estimation and reconstructed image J, while residue guidance further improves transmission and atmospheric-light estimation.Input guidance cannot fully separate rain streaks from low-frequency content, whereas streak-free residue guidance provides a more suitable low-frequency component.
- Refinement stage: The refinement network restores darkened distant-tree regions in J using relative depth derived from the estimated transmission map.The final output recovers contextual details with visually fine color.
- Real-world rain analysis: On real rain scenes, baseline methods damage background details, retain streaks, or leave veiling effects, whereas the depth-guided GAN removes appropriate veiling in remote areas.These differences are shown across moderate and heavy rain examples with increasing severity from top to bottom.
6. Conclusion
The paper concludes with a two-stage CNN for simultaneously removing rain streaks and accumulation. Its physics-based decomposition and conditional GAN refinement outperform baselines on synthetic and real rain data.
- Conclusion: The proposed two-stage CNN removes rain streaks and rain accumulation simultaneously.The first stage is physics-based, while the refinement stage uses a conditional GAN.
- Conclusion: Streak-aware decomposition separates entangled rain components for joint feature extraction, while transmission and atmospheric-light estimates provide depth and light information for refinement.The conditional GAN takes the reconstructed image from the first stage and produces the final clean image.
- Conclusion: Comprehensive experiments show that the method outperforms baselines on both synthetic and real rain data.