PaperlayerView exact revision on arXiv

Source-linked AI summary

Image De-raining Using a Conditional Generative Adversarial Network

He Zhang, Vishwanath Sindagi, Vishal M. Patel

arXiv:1701.05957v4cs.CV

TL;DR

Weather degradation harms image quality and can impair computer-vision algorithms, while existing de-raining methods leave additional information out of the optimization. ID-CGAN uses a conditional GAN to produce de-rained images, and its use boosted Faster-RCNN mean average precision by 78% on artificially degraded VOC images.

  • Problem

    De-raining methods do not incorporate additional information into optimization, despite weather degradation impairing computer-vision algorithms.

    Source: §I · pp. 1–2

  • Method

    ID-CGAN learns a rainy-to-de-rained mapping with a conditional GAN, combining generator, discriminator, and perceptual loss to target visual, quantitative, and discriminative performance.

    Source: §I · p. 2, §III · p. 3

  • Results

    On artificially rain-degraded VOC images, ID-CGAN boosted Faster-RCNN mean average precision by 78%, while removing most rain drops and maintaining background detail in real images.

    Source: 2) Evaluation on Real Rainy Images · p. 8, 3) Evaluation on Object Detection… · p. 9

  • Takeaways & Limitations

    The findings show that ID-CGAN outputs can improve downstream object detection on rain-degraded images, including real-world examples where Faster-RCNN initially failed.

    Source: 3) Evaluation on Object Detection… · p. 9

Abstract

from arXiv · show

Severe weather conditions such as rain and snow adversely affect the visual quality of images captured under such conditions thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect performance of vision systems. Hence, it is important to solve the problem of single image de-raining/de-snowing. However, this is a difficult problem to solve due to its inherent ill-posed nature. Existing approaches attempt to introduce prior information to convert it into a well-posed problem. In this paper, we investigate a new point of view in addressing the single image de-raining problem. Instead of focusing only on deciding what is a good prior or a good framework to achieve good quantitative and qualitative performance, we also ensure that the de-rained image itself does not degrade the performance of a given computer vision algorithm such as detection and classification. In other words, the de-rained result should be indistinguishable from its corresponding clear image to a given discriminator. This criterion can be directly incorporated into the optimization framework by using the recently introduced conditional generative adversarial networks (GANs). To minimize artifacts introduced by GANs and ensure better visual quality, a new refined loss function is introduced. Based on this, we propose a novel single image de-raining method called Image De-raining Conditional General Adversarial Network (ID-CGAN), which considers quantitative, visual and also discriminative performance into the objective function. Experiments evaluated on synthetic images and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performance.

I. INTRODUCTION

The paper proposes ID-CGAN for single-image de-raining, using adversarial guidance to target visual quality and preserve downstream vision performance. Its framework combines a densely connected generator, a multi-scale discriminator, and a refined perceptual loss without additional post-processing.

I. INTRODUCTION

  • Severe weather degrades image quality and can impair tracking, detection, and segmentation; collecting large training datasets from unconstrained conditions may be impractical.The paper therefore motivates automatically removing weather artifacts from single images to improve both their visual quality and usefulness to vision algorithms.Source: §I · p. 1
  • Rainy images are modeled as the superposition of a clean background and rain streaks, making de-raining a component-separation problem.For a single image, temporal information used by video-based methods – is unavailable.Source: §I · pp. 1–2
  • Prior single-image methods use sparsity –, GMM, patch-rank, or CNN mappings –, but CNNs typically minimize L2 error and need regularization for the ill-posed task.The paper motivates adversarial loss as an additional regularizer that can provide feedback toward clean-looking outputs.Source: §I · p. 2
  • The optimization should preserve detection and classification performance, improve visual appeal without losing detail, and avoid separate image-processing steps,.The authors frame these as additional information that prior de-raining optimization methods had not incorporated.Source: §I · p. 2
  • ID-CGAN uses adversarial loss as regularization, a densely connected generator, a multi-scale discriminator, and refined perceptual loss to produce visually pleasing de-rained images without post-processing.The discriminator captures hierarchical context at multiple scales; the paper also reports evaluating the method on public and synthesized datasets and testing object detection on VOC with Faster-RCNN,.Source: §I · p. 2

II. BACKGROUND · A. Single Image De-raining

Single-image de-raining is ill-posed and lacks temporal constraints, motivating prior-based methods spanning sparsity, low-rank, GMM, and deep learning; the background also reviews conditional GANs.

II. BACKGROUND

  • The background section briefly reviews existing single-image de-raining methods and conditional GANs.Source: §II · p. 2

A. Single Image De-raining

  • Ill-posedness and the absence of temporal constraints make single-image de-raining challenging, motivating methods that impose prior knowledge.The reviewed approaches include sparsity-based, low-rank, Gaussian mixture model, and deep learning methods.Source: A. Single Image De-raining · p. 3
  • Dictionary-based methods separate clean background and rain using sparse representations, with one approach assuming similarly oriented streaks and another producing artifacts,.The latter uses discriminative codes and mutually exclusive dictionary atoms for background and rain-streak patches.Source: A. Single Image De-raining · p. 3
  • Low patch-rank priors capture recurring rain patterns but can remove texture details and blur results; convolutional low-rank filters were proposed to capture rain pixels.The prior draws on patch-rank methods for characterizing texture in cartoon-texture decomposition,.Source: A. Single Image De-raining · p. 3
  • GMM-based methods use patch priors to model background and rain streaks separately, based on GMMs accommodating multiple streak orientations and scales.Source: A. Single Image De-raining · p. 3
  • CNN-based methods learn mappings from rainy inputs to clean targets; Fu et al. decompose images into background and detail layers, then remove rain from the detail layer,.Fu et al. later extended the network with Res-blocks.Source: A. Single Image De-raining · p. 3
  • Yang et al. proposed a CNN that jointly detects and removes rain streaks, and later work explored additional deep-learning approaches –[35].Source: A. Single Image De-raining · p. 3

B. Generative Adversarial Networks · III. PROPOSED METHOD

ID-CGAN directly maps rainy images to de-rained images with a conditional GAN, combining a generator, discriminator, and refined perceptual loss to guide outputs toward visual quality despite GAN training instability.

B. Generative Adversarial Networks

  • GANs model data distributions with generator and discriminator functions trained jointly through an alternating min-max game.The generator maps samples from a random uniform distribution to the data distribution, while the discriminator estimates whether samples belong to that distribution.Source: B. Generative Adversarial Networks · p. 3
  • Although GANs can preserve high-frequency detail in visually appealing images, they are difficult to train and prone to mode collapse.Related work explores conditioning [37], training improvements, task-specific costs [39], and an alternative discriminator interpretation [40].Source: B. Generative Adversarial Networks · p. 3

III. PROPOSED METHOD

  • ID-CGAN learns a direct mapping from a rainy input image to its de-rained background image using a conditional GAN.This replaces a decomposition-based approach and comprises a generator, discriminator, and perceptual loss function.Source: §III · p. 3
  • The generator is a densely connected, symmetric deep CNN with skip connections that synthesizes a de-rained image from a rainy input.The generator architecture is shown in Fig. 3.Source: §III · p. 3, Figure 3 · p. 4
  • A multi-scale discriminator distinguishes generated de-rained images from corresponding real ground-truth images and guides the generator.The proposed network has generator G and discriminator D sub-networks.Source: §III · p. 3, Figure 3 · p. 4
  • The refined perceptual loss is designed to address GAN training instability and artifacts while ensuring visually appealing de-rained images.The method considers discriminative and visual performance in its optimization, without additional image processing.Source: §III · p. 3, Table I · p. 4

A. GAN Objective Function · B. Generator with Symmetric Structure

The method uses a conditional adversarial objective to map rainy images to clear outputs, with a symmetric generator designed to remove rain while preserving background detail. Its generator combines densely connected blocks, multilevel skip connections, and sampling transitions.

A. GAN Objective Function

  • The conditional GAN alternately updates generator G and discriminator D so G maps rainy input x to output y while D distinguishes synthesized from real images.The objective follows a mini-max game, conditioning generation on the rainy image.Source: A. GAN Objective Function · p. 4

B. Generator with Symmetric Structure

  • The generator must remove rain streaks as much as possible without losing background detail, motivating a structure suited to pixel-level de-raining.Source: B. Generator with Symmetric Structure · p. 4
  • Prior sparse-coding, neural-network, and CNN approaches use symmetric encoding-decoding because separated background content must be transferred back from its processing domain.The passage describes sparse coding as decoding to coefficient maps and then using analysis dictionaries to obtain a clear output.Source: B. Generator with Symmetric Structure · pp. 4–5
  • The proposed symmetric generator learns an end-to-end rainy-to-ground-truth mapping using dense blocks rather than U-Net or ResNet blocks, with cross-block skip connections for feature reuse and convergence.The paper attributes strong gradient flow and parameter efficiency to dense blocks.Source: B. Generator with Symmetric Structure · p. 5
  • Each dense-block layer applies batch normalization, leaky ReLU, and a 3×3 convolution in sequence.Source: B. Generator with Symmetric Structure · p. 5
  • The generator follows dense blocks with up-sampling (Tu), down-sampling (Td), or no-sampling (Tn) transitions, and includes symmetric skip connections to improve training efficiency and convergence performance.Its listed sequence is CBLP(64)-D(256)-Td(128)-D(512)-Td(256)-D(1024)-Tn(512)-D(768)-Tn(128)-D(640)-Tu(120)-D(384)-Tu(64)-D(192)-Tu(64)-D(32)-Tn(16)-C(3)-Tanh.Source: B. Generator with Symmetric Structure · p. 5

C. Multi-scale Discriminator · D. Refined Perceptual Loss

ID-CGAN uses a multi-scale discriminator to incorporate local and global context, and combines pixel, perceptual, and adversarial losses to support visual, quantitative, and discriminatory performance.

C. Multi-scale Discriminator

  • A single-scale 70×70 patch discriminator can miss global context and tiny details, including a truck’s front mirror, motivating a discriminator that evaluates both local and global information.Source: C. Multi-scale Discriminator · p. 5
  • The multi-scale discriminator pools features at different scales, then upsamples and concatenates them before a 1×1 convolution and sigmoid produce a probability score.The pooled features explicitly incorporate global hierarchical context.Source: C. Multi-scale Discriminator · p. 5

D. Refined Perceptual Loss

  • GAN instability can introduce visual artifacts, while high-level perceptual losses can improve visual performance but may fail to preserve color and texture or achieve strong quantitative performance.Source: D. Refined Perceptual Loss · pp. 5–6
  • The refined loss combines pixel-to-pixel Euclidean, perceptual, and adversarial losses with weights to support visual quality, quantitative scores, and discriminatory performance.The adversarial component is the discriminator loss that guides the generator.Source: D. Refined Perceptual Loss · p. 6
  • Setting both loss weights to zero yields a Euclidean-only CNN; setting the perceptual weight to zero yields a GAN, while setting the adversarial weight to zero gives the structure proposed in.Source: D. Refined Perceptual Loss · p. 6
  • The per-pixel Euclidean loss compares the generated de-rained output with its corresponding ground truth image.The passage describes an input and ground-truth image pair with C channels, width W, and height H.Source: D. Refined Perceptual Loss · p. 6
  • The perceptual loss minimizes distance between high-level features transformed by a nonlinear CNN, with features computed at relu2_2 in VGG-16 [31].Source: D. Refined Perceptual Loss · p. 6

IV. EXPERIMENTS AND RESULTS

The section describes how ID-CGAN is evaluated, covering quality measures, datasets, training details, and comparisons with baseline and state-of-the-art methods.

  • The experiments evaluate ID-CGAN using quality measures and report dataset and training details alongside comparisons with baseline and recent state-of-the-art methods.Source: §IV · p. 6

A. Experimental Details … B. Model Details and Parameters

The experiments use synthetic rainy-image data for quantitative testing and a separate real-world set for visual evaluation, with specified image-quality metrics and network-training settings. The supplied passages describe datasets, evaluation, and model configuration but report no final performance comparison.

1) Synthetic dataset:

  • The synthetic dataset contains 700 training images and 100 test images sampled from UCID and BSD-500, with varied rain streaks added following.Training uses 500 UCID and 200 BSD-500 training images; testing uses 50 UCID and 50 BSD-500 test images. Samples are resized to 256×256.Source: 1) Synthetic dataset · pp. 6–7
  • Figures 5 and 6 show qualitative comparisons of baseline configurations, including GEN, GEN-CGAN-S, GEN-P, GEN-PS, ID-CGAN, and target images.Figure 6 labels one configuration GEN-CGAN-PS, whereas Figure 5 labels it GEN-PS.Source: Figure 5 · p. 7, Figure 6 · p. 7

2) Real-world rainy images dataset:

  • The real-world evaluation set contains 50 Internet-downloaded rainy images selected for diverse content and rain intensity and orientation, and is used only for testing.Source: 2) Real-world rainy images dataset · p. 7

3) Quality measures:

  • Synthetic-image performance is assessed using luminance-channel PSNR, SSIM, UQI, and VIF; real-set performance is evaluated visually because ground truth is unavailable.Source: 3) Quality measures · p. 7

B. Model Details and Parameters

C. Comparison with Baseline Configurations

The comparison tests how Euclidean, perceptual, and adversarial losses—and single- versus multi-scale discrimination—affect de-raining. ID-CGAN’s multi-scale discriminator recovers texture details missing from the single-scale variant and achieves the best results.

C. Comparison with Baseline Configurations

  • The five configurations, trained on synthetic training images, progress from Euclidean-only GEN to adding single-scale adversarial loss, perceptual loss, and finally multi-scale adversarial loss in ID-CGAN.GEN uses Euclidean loss; GEN-CGAN-S adds single-scale adversarial loss; GEN-P adds perceptual loss; GEN-CGAN-PS combines perceptual and single-scale adversarial losses; ID-CGAN uses a multi-scale discriminator.Source: C. Comparison with Baseline… · pp. 7–8
  • ID-CGAN’s multi-scale adversarial loss, combined with Euclidean and perceptual losses, recovers missing texture details and achieves the best results; the paper says quantitative results also demonstrate each module’s effectiveness.The reported recovered details include a car edge and a truck’s front-mirror structure.Source: C. Comparison with Baseline… · p. 8
  • On real test images, adversarial loss improves visual quality over the traditional CNN but introduces artifacts; adding perceptual loss reduces artifacts and sharpens results, though some texture remains missing.Results for the baseline configurations are shown in Fig. 5 and Fig. 6.Source: C. Comparison with Baseline… · p. 8

D. Comparison with State-of-the-art Methods

This section identifies recent state-of-the-art single-image de-raining methods used for comparison with ID-CGAN.

D. Comparison with State-of-the-art Methods

1) Evaluation on synthetic dataset: · 2) Evaluation on Real Rainy Images: · 3) Evaluation on Object Detection Results:

ID-CGAN outperforms recent methods on synthetic-image metrics and rainy-image comparisons, while de-raining also improves Faster-RCNN detection on artificially degraded VOC 2007 images. Visual results show rain removal alongside detail preservation, though some streaks remain in difficult real-image cases.

1) Evaluation on synthetic dataset:

  • ID-CGAN achieves superior PSNR, SSIM, UQI, and VIF results against recent methods on synthetic test images with available ground truth.The evaluation compares quantitative and qualitative performance on synthetic-dataset test images.Source: 1) Evaluation on synthetic dataset · p. 8
  • On difficult synthetic examples, ID-CGAN removes most rain streaks while preserving image details, unlike comparison methods that leave streaks or blur results.PRM removes streaks but produces blurred results; the other compared methods remove streaks only partially or reduce rain intensity. Fig. 7 shows two sample images.Source: 1) Evaluation on synthetic dataset · p. 8, Figure 7 · p. 9

2) Evaluation on Real Rainy Images:

  • On real rainy images, ID-CGAN removes most drops and performs better than the compared methods, while retaining background detail.DSC tends to add artifacts, while GMM, CNN, CCR, DNN, and JORDER leave visible drops in zoomed regions; ID-CGAN leaves a few streaks in these difficult cases. Figs. 9 and 11 provide additional comparisons and examples.Source: 2) Evaluation on Real Rainy Images · p. 8

3) Evaluation on Object Detection Results:

  • To demonstrate performance improvement after deraining with IDCGAN, the authors evaluated Faster-RCNN on the VOC 2007 dataset; single-image deraining can preprocess face recognition and object detection.Source: 3) Evaluation on Object Detection… · p. 8
  • 78%: Faster-RCNN performance on artificially rain-degraded VOC 2007 images improves after ID-CGAN de-raining.The study reports mean average precision for the entire VOC dataset; degraded images alone yield low average precision.Source: 3) Evaluation on Object Detection… · p. 9
  • Faster-RCNN fails on sample real-world rainy images but successfully detects scene objects after those images are processed by ID-CGAN.The paper presents sample detection results on rainy and de-rained images in Fig. 10.Source: 3) Evaluation on Object Detection… · p. 9

4) Computation times: · V. CONCLUSION

The paper proposes ID-CGAN, a conditional GAN for single-image rain-streak removal, with a refined loss and multi-scale discriminator. It reports fast GPU processing and improvements in image metrics and object detection.

4) Computation times:

  • About 0.3s: GPU ID-CGAN processes a 500 × 500 image and is reported fastest among GPU-based CNN methods, running as fast as Fu et al..The paper notes that baseline methods were implemented in MATLAB or MATLAB wrappers, while ID-CGAN was implemented in Torch.Source: 4) Computation times · p. 9, §V · p. 9

V. CONCLUSION

  • ID-CGAN uses generative modeling to synthesize a de-rained image from a rainy input, rather than relying on image-decomposition methods with prior information.The paper frames this as its approach to single-image rain-streak removal.Source: §V · p. 9, GMM [14] · p. 12

GMM [14]

  • Faster-RCNN detection performance improves when ID-CGAN is used as a pre-processing step on rainy images.The figure caption identifies rainy-image detections on the left and de-rained-image detections on the right.Source: Figure 10 · p. 11, GMM [14] · p. 12

V. CONCLUSION

  • A refined GAN loss is proposed to improve training stability and reduce output artifacts.The method also introduces a multi-scale discriminator to use features from different scales when judging whether a de-rained image is real or fake.Source: GMM [14] · p. 12
  • Experiments on synthetic and real-world datasets compare ID-CGAN with recent methods and report significant improvements across different metrics.The paper also reports conducting a detailed ablation study of improvements from different modules.Source: GMM [14] · p. 12
  • A detailed ablation study examines improvements due to different modules in ID-CGAN.The conclusion passage describes the study as illustrating the contributions of the method’s modules.Source: GMM [14] · p. 12

Related papers

Loading 1701.05957v4…