PaperlayerView exact revision on arXiv

Source-linked AI summary

Image De-raining Using a Conditional Generative Adversarial Network

He Zhang, Vishwanath Sindagi, Vishal M. Patel

arXiv:1701.05957v4cs.CV

TL;DR

Single-image de-raining is difficult because severe weather degrades images and vision performance, while the task is inherently ill-posed. The paper proposes ID-CGAN, a conditional GAN with refined perceptual loss, and reports improved visual and quantitative results, including a 78% boost in Faster-RCNN performance after de-raining.

  • Problem

    Severe weather degrades image quality and vision-system performance, while single-image de-raining remains ill-posed and lacks temporal information as an additional constraint.

  • Method

    ID-CGAN directly maps rainy images to de-rained images using a conditional GAN with densely connected generation, multi-scale discrimination, and refined perceptual loss.

  • Results

    78% boost in mean average precision is reported for Faster-RCNN when VOC images are de-rained with ID-CGAN before detection.

  • Takeaways & Limitations

    The method targets de-raining that preserves visual quality while also supporting downstream object detection performance.

Abstract

from arXiv · show

Severe weather conditions such as rain and snow adversely affect the visual quality of images captured under such conditions thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect performance of vision systems. Hence, it is important to solve the problem of single image de-raining/de-snowing. However, this is a difficult problem to solve due to its inherent ill-posed nature. Existing approaches attempt to introduce prior information to convert it into a well-posed problem. In this paper, we investigate a new point of view in addressing the single image de-raining problem. Instead of focusing only on deciding what is a good prior or a good framework to achieve good quantitative and qualitative performance, we also ensure that the de-rained image itself does not degrade the performance of a given computer vision algorithm such as detection and classification. In other words, the de-rained result should be indistinguishable from its corresponding clear image to a given discriminator. This criterion can be directly incorporated into the optimization framework by using the recently introduced conditional generative adversarial networks (GANs). To minimize artifacts introduced by GANs and ensure better visual quality, a new refined loss function is introduced. Based on this, we propose a novel single image de-raining method called Image De-raining Conditional General Adversarial Network (ID-CGAN), which considers quantitative, visual and also discriminative performance into the objective function. Experiments evaluated on synthetic images and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performance.

I. INTRODUCTION

Single-image de-raining is motivated by the damage severe weather causes to image quality and vision algorithms. ID-CGAN addresses this ill-posed problem by jointly targeting visual quality, quantitative performance, and discriminative fidelity.

  • Motivation: Severe weather degrades image quality and adversely affects tracking, detection, segmentation, face detection, and verification.Heavy rain makes images difficult for vision algorithms trained mainly on controlled-condition data.
  • Problem formulation: Single-image de-raining separates a rainy observation into clean background and rain-streak components without temporal information.Prior approaches use sparsity, Gaussian mixture model, patch-rank, or learned CNN-based approaches.
  • Design criteria: Existing methods motivate adding discriminative information so reconstructed images remain indistinguishable from their clean counterparts to vision algorithms.Visual appeal and preservation of important details are also identified as optimization criteria.
  • Proposed approach: ID-CGAN uses a conditional GAN framework with a densely connected generator and multi-scale discriminator for single-image de-raining.The generator maps rainy images to de-rained images, while the discriminator distinguishes rainy images from images without rain.
  • Evaluation: The method is evaluated on public and synthesized datasets for visual and quantitative performance, including Faster-RCNN object detection on VOC.The contributions explicitly include demonstrating improved high-level object detection performance.

II. BACKGROUND

The background reviews prior single-image de-raining methods and GANs. Existing de-raining work spans prior-based decomposition, CNN mappings, and later deep-learning extensions, while GANs provide adversarial distribution modeling.

  • Single Image De-raining: Single-image de-raining is difficult because it is ill-posed and lacks the temporal information available to video-based methods.Researchers therefore impose prior knowledge such as sparsity, low rank, or Gaussian mixture structure.
  • Single Image De-raining: Sparsity-based methods separate clean backgrounds and rain streaks using learned dictionary atoms, often assuming rain streaks have similar orientations.The reviewed approaches operate on representations such as high-frequency components or patches.
  • Single Image De-raining: CNN-based methods learn nonlinear mappings from rainy inputs to clean targets, with some jointly detecting and removing rain streaks.Several additional deep-learning methods have since been explored for the task.

B. Generative Adversarial Networks

GANs jointly train a generator and discriminator in a min-max framework. ID-CGAN adapts this structure to de-raining with a densely connected generator, multi-scale discriminator, and perceptual-loss component.

  • Generative Adversarial Networks: GANs model data distributions with a generator that synthesizes samples and a discriminator that estimates whether samples belong to the data distribution.The two networks are typically trained jointly by alternating updates under a game-theoretic min-max objective.
  • ID-CGAN: ID-CGAN directly learns a mapping from rainy images to de-rained background images instead of solving rain removal through image decomposition.Its three components are a generator, discriminator, and perceptual loss function.
  • ID-CGAN: The densely connected generator synthesizes de-rained images, while the multi-scale discriminator distinguishes them from corresponding ground-truth real images and guides the generator.The discriminator uses pooled features at different scales to combine local and global information.
  • Design properties: ID-CGAN is designed to avoid additional image processing while incorporating discriminative and visual performance into optimization.These properties are listed as desirable differences from existing methods.

B. Generator with Symmetric Structure

The generator uses a symmetric encoder-decoder structure to map rainy inputs to clean targets while preserving background detail. Densely connected blocks, transition operations, and skip connections define the architecture.

  • B. Generator with Symmetric Structure: Because de-raining requires pixel-level rain removal without losing background details, generator structure is central to producing the desired output.The paper frames generator design as the key architectural issue.
  • B. Generator with Symmetric Structure: Prior sparse-coding, neural-network, and CNN approaches commonly use symmetric encoding-decoding structures for transforming degraded inputs into clear outputs.These structures encode inputs and decode representations into the desired image domain.
  • B. Generator with Symmetric Structure: ID-CGAN adopts a symmetric generator that directly learns an end-to-end mapping from rainy inputs to corresponding ground truth images.Unlike generators using U-Net or ResNet blocks, it uses densely connected blocks and skip connections.
  • B. Generator with Symmetric Structure: Each dense block is followed by a transition block that performs up-sampling, down-sampling, or no sampling.The architecture also includes symmetric skip connections to improve training efficiency and convergence.

C. Multi-scale Discriminator

The proposed discriminator judges de-rained outputs against ground-truth images while incorporating features from multiple scales. The refined objective combines pixel, perceptual, and adversarial losses to improve visual and quantitative quality while reducing artifacts.

  • The discriminator classifies each de-rained image as real or fake relative to the ground-truth image.
  • A multi-scale pooling module aggregates features at different scales to incorporate global hierarchical context into the discriminator.Pooled features are upsampled, concatenated, and mapped to a probability score between 0 and 1.
  • Perceptual loss is introduced because adversarial training can produce artifacts and noisy or incomprehensible outputs.The paper illustrates these artifacts in qualitative comparisons and uses perceptual loss to improve visual performance.
  • The refined loss combines per-pixel Euclidean, perceptual, and adversarial losses with weights λp and λa.Setting either weight to zero reduces the objective to a corresponding simpler CNN, GAN, or perceptual-loss configuration.
  • The loss terms compare generated outputs with corresponding ground-truth images using image-space, high-level-feature, and discriminator-based objectives.The perceptual term uses high-level features, while the adversarial term guides a set of generated de-rained images.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate ID-CGAN using synthesized training and testing samples, qualitative comparisons, and resized images. The supplied passages establish the experimental setup but do not report the resulting performance values.

  • The experiments compare ID-CGAN with baseline and recent state-of-the-art methods using specified datasets and quality measures.
  • The synthesized training set contains 700 images drawn from UCID and BSD-500, while the test set contains 100 images from those datasets.
  • Figures 5 and 6 provide qualitative comparisons across baseline configurations and the target image.
  • All training and test samples are resized to 256×256.

2) Real-world rainy images dataset:

A separate real-world rainy-image dataset is used for evaluation only, with diverse content and rain characteristics. Because it lacks ground-truth references, evaluation uses measures suited to that setting, alongside the reported training configuration.

  • The real-world dataset contains 50 rainy images downloaded from the Internet and is used only for testing.The images were selected for diversity in content, rain intensity, and rain-pixel orientation.
  • Real-dataset evaluation lacks ground-truth reference images, limiting direct reference-based quantitative assessment.
  • The network is trained with batch size 1 for 100k iterations using Adam at a learning rate of 2×10^-3.
  • The adversarial-loss weight is λa = 6.6 × 10^-3 and is kept low so adversarial loss does not dominate the other losses.The parameters are selected by cross-validation.

C. Comparison with Baseline Configurations

The baseline study isolates pixel, perceptual, adversarial, and discriminator-scale components of ID-CGAN. Qualitative comparisons indicate that combining all losses with a multi-scale discriminator best recovers texture details among the configurations described.

  • Baseline configurations: GEN uses only per-pixel Euclidean loss, whereas GEN-CGAN-S adds adversarial loss from a single-scale discriminator.
  • Baseline configurations: GEN-P combines per-pixel Euclidean and perceptual losses, while GEN-CGAN-PS adds adversarial loss from a single-scale discriminator.
  • Baseline configurations: ID-CGAN combines per-pixel Euclidean, perceptual, and adversarial losses from a multi-scale discriminator.
  • Qualitative comparison: Adversarial loss improves visual quality over traditional CNN architectures but introduces artifacts, which perceptual loss reduces while producing sharper results.
  • Qualitative comparison: The full ID-CGAN configuration recovers texture details missing from single-scale alternatives and achieves the best results in the described comparisons.The passage identifies details such as a car edge and a truck mirror structure as recovered examples.

D. Comparison with State-of-the-art Methods

The study compares ID-CGAN with recent state-of-the-art methods for single-image de-raining, spanning GAN-, sparse-representation-, prior-, GMM-, CNN-, convolutional-coding-, and deep-detail-based approaches.

  • The comparison evaluates the proposed ID-CGAN method against recent state-of-the-art single-image de-raining methods.
  • Additional baselines include PRM, a prior-based method, and GMM, a Gaussian mixture model-based method.
  • The comparison also includes CNN-based methods CNN and JORDER, convolutional-coding method CCR, and Deep Detail Network method DDN.
  • Compared baselines include PAN, a GAN-based method, and SPM and DSC, based on sparse dictionaries and discriminative sparse coding.

1) Evaluation on synthetic dataset:

ID-CGAN is assessed on synthetic and real rainy images, with quantitative metrics, visual comparisons, and downstream Faster-RCNN detection evaluation. The reported results show stronger de-raining quality and improved detection after preprocessing.

  • Evaluation on synthetic dataset:: ID-CGAN achieves superior quantitative performance on the synthetic dataset across PSNR, SSIM, UQI, and VIF.The comparison uses test images with available ground truth and evaluates all four metrics in Table V.
  • Evaluation on synthetic dataset:: On difficult synthetic examples, ID-CGAN removes most rain streaks while maintaining image details better than the compared methods.PRM produces blurred results, while other methods either leave streaks or remove them only partially.
  • Evaluation on Real Rainy Images:: On real rainy images, ID-CGAN removes most rain drops while maintaining background details, although a few streaks remain in difficult cases.DSC adds artifacts, and several other methods leave visible rain drops in zoomed regions of interest.
  • Evaluation on Object Detection Results:: 78%: Faster-RCNN detection performance is boosted after artificially rain-degraded VOC 2007 images are de-rained with ID-CGAN.The de-rained images are then fed to Faster-RCNN, with mean average precision reported for the entire VOC dataset.
  • Evaluation on Object Detection Results:: On real-world rainy examples, Faster-RCNN fails under degradation but successfully detects objects after ID-CGAN preprocessing.The comparison uses the same detection method before and after de-raining.

4) Computation times:

The paper reports ID-CGAN’s computational performance alongside qualitative real-image and object-detection examples. On GPU, the method processes a 500 × 500 image in about 0.3 seconds.

  • 4) Computation times:: 0.3s: ID-CGAN processes an image of size 500 × 500 on average when running on GPU.The paper states that ID-CGAN achieves the fastest time compared with the listed methods and is as fast as Fu et al..
  • V. CONCLUSION: The proposed method is presented as a conditional GAN-based single-image rain-streak removal algorithm.The conclusion frames the approach as generative modeling that synthesizes a de-rained image from a rainy input.
  • Evaluation Examples: Real-world examples show detection results for rainy inputs and corresponding ID-CGAN de-rained images.Additional figures provide further real-world de-rained results and qualitative comparisons.
Loading 1701.05957v4…