PaperlayerView exact revision on arXiv

Source-linked AI summary

Clearing the Skies: A deep network architecture for single-image rain removal

Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, John Paisley

arXiv:1609.02087v2cs.CV

TL;DR

Single-image rain removal is difficult because rain can resemble object structure and real-world rainy images lack corresponding clean ground truth. DerainNet learns clean-to-rainy detail-layer mappings with a CNN trained on synthesized pairs, supplements the result with image enhancement, and transfers the learned network to real-world images. The authors report improved removal and detail preservation versus competing methods, along with faster testing after training.

  • Problem

    Single-image rain removal is difficult because rain streaks can resemble object structure, while corresponding clean ground truth is unavailable for real-world rainy images.

  • Method

    DerainNet decomposes images into base and high-frequency detail layers, trains a CNN on synthesized clean/rainy detail-layer pairs, and applies image enhancement.

  • Results

    DerainNet transfers well from synthesized training data to real-world images and outperforms competing methods in rain removal, detail preservation, and testing speed.

  • Takeaways & Limitations

    Training on sparse detail layers improves deraining without a large increase in training data or computational resources, while enhancement improves visual results.

Abstract

from arXiv · show

We introduce a deep network architecture called DerainNet for removing rain streaks from an image. Based on the deep convolutional neural network (CNN), we directly learn the mapping relationship between rainy and clean image detail layers from data. Because we do not possess the ground truth corresponding to real-world rainy images, we synthesize images with rain for training. In contrast to other common strategies that increase depth or breadth of the network, we use image processing domain knowledge to modify the objective function and improve deraining with a modestly-sized CNN. Specifically, we train our DerainNet on the detail (high-pass) layer rather than in the image domain. Though DerainNet is trained on synthetic data, we find that the learned network translates very effectively to real-world images for testing. Moreover, we augment the CNN framework with image enhancement to improve the visual results. Compared with state-of-the-art single image de-raining methods, our method has improved rain removal and much faster computation time after network training.

I. INTRODUCTION

Rain degrades images through blurring and haze, motivating effective single-image rain removal for applications such as enhancement and object tracking. Existing approaches span video-based and single-image methods, with the latter facing limited information and substantial room for improvement.

  • Motivation: Rain streaks reduce visual quality through blurring and light-scattering haze, affecting outdoor vision systems.The paper identifies image enhancement and object tracking as practical applications requiring rain removal.
  • Prior work: Rain-removal methods are categorized into video-based and single-image approaches.The introduction reviews both groups before presenting DerainNet.
  • Prior work: Video methods benefit from redundant temporal information, making rain streaks easier to identify and remove.The cited video approaches detect streaks using temporal correlations or neighboring-frame information.
  • Prior work: Single-image rain removal is more challenging because each image provides less information for detecting and removing streaks.The paper describes prior single-image methods based on kernel regression, non-local means, and related techniques.
  • Prior work: Dictionary-learning methods decompose rainy images into base and detail layers, then remove rain from the detail layer before recombination.The detail layer contains rain streaks and object details, while the base layer retains structure.

B. Contributions of our DerainNet approach

DerainNet addresses the difficulty of single-image rain removal by learning from high-frequency detail layers rather than raw images, using image-processing knowledge to improve a modest CNN. The approach combines synthetic training, detail-layer learning, and image enhancement, producing better deraining and faster convergence or testing without simply increasing network complexity.

  • Single-image deraining is difficult because low-level methods can confuse rain streaks with similarly oriented object structures, risking simultaneous removal of rain and structural detail.
  • DerainNet learns the nonlinear mapping between rainy and clean high-frequency detail layers, while image enhancement sharpens the reconstructed base and detail layers.
  • A. Training on high-pass detail layers: Training directly in the image domain can under-fit, and increasing depth can worsen the result, motivating objective-function modification instead of a more complex architecture.
  • A. Training on high-pass detail layers: Low-pass filtering makes rainy and clean base layers approximately equal, allowing the CNN to learn only the mapping needed in the detail layer.
  • A. Training on high-pass detail layers: Detail-layer training exploits sparsity, reducing the mapping range and making the regression problem easier without a large increase in network complexity.
  • A. Training on high-pass detail layers: The detail-layer approach produces clearly better rain-streak removal than the same CNN trained on the image domain and converges much faster in experiments.

B. Our convolutional neural network

DerainNet uses a modest three-operation CNN to map rainy detail layers to cleaned detail layers. Its hidden layers extract rain- and edge-related features, remove streaks, and reconstruct the output detail layer.

  • DerainNet’s network structure consists of three operations: two nonlinear hidden layers followed by an output layer.The hidden layers use tanh nonlinearities, while the output layer produces the cleaned detail layer.
  • Figure 6 visualizes 512 first-layer kernels, three output-layer kernels, and corresponding hidden activations across the detail-layer processing pipeline.The displayed activations show different views of rain being combined into a three-channel RGB output.
  • The first hidden layer extracts features from the input detail layer using learned filters that resemble edge detectors aligned with rain streaks and object edges.This feature extraction resembles representing image patches with dictionary elements.
  • The second hidden layer performs rain-streak removal, producing a smoother representation than the first hidden layer.

C. Training

The network is trained with stochastic gradient descent on synthetic rainy-clean image pairs because large real-world paired datasets are difficult to obtain.

  • Stochastic gradient descent minimizes the network’s objective function during training.
  • 350 clean outdoor images were used to synthesize 14 rainy variants each, creating 4,900 rainy images with corresponding clean ground truth.The clean images came from UCID, BSD, and Google image search.
  • Rain was synthesized with different streak orientations and intensities to form the training dataset.

D. Combining CNN with image enhancement

The framework combines CNN-based detail-layer deraining with post-processing enhancement. Enhancement improves heavy-rain results by modifying the base layer and amplifying recovered detail.

  • The de-rained image is obtained by adding the CNN’s output detail layer to the base layer.
  • Figure 8 visualizes enhanced intermediate results with amplified detail-layer intensities.
  • For heavy rain, enhancement applies a nonlinear function to the base layer and multiplies the CNN detail output by two.
  • Image enhancement improves global contrast and produces a better visual result than deraining without enhancement.The CNN performs virtually all rain removal on the detail layer, while base-layer enhancement improves the global appearance.

III. EXPERIMENTS

DerainNet is evaluated on synthetic and real-world rainy images using a network trained only on synthesized data and compared with three recent deraining methods. It preserves structures while removing rain and generalizes to new synthesized scenes.

  • III. EXPERIMENTS: The evaluation includes synthetic and real-world rainy images and compares DerainNet with three recent high-quality deraining methods.
  • A. Synthesized data: DerainNet preserves the rope while removing rain, whereas competing methods either oversmooth the rope or leave significant rain streaks.The comparison attributes DerainNet’s advantage to its multiple convolutional layers identifying rain without eliminating similar structures.
  • A. Synthesized data: SSIM is used for quantitative evaluation on synthesized images because the ground truth is known, with higher values indicating closer structural agreement.The ground truth has SSIM equal to 1, and enhancement is excluded from these synthetic experiments.
  • A. Synthesized data: DerainNet achieves the highest SSIM values among the compared methods on synthesized testing images.The result agrees with the visual comparisons in which competing methods oversmooth or retain rain streaks.
  • A. Synthesized data: On Rain12, a set of 12 photorealistically rendered rainy images, DerainNet still improves performance while other methods decrease or leave performance unchanged.The experiment highlights generalization to new scenes using the same trained algorithms.

B. Real-world data

DerainNet transfers from synthetic training data to real-world rainy images, where it is evaluated visually, with BIQI, and through a user study using enhancement.

  • B. Real-world data: 4900 synthesized training images were used to test DerainNet on real-world rainy data without corresponding ground truth.Enhancement was applied to all methods in the quantitative comparison, while competing methods used post-processing and DerainNet used its proposed enhancement pipeline.
  • B. Real-world data: Synthetic-data training was sufficient for DerainNet to remain effective when applied to real-world images.
  • B. Real-world data: The method arguably best removed rain while preserving details in real-world examples, although unavailable ground truth prevented definitive quantitative ranking by visual inspection.
  • B. Real-world data: BIQI was used as a reference-free measure, and DerainNet achieved the lowest BIQI on 100 newly obtained real-world testing images.Lower BIQI indicates higher image quality, though the authors caution that reference-free metrics may not always match subjective judgments.
  • B. Real-world data: Twenty participants ranked enhanced outputs from four algorithms and the original rainy image in an independent user study.The study used the same 100 real-world images evaluated with BIQI.

C. Parameter settings

Parameter experiments examine the testing setup and kernel-size choices, selecting 16-1-8 as the default configuration because larger kernels improved results while enlarging the nonlinear middle kernel helped little.

  • C. Parameter settings: Testing used 100 newly synthesized images and the Rain12 images, with the same training data as previously.
  • 1) Kernel size:: Kernel configurations 16-1-8, 4-1-2, 8-1-4, 16-3-8, and 16-5-8 were compared using average SSIM.
  • 1) Kernel size:: Larger first and third kernel sizes improved results by modeling more structure and texture, whereas enlarging the second kernel produced only limited improvement.
  • 1) Kernel size:: The 16-1-8 configuration was selected as the default kernel-size setting.

2) Network width:

The parameter study contrasts network width and depth: wider networks improve performance but cost runtime, while greater depth does not improve feed-forward deraining and three layers are retained.

  • 2) Network width:: Increasing kernel counts n1 and n2 from values in {64, 128, 256} toward the default 512 improved average SSIM but increased running time.
  • 3) Network depth:: Networks with depths 3, 5, and 10 showed no benefit from added depth for feed-forward deraining.
  • 3) Network depth:: DerainNet retained three layers because its detail-training strategy produced high-quality results with lower model complexity and computation time.
  • 2) Network width:: Direct image-domain training left visible rain streaks even when the training image itself was used as a sample.
  • 2) Network width:: The proposed and directly image-domain-trained approaches had virtually identical computational complexity, while the proposed approach better addressed image-domain drawbacks.

E. Impact of image enhancement step

Enhancement is evaluated as absent, post-reconstruction, or integrated with deraining; the experiments also test alternative low-pass filters and a bilateral-filter comparison with another method.

  • E. Impact of image enhancement step: Three strategies were compared on real-world data: no enhancement, post-processing enhancement, and simultaneous deraining and enhancement.
  • E. Impact of image enhancement step: The CNN alone removes rain streaks, while the enhancement step contributes to the subsequent visual processing.
  • E. Impact of image enhancement step: Guided, bilateral, and rolling-guidance low-pass filters produced qualitatively similar deraining results with nearly identical SSIM values.DerainNet can recognize and remove rain when rain is isolated to the detail layer.
  • E. Impact of image enhancement step: With bilateral filtering, method did not completely remove rain streaks, whereas DerainNet achieved considerably better removal in the shown comparison.

G. Training convergence and testing runtime

DerainNet requires substantial upfront training but enables efficient feed-forward testing, outperforming other methods in computational efficiency. The paper reports roughly half a minute for large images using a parallel GPU implementation.

  • Training convergence and testing runtime: Approximately two days of training are required before DerainNet can process new images.The reported training duration reflects a one-time computational cost rather than per-image processing.
  • Training convergence and testing runtime: DerainNet processes new images significantly faster than current state-of-the-art de-raining methods after training.Testing excludes training time and measures application to new data.
  • Training convergence and testing runtime: Roughly half a minute is required for typical camera-sized images with a parallel GPU implementation.Other cited methods require several minutes to over an hour for similarly large images.
  • Training convergence and testing runtime: The paper concludes that DerainNet improves both image quality and computational efficiency without requiring a very deep or wide network.The architecture uses image-processing domain knowledge to support this modest network size.
Loading 1609.02087v2…