PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Pyramid Networks for Image Deraining

Xueyang Fu, Borong Liang, Yue Huang, Xinghao Ding, John Paisley

arXiv:1805.06173v1cs.CV

TL;DR

Single-image deraining must remove rain from inputs where rain, edges, and backgrounds are difficult to separate, while existing deep models use many parameters. LPNet applies Gaussian-Laplacian pyramids with recursive and residual subnetworks to simplify the task. It uses fewer than 8K parameters while achieving reported strong deraining performance and showing potential for other vision tasks.

  • Problem

    Rain is difficult to learn directly in the image domain, while very deep deraining networks have huge parameter counts that can limit applications such as mobile devices.

  • Method

    LPNet decomposes a rainy image into a Laplacian pyramid, processes each level with a recursive and residual subnetwork and level-specific loss, and reconstructs a derained Gaussian pyramid.

  • Results

    LPNet contains fewer than 8K parameters, achieves state-of-the-art performance on single-image deraining, generalizes well from synthetic training data to real-world images, and has comparable or faster computational time than other deep models.

  • Takeaways & Limitations

    The pyramid-based lightweight architecture has potential value for other low- and high-level vision tasks and downstream applications such as object recognition.

Abstract

from arXiv · show

Existing deep convolutional neural networks have found major success in image deraining, but at the expense of an enormous number of parameters. This limits their potential application, for example in mobile devices. In this paper, we propose a lightweight pyramid of networks (LPNet) for single image deraining. Instead of designing a complex network structures, we use domain-specific knowledge to simplify the learning process. Specifically, we find that by introducing the mature Gaussian-Laplacian image pyramid decomposition technology to the neural network, the learning problem at each pyramid level is greatly simplified and can be handled by a relatively shallow network with few parameters. We adopt recursive and residual network structures to build the proposed LPNet, which has less than 8K parameters while still achieving state-of-the-art performance on rain removal. We also discuss the potential value of LPNet for other low- and high-level vision tasks.

I. INTRODUCTION

Rain degrades human perception and computer-vision systems, making efficient single-image rain removal important. The paper situates LPNet among methods addressing this challenging problem without temporal video information.

  • Rain streaks blur and obstruct objects through light refraction and scattering, with heavier rain increasing the severity.
  • Rainy inputs degrade computer-vision algorithms designed under clear-image assumptions, affecting applications such as autonomous driving and surveillance.
  • 1) Video based methods:: Video deraining methods exploit spatial and temporal information but require temporal video content.
  • 2) Single-image methods:: Single-image deraining is harder than video deraining because individual images contain substantially less information.
  • 2) Single-image methods:: Prior single-image approaches include kernels, low-rank approximations, dictionary learning, mixture models, and local-gradient methods.

B. Our contributions

LPNet addresses the parameter cost of deep image-deraining networks by combining Gaussian-Laplacian pyramids with lightweight recursive and residual subnetworks. It decomposes deraining into level-specific subproblems, using multi-task supervision to reconstruct the final image.

  • B. Our contributions: The network contains fewer than 8K parameters while maintaining excellent single-image deraining performance.
  • B. Our contributions: Recursive and residual subnetworks reconstruct a Gaussian pyramid of the derained image through level-specific losses and multi-task supervision.The final recovered image is the bottom level of the reconstructed Gaussian pyramid.
  • B. Our contributions: LPNet uses Gaussian-Laplacian pyramids to divide image deraining into easier subproblems at different scales.The pyramid separates large-scale edges from small-scale details, allowing each level to use a simple, lightweight subnetwork.
  • B. Our contributions: LPNet combines classical pyramid processing with CNNs to achieve comparable performance using relatively shallow networks with few parameters.The paper presents this combination as substantially more lightweight than deep networks with comparable performance.
  • B. Our contributions: The pipeline decomposes a rainy image into a Laplacian pyramid, trains a subnetwork at each level, and outputs the bottom Gaussian-pyramid level as the derained result.

A. Motivation

LPNet decomposes rainy images into Laplacian-pyramid levels, assigning each level a lightweight sub-network to reconstruct a derained Gaussian pyramid. This multi-scale design exploits sparsity and near-identity mappings while using residual and recursive learning.

  • B. Stage 1: The Laplacian pyramid: Laplacian-pyramid decomposition separates background, rain streaks, and details across spatial scales, simplifying deraining into level-specific high-frequency problems.The pyramid also exploits sparsity and has low computation cost because it relies mainly on Gaussian-filtering convolutions.
  • C. Stage 2: Sub-network structure: Residual learning is appropriate because pyramid-level processing is closer to identity mapping than direct image-domain deraining.Higher levels are more similar between rainy and clean representations, while lower levels retain rain streaks and details.
  • C. Stage 2: Sub-network structure: Each pyramid level receives an independently built sub-network that predicts the corresponding clean Gaussian-pyramid level.The final derained image is obtained from the bottom level of the reconstructed Gaussian pyramid.
  • C. Stage 2: Sub-network structure: Recursive blocks share parameters across inference layers, reducing the number of trainable parameters while residual connections support information and gradient propagation.Each recursive block uses three convolutional operations, with shared weights and biases across T blocks.
  • C. Stage 2: Sub-network structure: The reconstructed Gaussian pyramid uses nonnegative outputs at each level, and its bottom level provides the final derained image.The reconstruction combines predicted Laplacian levels with upsampled higher-level Gaussian outputs.

D. Loss Function

LPNet replaces a single MSE objective with level-specific combinations of ℓ1 and SSIM losses. The choice follows the different frequency characteristics of lower and higher pyramid levels.

  • D. Loss Function: MSE can produce over-smoothed results because its squared penalty handles image edges poorly.LPNet therefore minimizes a combination of alternative losses for each sub-network.
  • D. Loss Function: SSIM loss trains lower pyramid levels, where finer details and rain streaks require preserving high-frequency information.The paper assigns SSIM to levels {1, 2}.
  • D. Loss Function: ℓ1 loss trains higher pyramid levels, which contain larger structures and smoother background areas.The paper uses ℓ1 loss for all levels in addition to SSIM at levels {1, 2}.

E. Removing batch normalization

LPNet removes batch normalization because image-pyramid feature distributions, especially at lower Laplacian levels, are sparse rather than Gaussian.

  • E. Removing batch normalization: Batch normalization constrains feature maps toward a Gaussian distribution, which does not match the sparse distributions observed at lower Laplacian levels.The authors use this observation to argue that image-pyramid technology permits removing BN and improving network flexibility.

F. Parameter settings

LPNet uses a fixed five-level pyramid, shared architectural patterns with level-specific widths, and recursive blocks to keep the model lightweight. Training uses synthetic clean/rainy patches in an end-to-end setup.

  • F. Parameter settings: LPNet decomposes RGB images into a 5-level Laplacian pyramid using a fixed Gaussian smoothing kernel.The kernel is [0.0625, 0.25, 0.375, 0.25, 0.0625].
  • F. Parameter settings: Kernel sizes are 3 × 3 for W^{0,1,3,4}, while W^{2} uses 1 × 1 to increase non-linearity and reduce parameters.Each sub-network has the same structure but a different number of kernels.
  • F. Parameter settings: Kernel counts decrease from low to high pyramid levels as 16, 8, 4, 2, and 1 because higher levels are closer to identity mappings.The top-level sub-network uses one kernel because it mainly performs global contrast adjustment.
  • F. Parameter settings: 7,548 trainable parameters result from the diminishing level widths and recursive design.The authors report this as far fewer than the hundreds of thousands often used in deep learning.
  • F. Parameter settings: Training uses one million 80 × 80 clean/rainy patch pairs generated from synthetic data and runs end-to-end with Adam for 3 epochs.The dataset contains 1,800 heavy-rain images and 200 light-rain images; the mini-batch size is 10 and the learning rate is 0.001.

III. EXPERIMENTS

LPNet is evaluated on synthetic and real-world rainy images using visual, quantitative, and subjective measures. It achieves comparable visual quality to JORDER while using substantially fewer parameters, and shows a dehazing effect on dense rain.

  • Synthetic data: LPNet has comparable visual results with JORDER and outperforms GMM, SRCNN, and DDN on synthetic datasets.GMM fails on heavy rain, while SRCNN and DDN tend to generate obvious artifacts.
  • Synthetic data: LPNet has comparable SSIM values with JORDER while outperforming other methods, despite a lower PSNR value than JORDER.The authors attribute the discrepancy to PSNR measuring global pixel errors without considering local image characteristics.
  • Real-world data: LPNet trained on synthetic data generates consistently promising derained results on real-world images with different kinds of rain streaks.The light-rain model is used for testing on real-world images.
  • Real-world data: A user study of 50 real-world rainy images and 20 participants provides additional support for LPNet's real-world deraining improvement.Participants ranked outputs from five methods and the original rainy image on a 1-to-5 quality scale.
  • Real-world data: LPNet trained on heavy-rain images has a dehazing effect under dense rain that can further improve visual quality.The effect is attributed to contrast adjustment by the highest-level low-pass sub-network.

C. Running time and convergence

LPNet is designed for efficient inference and rapid training convergence. Its shallow per-level networks yield competitive or faster computation than other deep models on both CPU and GPU.

  • Running time: Running time is compared across three image sizes, with each size tested over 100 images.Deep CNN methods are evaluated on CPU and GPU, while GMM is implemented on CPUs.
  • Running time: LPNet has comparable and sometimes faster computational time than other deep models on both CPU and GPU.The authors attribute this to relatively shallow networks at each pyramid level, requiring fewer convolutions.
  • Convergence: LPNet converges quickly when trained on both light- and heavy-rain datasets.Training error for heavy rain exhibits vibration because heavy rain streaks are harder to handle.

D. Parameter settings

LPNet uses diminishing parameter counts across pyramid levels to balance effectiveness and efficiency. Increasing parameters improves quantitative scores but requires more storage and computation while producing nearly identical visual quality.

  • Increasing parameter number: Using 16 feature maps in every convolution layer improves SSIM over JORDER and also improves PSNR on Rain100H.The experiment evaluates an increased-parameter model.
  • Increasing parameter number: Increasing parameter numbers requires more storage and computing resources, while visual quality is almost the same.The authors therefore retain the diminishing parameter setting to balance effectiveness and efficiency.
  • Skip connections: Skip connections are retained because they preserve information flow and improve gradient back-propagation during training.The paper studies training curves with and without all skip connections on the heavy-rain dataset.

3) Loss function:

The paper uses SSIM alongside pixel-based losses to emphasize local visual characteristics, then examines LPNet's broader restoration and detection applications. These experiments suggest utility beyond deraining while preserving a lightweight deployment profile.

  • Loss function: SSIM is included in the loss because it captures local contrast, luminance, and details associated with rain streaks and visual quality.The authors contrast this local sensitivity with PSNR's emphasis on global pixel error.
  • Low-level vision tasks: LPNet generalizes to image denoising and JPEG artifact reduction, which share corruption by high-frequency content with rainy images.The result is presented as evidence of potential value for similar image restoration problems.
  • Loss function: Using SSIM + ℓ1 loss generates a sharper deraining result.This comparison is shown in an example using different losses.
  • High-level vision tasks: After LPNet deraining, Faster R-CNN detection performance shows notable improvement over naive Faster R-CNN on a real-world rainy image.Rain causes missed detections and low recognition confidence before deraining.
  • High-level vision tasks: Processing a 1024×1024 color image takes 3.7 seconds for Faster R-CNN and 4.0 seconds for LPNet + Faster R-CNN.The small increase illustrates the lightweight architecture's limited added complexity in this experiment.

IV. CONCLUSION

LPNet is a lightweight Gaussian-Laplacian pyramid network for single-image deraining, with potential value beyond rain removal in other low- and high-level vision tasks.

  • LPNet is based on the classical Gaussian-Laplacian pyramid and predicts the clean Gaussian pyramid from the input Laplacian pyramid.The network uses several sub-networks to process the pyramid representation.
  • Recursive blocks share parameters, enabling LPNet to use fewer than 8K parameters while maintaining good deraining performance.
  • LPNet’s generality and lightweight architecture give it potential value for other low- and high-level vision tasks.
Loading 1805.06173v1…