PaperlayerView exact revision on arXiv

Source-linked AI summary

A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing

Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, David Wipf

arXiv:1708.03474v2cs.CV

TL;DR

Low-level vision tasks such as reflection removal and image smoothing need edge-sensitive processing, while existing pipelines often rely on task-specific filtering or optimization. The paper proposes CEILNet, which cascades CNN-based target-edge prediction with image reconstruction and trains reflection removal using synthetic weak supervision. It reports a first learning-based single-image reflection-removal solution and state-of-the-art image-smoothing performance, while noting that reflection removal remains difficult and depends on a relative-blurriness assumption.

  • Problem

    Existing edge-sensitive low-level vision pipelines rely on traditional filtering or optimization, while single-image reflection removal remains ill-posed and difficult to supervise.

  • Method

    CEILNet cascades an edge-prediction CNN with an image-reconstruction CNN and uses synthetic reflections for weakly supervised reflection-removal training.

  • Results

    CEILNet provides the first learning-based single-image reflection-removal solution and achieves state-of-the-art visual and numerical performance on image smoothing.

  • Takeaways & Limitations

    A convolution-only edge-and-image pipeline transfers across reflection removal and image smoothing without handcrafted, application-specific components.

  • Takeaways & Limitations

    Reflection removal assumes reflections are somewhat blurrier than the background, and real-image evaluation lacks ground-truth databases; recovered backgrounds may therefore remain imperfect.

Abstract

from arXiv · show

This paper proposes a deep neural network structure that exploits edge information in addressing representative low-level vision tasks such as layer separation and image filtering. Unlike most other deep learning strategies applied in this context, our approach tackles these challenging problems by estimating edges and reconstructing images using only cascaded convolutional layers arranged such that no handcrafted or application-specific image-processing components are required. We apply the resulting transferrable pipeline to two different problem domains that are both sensitive to edges, namely, single image reflection removal and image smoothing. For the former, using a mild reflection smoothness assumption and a novel synthetic data generation method that acts as a type of weak supervision, our network is able to solve much more difficult reflection cases that cannot be handled by previous methods. For the latter, we also exceed the state-of-the-art quantitative and qualitative results by wide margins. In all cases, the proposed framework is simple, fast, and easy to transfer across disparate domains.

1. Introduction

The paper introduces CEILNet as a generic, fully convolutional framework for edge-sensitive tasks, targeting single-image reflection removal and image smoothing. It replaces task-specific filtering or optimization components with cascaded edge prediction and image reconstruction, supported by synthetic reflection data.

  • Prior image-smoothing networks often combine CNNs with traditional filtering or optimization, limiting extensibility to other effects and layer-separation problems.
  • Single-image reflection removal is ill-posed because the observed image can be decomposed into infinitely many background and reflection layers.
  • CEILNet predicts target-image edges first, then reconstructs the target image with cascaded CNNs and no handcrafted modules.
  • The framework is applied to both single-image reflection removal and image smoothing as a transferable architecture for edge-sensitive processing.
  • A weakly supervised reflection-removal pipeline trains on synthetic reflections designed to mimic physical properties of natural scenes.
  • The authors report state-of-the-art visual and numerical image-smoothing performance and improved handling of difficult reflection cases.

3. Network Structure

CEILNet uses two cascaded CNN sub-networks: E-CNN predicts a target edge map, and I-CNN reconstructs the target image from the source image and that prediction. The same convolutional design supports both reflection removal and image smoothing.

  • E-CNN: The Edge Prediction Network: E-CNN predicts the target image’s edge map from the source image, rather than predicting the source image’s edges.
  • E-CNN: The Edge Prediction Network: The edge representation is the mean absolute RGB color difference between each pixel and its four-connected neighbors.
  • I-CNN: The Image Reconstruction Network: I-CNN reconstructs the target image by processing the source image together with the E-CNN-predicted target edge map.
  • I-CNN: The Image Reconstruction Network: For image smoothing, I-CNN replaces traditional fixed filtering or optimization-based postprocessing with a multi-purpose, data-driven reconstruction module.
  • Two-phase network training: Training first optimizes the sub-networks separately, then fine-tunes the complete network end-to-end so they can cooperate.
  • Network Structure: The two sub-networks share the same overall CNN structure but differ in final output channels: 1 for E-CNN and 3 for I-CNN.

4. Network Training

CEILNet is trained in two phases, combining independently trained edge and image subnetworks before end-to-end fine-tuning. Training data are generated through task-specific procedures, including weakly supervised synthetic reflections and filter-produced smoothed images.

  • The training pipeline applies independently across data sources, followed by application-specific training-sample generation.
  • The network first trains its subnetworks separately, then fine-tunes the complete model end-to-end so the subnetworks can cooperate.
  • Edge predictions use mean squared error, while image predictions combine color and gradient discrepancies to discourage blurry outputs.
  • Training uses ADAM with batch size 1, initially learning rate 0.01, and fine-tuning learning rate 0.001; loss coefficients are α = 0.2 and β = γ = 0.4.
  • For reflection removal, synthetic samples use weak supervision and assume reflections are somewhat blurrier than sharper background layers.
  • Adaptive subtraction and clipping generate reflection-free regions, strong reflections, and better-preserved reflection contrast than naive brightness scaling.
  • For image smoothing, training and testing data are produced by applying existing filters to image databases.

5. Experiments

Experiments evaluate CEILNet through architectural self-comparisons and task-level comparisons for reflection removal and image smoothing. Across these tests, the predicted target edge map, cascaded design, synthetic reflection data, and fully convolutional implementation support stronger qualitative, quantitative, and runtime results.

  • Network Analysis: Target-edge prediction substantially improves image reconstruction over I-CNN alone, input-edge guidance, and Domain Transform for L0 smoothing.CEILNet with predicted edges outperformed the alternatives by significant or large margins, with qualitative confirmation in Figure 4.
  • Network Analysis: CEILNet removed more reflection and produced clearer backgrounds than I-CNN without E-CNN and naive image mixing on a real image.The synthetic-data strategy avoids brightness overflow, preserves reflection contrast, and allows reflection-free regions and strong reflections to coexist.
  • Reflection Removal: CEILNet significantly outperformed Li and Brown on PSNR and SSIM across 100 synthetic reflection-removal test images.Qualitative comparisons on synthetic and real images found that CEILNet handled bright and shiny reflections without jeopardizing reflection-free regions, whereas often produced blurry reflection layers.
  • Image Smoothing: CEILNet achieved better PSNR and SSIM than Xu et al. for all 8 smoothing filters and significantly outperformed for all 4 publicly available filters.The comparison used 100 test images; PSNR and SSIM are reported for the smoothing tasks, with larger values better.
  • Image Smoothing: CEILNet ran faster than the other evaluated methods in most image-size cases and approximated traditional algorithms at about 100 fps for 1280×720 images.Running times were measured on the same NVIDIA DGX-1 computer for eight traditional algorithms and three deep-learning methods.

6. Conclusions and Future Work

CEILNet is presented as a generic architecture for edge-sensitive image processing, providing a learning-based solution for single-image reflection removal and advancing deep-network image smoothing. The authors identify testing on additional image-processing tasks as future work.

  • CEILNet provides a generic deep architecture for edge-sensitive image processing.
  • The framework supplies a learning-based solution to single-image reflection removal using a novel reflection-image synthesis method.
  • The authors report significant advances over the state of the art in DNN-based image smoothing.
  • Future work includes testing CEILNet on more image-processing tasks.The authors mention promising preliminary results for image denoising and inpainting.

1 Outline

The supplementary document adds analyses and results covering synthetic reflection-data generation, network comparisons, edge-map visualizations, and qualitative comparisons across reflection removal and smoothing.

  • The supplement analyzes the synthetic data-generation method used to train CEILNet for reflection removal.
  • It compares varying-depth I-CNN networks with the full CEILNet pipeline.
  • It visualizes edge maps predicted by E-CNN for reflection removal and image smoothing.
  • It presents additional visual comparisons for deep image smoothing, reflection removal, and different smoothing filters.

2 Complete description of our synthetic reflection image generation process

The proposed synthetic reflection-image process uses two natural images, blurs and attenuates the reflection without scaling, and constructs a synthesized image with known background and reflection layers. This strategy is motivated by limitations of naive linear mixing and is illustrated through a stepwise pipeline.

  • Naively mixing background and reflection images with I = ν1B + ν2R scales both layers and does not perform well for training.
  • The alternative method sums an unmodified background image with an attenuated reflection image produced by subtracting one adaptively computed scalar.
  • Subtracting a scalar largely avoids oversaturation, preserves reflection gradients, and accommodates strong reflections without the color-range constraints of scaling.
  • The pipeline randomly selects normalized natural images as B and R, then applies Gaussian blur to R with σ sampled uniformly from [2, 5].
  • It tentatively forms I = B + ˜R, computes the mean overflow above 1, and subtracts γ · (m − 1) from the reflection layer with γ = 1.3.
  • The modified reflection is clipped to [0, 1], and the final image is generated by combining it with the background before clipping the image colors.
  • The generated image uses B as ground-truth background and I − B as ground-truth reflection.

3 Performance using an I-CNN without predicted edges

The edge-aware CEILNet pipeline outperforms an image-only CNN even when the latter is made deeper and has comparable parameter count, indicating that the benefit is not explained by model size alone.

  • 37.10 dB PSNR for CEILNet exceeds 33.37 dB PSNR for the best 70-layer I-CNN.
  • The 70-layer I-CNN performs worse despite having more layers, while CEILNet has approximately the same parameter count as a 64-layer I-CNN.
  • The I-CNN-only performance saturates quickly as convolutional layers are added.
  • The comparison attributes CEILNet’s advantage to purposeful integration with the E-CNN rather than simply to a larger parameter set.

4 Edge map visualization and analysis

The predicted edge maps support both image smoothing and reflection removal by preserving salient structure while suppressing irrelevant or reflection-specific edges. Detailed comparisons indicate that CEILNet avoids the inflexible post-processing limitations of prior approaches.

  • Edge map behavior: In image smoothing, E-CNN removes insignificant details while preserving and visually clarifying prominent edges.The predicted edge map guides I-CNN in generating the smoothed image.
  • Edge map behavior: In reflection removal, E-CNN targets background-structure edges and suppresses edges belonging to reflections before I-CNN reconstructs the background.This differs from smoothing, where the target edge map represents the desired smoothed image structure.
  • Edge map behavior: Figure 3 visualizes predicted edge maps for smoothing and reflection removal, with ground truth denoted by GT.The top two rows show smoothing examples, while the bottom two rows show reflection-removal examples.
  • Comparison with prior methods: Additional qualitative comparisons on image smoothing evaluate methods trained to approximate L0 smoothing, with PSNR values shown in Figure 4.The cited comparison states that the proposed results are visually closer to ground truth.
  • Comparison with prior methods: Prior smoothing methods predict gradients or weight maps and then rely on separate optimization or recursive filtering operations.These additional operations can spread gradient errors or make it difficult to approximate diverse edge-aware filters.

6 Detailed comparison with our baselines and previous methods on the reflection removal task

CEILNet is compared with naïve training, an image-reconstruction-only baseline, and prior reflection-removal methods on synthetic and real images. The comparisons emphasize the value of synthetic training and edge prediction, while highlighting the difficulty of real-image evaluation.

  • Baselines and prior methods: CEILNet outperforms naïvely trained CEILNet and I-CNN-only on real reflection images, removing more reflections and producing cleaner results.The naïve baseline uses constant coefficients for background and reflection layers, while I-CNN-only omits predicted edges.
  • Baselines and prior methods: The method of performs comparably when its assumptions hold, but CEILNet excels on more difficult reflection cases.Method assumes smooth reflection color transitions and reflections covering most of the image.
  • Baselines and prior methods: Method is restricted to single-image reflection examples containing ghost effects with two ghost layers.For reflection images beyond that scope, the cited passage states that it does not work.
  • Evaluation scope: Because no real reflection database with ground truth exists, the model is trained solely on synthetically generated reflection images.The authors report reasonable results in most real-world situations, although reflections may not be completely removed.

8 More results for image smoothing

Additional image-smoothing results show CEILNet approximations for five established smoothing algorithms. The supplied figures cover L0, L1, RTV, RGF, and WLS filters.

  • Filter approximations: Additional results evaluate CEILNet on five image-smoothing targets: L0, L1, RTV, RGF, and WLS.Figures 12–16 correspond respectively to these filters.
  • Filter approximations: Figure 12 presents CEILNet’s approximation of the L0 image-smoothing algorithm.
  • Filter approximations: Figure 13 presents CEILNet’s approximation of the L1 image-smoothing algorithm.
  • Filter approximations: Figure 14 presents CEILNet’s approximation of the RTV image-smoothing algorithm.
  • Filter approximations: Figures 15 and 16 present CEILNet’s approximations of the RGF and WLS image-smoothing algorithms.
Loading 1708.03474v2…