Source-linked AI summary
Single Image Reflection Separation with Perceptual Losses
Xuaner Zhang, Ren Ng, Qifeng Chen
TL;DR
Single-image reflection separation is an ill-posed problem because an observed image mixes transmission and reflection layers. The paper trains a fully convolutional network with perceptual and exclusion losses, builds a ground-truth real-world dataset, and reports superior benchmark and user-study performance while extending the model to flare removal and dehazing. A challenging boundary remains when reflection is nearly as sharp as transmission.
Problem
Recovering the transmission layer from a single image is ill-posed because the image is approximately modeled as I = T + R.
Method
A fully convolutional network uses low- and high-level perceptual losses, gradient-domain exclusion, and a real-world dataset with ground-truth transmission layers.
Results
The method outperforms state-of-the-art approaches in SSIM, PSNR, and perceptual user studies, and transfers to flare removal and dehazing.
Takeaways & Limitations
Combined low- and high-level image features are effective for decomposing images into transmission and reflection layers, with the approach extending to two enhancement applications.
Takeaways & Limitations
Performance remains challenging when the reflection layer is almost as sharp as the transmission layer in a real-world image.
Abstract
from arXiv · showhide
We present an approach to separating reflection from a single image. The approach uses a fully convolutional network trained end-to-end with losses that exploit low-level and high-level image information. Our loss function includes two perceptual losses: a feature loss from a visual perception network, and an adversarial loss that encodes characteristics of images in the transmission layers. We also propose a novel exclusion loss that enforces pixel-level layer separation. We create a dataset of real-world images with reflection and corresponding ground-truth transmission layers for quantitative evaluation and model training. We validate our method through comprehensive quantitative experiments and show that our approach outperforms state-of-the-art reflection removal methods in PSNR, SSIM, and perceptual user study. We also extend our method to two other image enhancement tasks to demonstrate the generality of our approach.
1. Introduction
Single-image reflection separation aims to recover clean transmission content from an ill-posed mixture of transmission and reflection layers. The paper proposes perceptual, end-to-end separation with a new dataset and reports stronger results than existing methods.
- Motivation: Reflection is common in photographs and can damage image quality, motivating recovery of clean transmission content.The task is especially relevant for photographs taken through aquarium glass or skyscraper windows.
- Problem: Given I = T + R, the goal is to recover transmission layer T from input image I, but the problem is ill-posed without additional constraints or priors.Earlier approaches often use multiple images, user guidance, or hand-crafted priors that may be inconvenient or fail to generalize.
- Approach: The proposed fully convolutional network takes one image and synthesizes both reflection and transmission layers using low-level and high-level perceptual information.Its perceptual supervision includes a visual-network feature loss and an adversarial loss refining the transmission output.
- Approach: A gradient-domain exclusion loss encourages independence between transmission and reflection layers at the pixel level.The loss is designed to emphasize separation of the layers rather than overlapping edge structure.
- Evaluation: The authors build a real-world dataset with corresponding ground-truth transmission layers for quantitative evaluation and comparison with existing algorithms.The dataset supports training and evaluation across diverse indoor and outdoor environments.
- Results: Experiments on real and synthetic data indicate better SSIM, PSNR, and perceptual user-study performance than state-of-the-art methods, with direct extension to flare removal and dehazing.The trained reflection-separation model is applied to both additional image-enhancement tasks.
2. Related Work
Prior reflection-removal methods use multiple images, user assistance, hand-crafted priors, or low-level losses. The paper positions perceptual learning as a way to address limitations of these approaches and introduces a quantitative real-world benchmark.
- Multiple-image methods: Multiple-image methods use motion, polarization, or gradient cues, but capturing multiple images can be impossible for existing or legacy photographs.These methods infer separation from calibrated-camera motion, motion parallax, or polarized images.
- Single-image methods: Single-image methods commonly rely on gradient sparsity and other low-level heuristics, which are limited when high-level image understanding is needed.Some methods combine gradient constraints with data fidelity or user assistance.
- Hand-crafted priors: Relative-smoothness priors model reflection as out of focus, but the formulation can break down when the reflection layer has high contrast.Depth-of-field guidance and ghost-effect modeling address narrower variants of the problem.
- Deep learning methods: CEILNet learns from color and gradient losses and an intermediate edge map, but does not explicitly use perceptual information during training.The paper identifies this lack of perceptual training as its main difference from CEILNet.
- Datasets: The paper builds a real-world benchmark dataset because an earlier reflection-removal dataset had not been publicly released at submission time.The dataset is intended to support quantitative comparisons on real-world images.
3. Overview
The method uses a fully convolutional network to decompose one reflection-containing image into transmission and reflection layers, trained with feature, adversarial, and exclusion losses. It augments the input with VGG-19 hypercolumns and uses a large-receptive-field architecture for evaluation against existing methods.
- The network decomposes a single input image into predicted transmission and reflection layers using one fully convolutional model.The model is trained on paired images containing the input, transmission layer, and reflection layer.
- The combined objective contains feature, adversarial, and exclusion losses, weighted by w1 = 0.1, w2 = 0.01, and w3 = 1.The feature loss compares feature-space representations, the adversarial loss refines realism, and the exclusion loss separates layers in the gradient domain.
- VGG-19 hypercolumn features from five selected layers augment the input with 1472 dimensions of pretrained visual information.The augmented input concatenates the original three-channel image with these hypercolumn features.
- A 513 × 513 receptive field and dilated convolutions aggregate global image information in the fully convolutional architecture.The network uses eight 3 × 3 dilated-convolution layers with dilation rates from 1 to 128.
- The evaluation compares the method with CEILNet, an optimization-based approach, and Pix2pix on synthetic and collected real-world images.The experiments use publicly available CEILNet data together with the authors' real-world dataset.
4. Training
Training combines perceptual feature supervision, conditional adversarial refinement, and gradient-domain exclusion with reflection-layer supervision. Ablations and implementation choices show how these components affect smoothness, residual reflection, realism, and robustness to unequal gradient magnitudes.
- Feature loss: The feature loss compares predicted and ground-truth transmission images in VGG-19 feature space across five selected layers.It uses an L1 difference between corresponding pretrained representations, with weights balancing the layer terms.
- Adversarial loss: The conditional adversarial loss uses a discriminator that judges transmission-image patches conditioned on the input image.The adversarial term addresses unrealistic color degradation and subtle residuals in the transmission output.
- Loss ablations: Without feature loss, outputs become overly smooth, while removing adversarial refinement eliminates some color and residual improvements.The ablation reports cleaner and more natural results when adversarial refinement is retained.
- Exclusion loss: The exclusion loss minimizes correlation between transmission and reflection gradients using normalized gradient products at multiple spatial resolutions.The formulation uses Frobenius normalization, element-wise multiplication, and downsampling factors of 2^(n−1).
- Exclusion loss: Normalization factors are critical because reflection gradients may be far smaller or larger than transmission gradients, causing unbalanced layer updates.The figure shows that omitting normalization heavily suppresses reflection prediction, whereas normalization yields better separation.
- Loss ablations: Disabling the exclusion loss leaves visible reflection content in the predicted transmission image.This ablation supports its role in pixel-level layer separation.
- Implementation: A color-space L1 loss constrains the predicted reflection layer when ground-truth reflection is available, but it is disabled for real images.The authors avoid estimating real-image reflection by subtraction because overexposure can violate I = R + T and introduce artifacts.
- Implementation: Training uses 5000 synthetic images and patches from 90 real-world images for 250 epochs with batch size 1 and learning rate 10^-4.Image patches are randomly resized while preserving their original aspect ratio.
5. Dataset
The paper builds synthetic and real reflection datasets, including paired real images with ground-truth transmission layers, to support training and quantitative evaluation.
- Synthetic data: 5,000 random Flickr image pairs form the synthetic dataset, with one image assigned as transmission and the other as reflection.The paired images combine indoor and outdoor content.
- Synthetic data: Gaussian smoothing applies random 3–17-pixel kernels to reflection images, modeling different focal-plane blurriness.The construction assumes the transmission and reflection layers have different focal planes.
- Synthetic data: Linear-space composition varies reflection intensity decay because real reflections can be comparable to or brighter than transmission.
- Real data: 110 real image pairs capture reflected scenes and corresponding transmission images using the same exposure settings, with glass removed for ground truth.Images were captured using a Canon 600D on a tripod with portable glass.
- Evaluation setup: Table 1 compares three previous methods and an input-image baseline on CEILNet synthetic data and the real test set.
- Real data: The real dataset spans indoor and outdoor environments, three lighting conditions, viewing angles, and apertures from f/2.0 to f/16.Ninety images provide training patches and 20 images support quantitative evaluation.
6. Experiments
Experiments compare the method with prior approaches on synthetic and real images using quantitative metrics, user preferences, qualitative results, and loss ablations.
- Quantitative comparison: The method demonstrates strong quantitative performance over previous works on both synthetic and real data.PSNR and SSIM are computed against ground-truth transmission layers.
- User study: 84.2% of comparisons against CEILNet and 87.8% against Li and Brown rated the method as containing less reflection.The differences were statistically significant with p < 10^-3 across 20 users.
- Ablation study: Table 3 evaluates retrained models with individual losses removed and an adversarial-only variant on synthetic and real data.The complete model shows better performance on both data types.
- Qualitative results: Qualitative evaluations use real images from the authors’ dataset with ground truth and CEILNet images without ground truth.
- Qualitative results: On the authors’ real dataset, the method produces better and cleaner transmission and reflection predictions than CEILNet.
- Qualitative results: On the CEILNET dataset, the method predicts a cleaner reflection layer than CEILNet and Li and Brown despite lacking real-data reflection supervision.
- Ablation study: Replacing feature loss with color-space L1 produces overly smooth outputs, while removing exclusion loss allows reflection content into transmission predictions.
7. Extensions
The trained reflection-separation model is directly applied to flare removal and dehazing, treating both as removal of an undesirable image layer.
- Extension applications: The model performs flare removal and dehazing without training or fine-tuning on either task’s dataset.
- Extension applications: Flare removal targets optical artifacts caused by reflection and scattering inside the lens, while dehazing targets the hazy layer.
- Extension applications: The extension results show image enhancement by removing undesirable layers from input images.
8. Discussion
The paper concludes that end-to-end learning with perceptual and exclusion losses is effective for separating reflection and transmission layers. A real-world dataset with ground-truth transmission layers supports evaluation, while challenging cases remain open for improvement.
- The approach combines low-level and high-level image features to decompose an image into transmission and reflection layers.The model uses perceptual losses together with a customized exclusion loss.
- A new real-world reflection-removal dataset contains ground-truth transmission layers for evaluating different methods.
- The method was extended to two other photo-enhancement applications to demonstrate generality for layer-separation problems.
- The model outperforms state-of-the-art approaches on both synthetic and real images, but its performance can still improve.
- A challenging real-world case occurs when the reflection layer is almost as sharp as the transmission layer.