Source-linked AI summary
Single Image Reflection Removal Exploiting Misaligned Training Data and Network Enhancements
Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, Hua Huang
TL;DR
Single-image reflection removal remains difficult because the problem is ill-posed and densely labeled training data are insufficient, especially for general real-world images. The paper enhances the network with contextual encoding modules and introduces an alignment-invariant loss to exploit easier-to-collect misaligned data, achieving state-of-the-art real-world benchmark performance.
Problem
Single-image reflection removal remains largely unsolved across disparate imaging conditions and varying scene content because of intrinsic ill-posedness and insufficient densely labeled training data.
Method
The paper embeds channel-wise and multi-scale spatial context modules into a reflection-removal network and introduces an alignment-invariant loss for misaligned training pairs.
Results
The approach sets a new state-of-the-art on real-world single-image reflection-removal benchmarks quantitatively and visually, while misaligned-data supervision surpasses aligned-only results.
Takeaways & Limitations
Context encoding helps discriminate and suppress reflection components, while misaligned real-world training data can significantly alleviate the burden of data capture.
Takeaways & Limitations
The alignment-invariant loss can handle shifts of up to 20 pixels.
Abstract
from arXiv · showhide
Removing undesirable reflections from a single image captured through a glass window is of practical importance to visual computing systems. Although state-of-the-art methods can obtain decent results in certain situations, performance declines significantly when tackling more general real-world cases. These failures stem from the intrinsic difficulty of single image reflection removal -- the fundamental ill-posedness of the problem, and the insufficiency of densely-labeled training data needed for resolving this ambiguity within learning-based neural network pipelines. In this paper, we address these issues by exploiting targeted network enhancements and the novel use of misaligned data. For the former, we augment a baseline network architecture by embedding context encoding modules that are capable of leveraging high-level contextual clues to reduce indeterminacy within areas containing strong reflections. For the latter, we introduce an alignment-invariant loss function that facilitates exploiting misaligned real-world training data that is much easier to collect. Experimental results collectively show that our method outperforms the state-of-the-art with aligned data, and that significant improvements are possible when using additional misaligned data.
1. Introduction
Single-image reflection removal remains difficult because separating transmitted content from reflections is ill-posed and densely labeled real-world training data are scarce. The paper addresses these challenges with contextual network modules, an alignment-invariant loss, and misaligned training pairs.
- Single-image reflection removal remains largely unsolved across varied imaging conditions and scene content.
- The problem is ill-posed because one captured image admits infinitely many decompositions into transmitted and reflection layers with overlapping appearances.The paper models the image as I = T + R.
- Real densely labeled reflection-removal data are scarce, while heuristic synthesis can create a domain gap that harms accuracy on real images.
- Context encoding modules use channel-wise and multi-scale spatial context to distinguish transmitted content from reflections, especially in strongly reflected regions.
- An alignment-invariant loss enables training with misaligned input and reflection-free pairs that are easier to collect than strictly aligned data.Misalignment can result from camera or object movement, while aligned collection requires static camera, scene, and lighting conditions.
2. Related Work
Related work includes traditional optimization methods that rely on priors or user input and CNN-based methods that replace handcrafted assumptions with data-driven learning. Existing approaches use varied layer-prediction architectures and commonly train with image-intensity discrepancies.
- Traditional methods: Traditional reflection-removal methods use manual intervention or priors such as smoothness, depth-of-field confidence, ghost cues, or gradient sparsity.
- Traditional methods: These traditional methods can perform decently when their assumptions hold but generalize poorly across diverse real-world imaging conditions and scene content.
- Deep learning based methods: CNN-based methods replace handcrafted priors with data-driven learning for single-image reflection removal.
- Deep learning based methods: CNN approaches have predicted background edges, jointly estimated edges and intensities, or interleaved background and reflection layers, often using MSE-based training.
3. Approach
The approach combines a context-enhanced CNN with losses for both aligned and misaligned training data. Context encoding broadens feature information, while perceptual and adversarial objectives support reflection removal and alignment-invariant training.
- Base architecture: The network starts from a reconstruction CNN, removes batch normalization, widens feature maps from 64 to 256, and adds pretrained VGG-19 hypercolumn features.These changes define the baseline architecture before targeted context modules are introduced.
- Motivation: Performance on real reflection images degraded appreciably relative to synthetic data, motivating broader contextual information in local convolutional filters.The degradation was especially observed on 20 real images from [48].
- Context encoding: Multi-scale spatial context applies pyramid pooling at sizes 4, 8, 16, and 32 before final image construction.This complements channel-wise contextual information with broader spatial information within each channel.
- Context encoding: Channel-wise context uses global average-pooled channel descriptors to predict channel gates that recalibrate feature maps using global statistics.The module introduces only modest parameter overhead through small additional weight matrices.
- Aligned-data loss: Aligned training combines pixel, VGG-19 feature, and adversarial losses to penalize image differences and improve background-image realism.The pixel loss includes intensity and horizontal and vertical gradient differences; the feature loss uses conv2_2, conv3_2, conv4_2, and conv5_2.
4. Experiments
Experiments show that ERRNet’s contextual modules improve reflection removal over the baseline, while alignment-invariant supervision makes misaligned data beneficial. On real benchmarks, ERRNet reaches state-of-the-art performance on some datasets, although Wild remains difficult for all methods.
- Ablation Study: ERRNet combines BaseNet with channel-wise and multi-scale spatial context modules, and the full model performs best among the compared settings.The ablation compares BaseNet, each context module separately, and ERRNet with both modules.
- Ablation Study: BaseNet leaves obvious reflection residuals, whereas ERRNet produces cleaner transmitted images by better encoding contextual clues.The visual comparison focuses on strong-reflection regions where local features are difficult to interpret.
- Ablation Study: Increasing aligned real data from 50 to 90 images improves ERRNet’s testing results, but adding 40 unaligned images with pixel-wise loss degrades performance.The experiment compares synthetic data with progressively larger aligned real-data mixtures before testing misaligned supervision.
- Ablation Study: Alignment-invariant and contextual losses both outperform analogous aligned-only training when supervising unaligned data, while the proposed loss is simpler and more computationally efficient.The alignment-invariant loss handles shifts up to 20 pixels and is reported as a lightweight alternative to contextual loss.
- Method Comparison on Benchmarks: ERRNet achieves state-of-the-art results on Real20 and Objects, is comparable to BDN-F on Postcard, and no method surpasses Input on Wild.The benchmark comparison uses PSNR, SSIM, NCC, and LMSE across four real-world datasets.
- Method Comparison on Benchmarks: On real images, all compared methods fail on some strong reflections, but ERRNet more accurately removes many artifacts, including reflected tree branches.The visual comparison includes Real20 images and samples from the collected unaligned dataset.
- Training with Unaligned Data: The collected unaligned dataset contains 450 image pairs, split into 400 training samples and 50 testing samples, from DSLR and smartphone captures.Samples cover varied indoor and outdoor environments, including dynamic scenes with vehicles and people.
- Training with Unaligned Data: Human observers preferred outputs from models fine-tuned with alignment-invariant loss and unaligned data over their raw outputs for both BDN-F and ERRNet.The assessment used 30 observers and 3,000 judgments because no numerical metric was available for unaligned data.
5. Conclusion
The paper proposes an enhanced reflection removal network with context encoding modules and an alignment-invariant loss, enabling the use of misaligned training data. Experiments report state-of-the-art performance on real-world benchmarks, quantitatively and visually.
- The approach combines an enhanced reflection removal network with an alignment-invariant loss function.The loss supports direct use of misaligned training data, while the network targets the difficulty of single image reflection removal.
- Context encoding modules help the network discriminate and suppress the reflection component.They are embedded into the network to extract underlying knowledge from real training data.
- The method sets a new state-of-the-art on real-world single image reflection removal benchmarks.The reported advantage is both quantitative and visual.