Source-linked AI summary
CRRN: Multi-Scale Guided Concurrent Reflection Removal Network
Renjie Wan, Boxin Shi, Ling-Yu Duan, Ah-Hwee Tan, Alex C. Kot
TL;DR
Reflections can obscure scenes viewed through glass, and handcrafted priors may not describe real-world reflections comprehensively. CRRN concurrently combines gradient and image inference with multi-scale guidance, a perceptually motivated loss, and captured real-world training data; the authors report favorable comparisons with state-of-the-art methods and generalization to unseen data.
Problem
Handcrafted priors have limited ability to describe real-world reflections, while existing learning methods retain two-stage processing, pixel-wise losses, and mainly synthetic training data.
Source: §1 · p. 1Method
CRRN concurrently estimates background and reflection layers with multi-scale gradient guidance, a perceptually motivated loss, and a captured real-world reflection dataset.
Source: §1 · pp. 1–2, §4.1 · p. 4Results
CRRN shows better quantitative values and visual qualities than state-of-the-art methods and is reported to generalize effectively to unseen data.
Source: §6 · p. 8Takeaways & Limitations
The authors conclude that the concurrent framework, real-world dataset, and training strategy support effective generalization to unseen data.
Source: §6 · p. 8CRRN’s performance may drop when whole images are dominated by reflections.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Removing the undesired reflections from images taken through the glass is of broad application to various computer vision tasks. Non-learning based methods utilize different handcrafted priors such as the separable sparse gradients caused by different levels of blurs, which often fail due to their limited description capability to the properties of real-world reflections. In this paper, we propose the Concurrent Reflection Removal Network (CRRN) to tackle this problem in a unified framework. Our proposed network integrates image appearance information and multi-scale gradient information with human perception inspired loss function, and is trained on a new dataset with 3250 reflection images taken under diverse real-world scenes. Extensive experiments on a public benchmark dataset show that the proposed method performs favorably against state-of-the-art methods.
1. Introduction
Reflections obscure glass-through scenes, while handcrafted priors and existing learning methods have limitations in real-world reflection modeling and background recovery. CRRN combines concurrent gradient and image inference, multi-scale guidance, a perceptually motivated loss, and a captured dataset.
1. Introduction
- Reflections obstruct, deform, or blur the scene behind glass, reducing visibility and potentially causing computer vision systems to fail.Source: §1 · p. 1
- Existing methods face several limits: handcrafted priors can fail when background and reflection structures resemble each other, while learning methods use two stages, pixel-wise losses, and mainly synthetic training images.The cited limitations include incomplete exploration of multi-scale information for background recovery and blurry predictions from pixel-wise losses.Source: §1 · p. 1
- CRRN’s multi-scale gradient features guide background reconstruction in a concurrent framework that estimates background and reflection layers.The image inference network is guided by associated gradient features from the gradient inference network.Source: §1 · pp. 1–2, Figure 1 · p. 2
- The authors introduce a perceptually motivated loss to suppress blurry artifacts associated with pixel-wise losses.Source: §1 · p. 2
- The authors report capturing a large-scale reflection image dataset for training, which improves CRRN’s performance and generality on real data.They describe it as the first reflection image dataset for data-driven methods.Source: §1 · p. 2
2. Related Work
Related work includes non-learning methods based on handcrafted priors or multiple views, and learning-based approaches. The former can rely on restrictive assumptions or capture facilities, while the latter can retain a two-stage framework.
2. Related Work
- Non-learning methods use handcrafted priors, but edge labeling can be labor-intensive and fail in textured regions.Other prior assumptions, including differences in blur levels, may be weakly observed or violated in general scenes.Source: §2 · p. 2
- Multiview methods exploit motion cues between background and reflection layers, but their capture-facility requirements limit practical use, especially for mobile devices or images from the Internet.Source: §2 · p. 2
- Learning-based methods can better capture image properties, yet conventional two-stage frameworks ignore intrinsic correlations and degrade performance [7].Source: §2 · p. 2
3. Dataset Preparation
The authors introduce RID, a 3,250-image real-world reflection dataset, to address limited training data and the incomplete reflection properties represented by synthetic data. They generate training mixtures by combining RID reflections with natural-image backgrounds at varying weights and augmenting the results.
3.1. Real-world refection image dataset for data-driven methods
- RID contains 3,250 real-world reflection images, addressing the limited scale of existing data and the incomplete real-world properties of synthetic alternatives.The paper notes that the largest available dataset, SIR2, has approximately 500 image triplets, while synthetic data may omit ghosting and imaging-pipeline noise.Source: §3.1 · p. 3
3.2. Generating training data
- Training mixtures combine RID reflection images with natural-image backgrounds under the stated linear image-formation model.The model represents the mixture image as a linearly weighted additive combination of background and reflection.Source: §3.2 · pp. 3–4
- The generated training set contains 14,754 images after sampling mixture weights and applying rotation and flipping augmentation.The reflection coefficient β is sampled from 0.1 to 0.5, and the background coefficient α from 0.8 to 1.Source: §3.2 · p. 4
4. Proposed Method
CRRN jointly estimates the background, reflection, and background gradients, using multi-scale gradient guidance to support background recovery. It combines this network design with perceptually motivated losses and whole-image, multi-size training.
4.1. Network architecture
- CRRN concurrently estimates background B, reflection R, and background gradients ∇B, using the gradients to guide reflection removal.Its multi-task strategy addresses the correlation between estimating B and R.Source: §4.1 · p. 4
- GiN extracts background gradients across scales, while IiN uses VGG16 features to estimate B and R and shares convolutional layers with GiN.GiN links early encoder features to corresponding decoder layers to preserve sharp details; IiN uses multi-context features for background prediction.Source: §4.1 · pp. 4–5
4.2. Loss function
- IiN uses SSIM-based similarity losses for B and R, with L1 loss on B to balance SSIM-related brightness and color shifts; GiN omits SSIM’s luminance and contrast components.SSIM measures luminance, contrast, and structure, while the GiN loss omits luminance and contrast because those components become undefined there.Source: §4.2 · p. 5
4.3. Training strategy
- CRRN trains GiN independently before end-to-end fine-tuning and uses whole images at coarse 96 × 160 and fine 224 × 288 scales.The multi-size strategy is intended to make the network scale-invariant and avoid potential degradation from training on patches without obvious reflections.Source: §4.3 · p. 5
5. Experiments
On SIR2 wild images, CRRN leads four state-of-the-art methods across four metrics; qualitative and generalization comparisons and component ablations further assess its behavior.
5.1. Comparison with the state-of-the-arts
- Across four wild scenes, CRRN removes reflections more effectively and restores background details more clearly than the compared methods.The examples include close-ups and SSIM and SSIMr values.Source: Figure 4 · p. 6, §5.1 · p. 7
5. Experiments
- The evaluation uses SIR2 wild data; manually labeled reflection-dominant regions provide complementary SSIMr and SIr measures alongside whole-image SSIM and SI.The authors use regional metrics because global metrics may not fairly reflect reflection-removal performance.Source: §5 · pp. 6–7
5.1. Comparison with the state-of-the-arts
- On FY17 [7]’s released validation dataset, FY17 performs well with its trained model, while CRRN also achieves reasonably good results and performs even better in some regions.Source: Figure 5 · p. 7, §5.1 · p. 8
- CRRN consistently outperforms FY17 [7], NR17, WS16, and LB14 on all four error metrics, whose values are means over all 100 wild-image sets in SIR2.Source: §5.1 · p. 7
5.2. Network analysis
- CRRN’s embedded IiN and GiN outputs show fewer visible residual edges than outputs from the standalone branches.The comparison supports the authors’ claim that the subnetworks benefit each other in the estimation process.Source: Figure 6 · p. 8, §5.2 · p. 8
- None of the three variants—L1-only, IiN-only, or GiN-only—outperforms full CRRN using perceptually motivated losses.The variants test pixel-wise loss alone and each subnetwork without the other.Source: Table 2 · p. 8, §5.2 · p. 8
6. Conclusion
CRRN jointly estimates the background and reflection using image appearance and multi-scale features. It performs better than state-of-the-art methods in reported quantitative and visual comparisons and generalizes to unseen data, but performance may drop when reflections dominate the whole image.
6. Conclusion
- CRRN unifies gradient inference and image inference in a concurrent framework integrating image appearance and multi-scale low-level features.The network concurrently estimates the background and reflection under gradient guidance.Source: §6 · p. 8
- The method outperforms state-of-the-art methods in quantitative values and visual quality and is reported to generalize effectively to unseen data.The conclusion attributes these results to the collected real-world reflection dataset and its training strategy.Source: §6 · p. 8
- Performance may drop on whole-image-dominant reflections, where residual edges remain despite removing most reflections and restoring background details.The conclusion says CRRN still performs better than other state-of-the-art methods on these examples.Source: §6 · p. 8