Source-linked AI summary
Single Image Reflection Removal through Cascaded Refinement
Chao Li, Yixiao Yang, Kun He, Stephen Lin, John E. Hopcroft
TL;DR
Single-image reflection removal is an ill-posed problem complicated by limited data and varying imaging conditions. The paper proposes IBCLN, which iteratively refines transmission and reflection estimates using cascaded subnetworks, LSTM transfer, and residual reconstruction supervision, together with a densely labeled real-world dataset. Experiments report effective reflection removal on real and synthetic images and stronger overall performance across real-world test datasets.
Problem
Single-image reflection removal is an ill-posed task affected by limited training data, disparate imaging conditions, varying scene content, and limited physical understanding.
Method
IBCLN uses cascaded subnetworks to iteratively refine transmission and reflection estimates, transfers information with convolutional LSTM, and adds residual reconstruction loss and a real-world dataset.
Results
IBCLN achieves the best performance on four of five real-world datasets and surpasses other methods overall across the test datasets.
Takeaways & Limitations
The proposed cascaded refinement framework effectively removes reflections in real and synthetic images compared with state-of-the-art methods.
Takeaways & Limitations
The authors found that allowing the model to learn the total cascade steps automatically caused performance to decay.
Abstract
from arXiv · showhide
We address the problem of removing undesirable reflections from a single image captured through a glass surface, which is an ill-posed, challenging but practically important problem for photo enhancement. Inspired by iterative structure reduction for hidden community detection in social networks, we propose an Iterative Boost Convolutional LSTM Network (IBCLN) that enables cascaded prediction for reflection removal. IBCLN is a cascaded network that iteratively refines the estimates of transmission and reflection layers in a manner that they can boost the prediction quality to each other, and information across steps of the cascade is transferred using an LSTM. The intuition is that the transmission is the strong, dominant structure while the reflection is the weak, hidden structure. They are complementary to each other in a single image and thus a better estimate and reduction on one side from the original image leads to a more accurate estimate on the other side. To facilitate training over multiple cascade steps, we employ LSTM to address the vanishing gradient problem, and propose residual reconstruction loss as further training guidance. Besides, we create a dataset of real-world images with reflection and ground-truth transmission layers to mitigate the problem of insufficient data. Comprehensive experiments demonstrate that the proposed method can effectively remove reflections in real and synthetic images compared with state-of-the-art reflection removal methods.
1. Introduction
Single-image reflection removal is difficult because reflections vary across scenes and imaging conditions while labeled real-world data remains limited. The paper addresses these challenges with an iteratively refined cascaded network, LSTM-based information transfer, residual reconstruction supervision, and a densely labeled real-world dataset.
- Reflections degrade image quality and can affect downstream tasks such as object detection and semantic segmentation.
- Learning-based single-image methods remain limited by scarce training data, disparate imaging conditions, varying scene content, incomplete physical understanding, and model performance constraints.
- IBCLN introduces cascaded transmission and reflection decomposition that progressively refines both estimates across iterations.
- Convolutional LSTM transfers information between cascade steps and allows gradients to flow unchanged, addressing training difficulty in long cascades.
- The two subnetworks exchange prior-step outputs, while residual reconstruction loss provides supervision at each cascade step.
- The paper contributes a densely labeled real-world dataset collected across disparate imaging conditions and varying scenes, and reports state-of-the-art quantitative results on real-world benchmarks.
2. Related Work
Prior single-image reflection-removal work combines image-decomposition formulations, perceptual or adversarial objectives, and improved synthetic or real-world training data. These approaches seek better transmission estimates while addressing the mismatch between simplified models, limited labels, and real imaging conditions.
- SIRR models a captured image I as a linear combination of transmission T and reflection R, with the goal of inferring reflection-free transmission.
- Recent methods use perceptual losses to incorporate low-level and high-level image information through features extracted from ImageNet-pretrained networks.
- Adversarial losses have been applied to improve the realism of predicted transmission layers.
- Dataset-oriented methods improve synthesis with learned nonlinear alpha-blending masks or exploit misaligned real-world images to reduce the burden of acquiring densely labeled data.
3. Proposed Method
IBCLN decomposes a reflected image with two collaborating convolutional LSTM sub-networks that iteratively refine transmission and residual reflection estimates. Residual reconstruction and multi-scale losses provide supervision across cascade steps.
- Collaborative refinement: IBCLN uses separate transmission and reflection sub-networks whose outputs provide complementary auxiliary information to improve one another iteratively.Each sub-network receives outputs from both sub-networks, enabling collaborative cascaded refinement.
- Residual reconstruction: Residual reflection is computed from the input image and transmission estimate, avoiding the need to model the complicated physical reflection-formation process.The method uses eR = I − α · T instead of an artificially constructed reflection layer based on an element-wise blending mask.
- Network architecture: At each cascade step, predicted transmission and residual reflection are fed back as auxiliary inputs, with the final transmission estimate produced at the last time step.The outputs from step t−1 serve as inputs at step t, and the final transmission output is the model result.
- Collaborative refinement: The transmission and reflection branches are convolutional LSTM networks that preserve information across time steps and ease training for long cascades.The same sub-network parameters are reused across time steps, while LSTM units connect adjacent steps.
- Objective function: Residual reconstruction loss supervises both predicted outputs at every cascade step, while multi-scale perceptual loss captures low- and high-level information at different resolutions.The residual loss forms a reconstruction constraint from the estimated transmission and residual reflection; the multi-scale loss compares outputs across decoder scales.
4. Experiments
Experiments evaluate IBCLN on real-world reflection-removal datasets, including the newly collected Nature dataset, and through quantitative, qualitative, ablation, and time-step studies. IBCLN achieves the strongest overall real-world performance, while the complete architecture and objective function perform best in controlled comparisons.
- Dataset Preparation: The Nature dataset contains 220 real-world image pairs, with 200 training images and 20 images for quantitative evaluation.Images vary across environments, lighting, glass thickness, camera distance, viewing angle, exposure, and aperture.
- Quantitative Evaluations: IBCLN surpasses competing methods in overall performance across the real-world test datasets and achieves the best performance on four datasets.It does not achieve the best result on the 20-image Zhang et al. dataset, where ERRNet and Zhang et al. perform better.
- Qualitative Evaluations: On qualitative comparisons across Nature, SIR2, and Zhang et al., IBCLN more accurately removes most undesirable reflections than the compared methods.Zhang et al. tends to over-remove reflections, whereas other baseline methods tend to under-remove them.
- Controlled Experiments: Ablations show that the two iterative sub-networks, time steps, Ladv, Lresidual, and LMP each enhance IBCLN’s removal performance.The complete model, combining all structures and objective-function terms, yields the best results.
- Controlled Experiments: A total of N = 3 time steps gives the best performance, with outputs approximately converging at that setting.Automatically learning the total number of time steps reduced performance.
5. Conclusion
IBCLN removes reflections through cascaded mutual refinement of transmission and reflection layers, using LSTM-based information transfer and residual reconstruction guidance. Evaluations on five datasets show it outperforms state-of-the-art methods for single-image reflection removal.
- IBCLN iteratively refines transmission and reflection layers so each prediction boosts the other across cascade steps.The model uses complementary layer estimates to improve decomposition progressively.
- Convolutional LSTM transfers information across cascade steps and facilitates training over multiple iterations.LSTM addresses vanishing gradients by preserving information from previous iterations.
- Residual reconstruction loss provides additional training guidance at each cascade step, alongside a multi-scale perceptual loss.
- Evaluations on five datasets, including the authors’ dataset, show IBCLN outperforms state-of-the-art methods for single-image reflection removal.