Source-linked AI summary
Polarized Reflection Removal with Perfect Alignment in the Wild
Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, Qifeng Chen
TL;DR
Existing reflection-removal methods face restrictive assumptions and real-world datasets with misaligned targets. This paper builds an aligned M-R dataset and proposes a two-stage polarized model with PNCC loss; the authors report that it outperforms state-of-the-art methods. They also identify a limitation of DoubleDIP in their setting.
Problem
Prior reflection-removal methods impose appearance assumptions, and available real-world datasets have misaligned transmission targets and limited glass diversity.
Source: §1 · p. 1Method
The paper collects raw mixed and reflection images to derive aligned transmission targets, then estimates reflection and transmission in two stages with PNCC loss.
Source: §1 · p. 2, §4.2 · p. 6Results
The authors report that their method outperforms all state-of-the-art methods and achieves better generalization.
Source: §1 · p. 2Takeaways & Limitations
The M-R dataset covers more than 100 glass types, and PNCC is also applicable to other image-decomposition tasks.
Source: §4.2 · p. 6, §6 · p. 8DoubleDIP fails in this setting; the authors suggest its spatial-invariant mixture-coefficient assumption may be broken by real-world data.
Source: §5.1 · p. 6
Abstract
from arXiv · showhide
We present a novel formulation to removing reflection from polarized images in the wild. We first identify the misalignment issues of existing reflection removal datasets where the collected reflection-free images are not perfectly aligned with input mixed images due to glass refraction. Then we build a new dataset with more than 100 types of glass in which obtained transmission images are perfectly aligned with input mixed images. Second, capitalizing on the special relationship between reflection and polarized light, we propose a polarized reflection removal model with a two-stage architecture. In addition, we design a novel perceptual NCC loss that can improve the performance of reflection removal and general image decomposition tasks. We conduct extensive experiments, and results suggest that our model outperforms state-of-the-art methods on reflection removal.
1. Introduction
The introduction motivates polarized reflection removal and identifies both restrictive assumptions in prior methods and misaligned, limited real-world training data. It presents a raw-data M-R collection pipeline and a two-stage polarized model with PNCC loss.
1. Introduction
- The paper distinguishes background B from transmission T: glass makes T darker and spatially shifted, while raw-space mixed image M is formed by adding reflection R to T.The figure describes recovering T by subtracting R from M in raw data space.Source: §1 · p. 1, Figure 1 · p. 1
- Prior methods often assume reflection is out of focus or contains ghost cues, limiting them when reflections are sharp or glass is thin.The introduction also notes that polarization-based approaches commonly assume unpolarized light sources, an assumption that can be violated in real-world scenes.Source: §1 · pp. 1–2
- Existing real-world datasets use background images as transmission targets, although refraction makes them misaligned, brighter, and potentially color-distorted relative to transmission through glass.Using only one glass type also limits generalization to other glass types.Source: §1 · p. 1
- The M-R pipeline captures raw mixed image M and reflection R, then derives T as M − R to collect aligned real-world data across diverse glass types.The authors state that capturing raw data preserves the linear formulation.Source: §1 · p. 2
- The proposed polarized reflection-removal model estimates reflection first and transmission second, using PNCC loss to reduce their similarity without assuming a particular reflection appearance.The paper also describes PNCC as applicable to other image-decomposition tasks.Source: §1 · p. 2
- Experiments are reported to show that the method outperforms state-of-the-art methods and generalizes better.The contribution summary also reports state-of-the-art performance across various metrics.Source: §1 · p. 2
2. Related Work
Prior reflection-removal work uses assumptions about image appearance, motion, or light sources, while real-world datasets and synthetic data each have stated limitations. The section also describes polarization-based and multi-image approaches.
2. Related Work
- Single-image methods rely on appearance assumptions such as defocused reflection, and methods built on these assumptions may fail on sharp reflections.Real-world datasets are described as misaligned and insufficiently diverse, while neural-network-synthesized data raise questions about physical authenticity.Source: §2 · p. 2
- Polarization has aided reflection removal for decades: Schechner and Bronstein use independent component analysis, Kong optimizes layer separation assuming unpolarized sources, and Wieschollek combines polarization with deep learning while retaining that assumption.Source: §2 · pp. 2–3
- Figure 3 contrasts subtraction after ISP, gamma-based subtraction, and raw-space subtraction; it states that non-raw subtraction produces undesirable residuals.The caption defines “Gamma M-R” as M^2.2 − R^2.2 and “Raw M-R” as subtraction on raw images.Source: Figure 3 · p. 3
- Multi-image methods exploit flash differences or relative motion, with cited approaches assuming particular motion, scene stability, or transmission dominance.The passage identifies the flash/noflash approach as and motion-based works as.Source: §2 · p. 3
3. M-R Dataset
The M-R pipeline collects mixed and reflection images, then uses raw-space subtraction to obtain aligned transmission ground truth. Its dataset supports diverse everyday glass, polarization measurements, and synthesis of realistic image mixtures.
3. M-R Dataset
- M-R collects mixed image M and reflection R, then derives transmission T = M − R from raw data, where sensor linearity supports the subtraction.The paper presents this as a physically faithful way to obtain paired data without capturing T directly.Source: §3 · pp. 3–4
- The polarization dataset records four polarizer angles in one shot and forms four 1024 × 1224 images from RAW sensor data.The angles are 0°, 45°, 90°, and 135°; RAW data preserves the linear-intensity conditions used by the polarization equations.Source: §3 · p. 3
- Unlike prior pipelines limited to removable, thin, non-colored glass, M-R can use diverse everyday glass, including curved, thick, or colored glass.The dataset is described as having perfect alignment and little intensity decay compared with prior datasets.Source: §3 · p. 4, Figure 4 · p. 4, Table 1 · p. 4
- Using transmission T rather than background B avoids treating them as identical: B can differ in color, intensity, and position, while M-R provides perfectly aligned pairs.Refraction shifts B relative to T, and glass can make T darker than B.Source: §3 · p. 4
- Raw-data linearity also supports simulating realistic mixtures as M = a * R + b * T, with coefficients a and b varying from 0 to 1.The simulation uses unpaired R and T images.Source: §3 · p. 4
- Dataset cleaning discards R–T pairs with mean intensity ratios above 10 or below 0.1, sets subtraction negatives to zero, and crops multi-layer glass regions.Overexposed areas are also masked using a threshold δ = 0.98.Source: §3 · pp. 4–5
4. Method
The framework uses polarization differences to separate reflection from transmission in two stages. Its PNCC loss compares normalized feature maps, while the implementation augments the network input with VGG-19 features.
4.1. Reflection-Based Framework
- Polarization differs between reflection and transmission, and the model first estimates reflection before recovering transmission with that estimate.The two-stage design uses separate decoders to learn features specific to each layer; without polarization, the authors note this design may deteriorate performance.Source: §4.1 · p. 5
4.2. Loss function
- PNCC compares normalized VGG-19 feature maps to minimize correlation between estimated reflection and transmission.The loss uses conv2_2, conv3_2, and conv4_2; after normalization, it decreases as synthetic input pairs become more mixed, including cases with large reflection–transmission intensity differences.Source: §4.2 · pp. 5–6
4.3. Implementation
- The U-Net inputs are augmented with VGG-19 hypercolumn features extracted from four polarization-angle images and intensity I.The extracted conv1_2 features are bilinearly upsampled to the input resolution; RAW inputs receive gamma correction before entering the network.Source: §4.1 · p. 5, §4.3 · p. 6
5. Experiments
Experiments on the M-R dataset compare the proposed method with reflection-removal baselines and test its qualitative behavior and design components. The evaluations report strong benchmark performance and gains from polarization, the two-stage design, and PNCC.
5.1. Experimental procedure
- Experiments center on M-R, the only available raw-image dataset, using 100 validation pairs and 107 testing pairs.Zhang et al. [34] and Wei et al. [28] are retrained; several other baselines use pretrained models or third-party implementations. DoubleDIP [9] fails in this setting, possibly because real-world data violate its spatial-invariant coefficient assumption.Source: §5.1 · pp. 6–7
5.2. Comparisons with baselines
- Our method achieves the best PSNR and SSIM on M-R, with Zhang et al. [34] and Wei et al. [28] closest overall.Traditional polarization-based methods rank lower, which the authors attribute to their unpolarized-light-source assumption.Source: Table 2 · p. 7, §5.2 · p. 7
- Qualitative cases show the method handles varied reflections without artifacts and performs on curved glass, non-ideal data, and blurry mixtures where blur-assumption baselines over-remove content.Zhang et al. [34] and Wei et al. [28] leave visible residual reflection in the compared examples.Source: Figure 8 · p. 7, Figure 9 · p. 7, §5.2 · p. 8
5.3. Ablation study
- Ablations show polarization contributes most, the two-stage design boosts performance, and PNCC adds further gains; without polarization, the model can confuse transmission with reflection.The study replaces polarization channels with a single image input, removes the reflection loss for the two-stage test, and evaluates a setting without PNCC.Source: §5.3 · p. 8, Table 3 · p. 8, Figure 10 · p. 8
- In DoubleDIP [9], PNCC outperforms the exclusion loss both in the official implementation and after its hyperparameters are tuned.The comparison evaluates the two losses within the DoubleDIP image-decomposition framework.Source: §5.3 · p. 8, Table 4 · p. 8
6. Discussion
The discussion presents a two-stage polarized reflection-removal model and an aligned real-world dataset spanning more than 100 glass types. It also proposes PNCC as a general decomposition loss and reports experiments demonstrating the model's effectiveness.
6. Discussion
- The proposed two-stage model targets polarized reflection removal with perfectly aligned input-output image pairs, and the M-R formulation is used to bypass background-to-mixed-image misalignment.The resulting real-world dataset covers more than 100 types of glass.Source: §6 · p. 8
- PNCC is proposed as a general decomposition loss that minimizes correlation between two images at different feature levels.Source: §6 · p. 8
- The authors report thorough experiments demonstrating the effectiveness of their model and suggest that the model formulation and M-R dataset may inspire future reflection-removal research.Source: §6 · p. 8