Source-linked AI summary
From Synthesis to Removal: Physics-Grounded Reflection Simulation and Diffusion-Based Video Dereflection
Zepeng Wang, Jiagao Hu, Fuhao Li, Yuxuan Chen, Fei Wang, Daiguo Zhou
TL;DR
Video reflection removal lacks paired video data, temporally coherent models, and dedicated benchmarks, despite reflections degrading visual quality and downstream vision. The paper introduces a physics-grounded synthesis pipeline, a diffusion-based one-step remover, and a benchmark, reporting state-of-the-art performance and validated synthesis effectiveness.
Problem
Video reflection removal remains underexplored because paired data, temporally coherent models, and task-specific benchmarks are lacking, while reflections degrade visual quality and downstream vision.
Method
The framework synthesizes paired videos with structure-space physics-grounded augmentation, adapts a pretrained video diffusion prior for one-step removal, and evaluates it with S2R-Bench.
Results
The framework achieves state-of-the-art performance on S2R-Bench and multiple public image reflection-removal benchmarks, while data ablations validate S2R-Synthesis.
Takeaways & Limitations
The framework provides a foundation for training and evaluating video dereflection methods through paired synthesis, reflection-aware removal, and benchmark evaluation.
Takeaways & Limitations
Nested reflections from multiple glass layers can remain residual or be over-removed because the restoration target is ambiguous.
Abstract
from arXiv · showhide
Videos captured through glass often contain reflections that degrade visual quality and interfere with downstream vision tasks. Although single-image reflection removal has been extensively studied, video reflection removal remains largely underexplored due to the lack of paired video data, temporally coherent removal models, and dedicated evaluation benchmarks. We present a closed-loop framework that unifies physics-grounded reflection simulation, diffusion-based video dereflection, and benchmark evaluation. Our S2R-Synthesis pipeline generates paired reflected and reflection-free videos by performing physics-grounded augmentation in the structure space and rendering realistic reflected videos with a trained video diffusion renderer; the augmentation models key glass-related effects including roughness-induced blur, thickness-induced ghosting, and reflectance variation. Based on the synthesized data, we introduce S2R-Removal, the first diffusion-based video reflection removal model, which adapts a pretrained video diffusion prior through reflection-aware latent adaptation and one-step pixel-geometric refinement, recovering the clean transmission in a single denoising step. We further build S2R-Bench, the first benchmark for video reflection removal, supporting both full-reference evaluation and real-world human perceptual assessment. Experiments on S2R-Bench and multiple public image benchmarks demonstrate state-of-the-art performance and faster inference than even non-diffusion baselines, and validate the effectiveness of S2R-Synthesis. Project page: https://codingwzp.github.io/VideoDereflection_S2R.
1 Introduction
Video reflection removal is limited by the temporal behavior of reflections, scarce paired video data, and missing coherent models and benchmarks. The paper addresses these gaps with a closed-loop synthesis, removal, and evaluation framework.
- Reflections vary over time, move independently from transmission, and interact with camera motion, making video removal more difficult than single-image removal.
- Frame-by-frame image methods often cause flickering, inconsistent removal strength, and unstable background details because they lack temporal modeling.
- Paired reflected and reflection-free videos are difficult to obtain, while existing synthesis methods lack scalable, realistic, and controllable glass effects.
- The framework integrates physics-grounded synthesis, diffusion-based removal, and benchmark evaluation in a closed-loop video dereflection pipeline.
- S2R-Synthesis generates realistic, controllable reflected videos using structure-space augmentation and a trained video renderer.
- S2R-Removal adapts a video diffusion prior with reflection-aware latent training and pixel-geometric refinement to recover clean transmission in one denoising step.
- S2R-Bench supports full-reference objective evaluation and real-world perceptual assessment, while experiments report state-of-the-art performance and validate the synthesis pipeline.
2 Related Work
Prior reflection-removal research has focused mainly on independently processed images, while video-specific synthesis, temporal modeling, and benchmarks remain limited. S2R-Synthesis and S2R-Bench target these gaps.
- Single-image reflection removal has advanced through priors, multi-image constraints, deep layer estimation, perceptual supervision, and robustness-oriented designs.
- Applying image methods independently to video produces flickering, and video reflection removal has received comparatively little attention.
- Existing image datasets use controlled capture, real-world collection, or synthetic composition, but paired video data additionally require temporal coherence and aligned ground truth.
- S2R-Bench fills the missing benchmark gap with paired videos and controllable synthesis for video reflection removal.
- Prior diffusion-based dereflection methods remain confined to images, whereas S2R-Removal extends diffusion-based dereflection to video.
3 Physics-Grounded Reflection Simulation
S2R-Synthesis replaces simplistic RGB blending with structure-space conditioning and physics-grounded augmentation. A learned video renderer produces realistic paired reflected and clean videos under controllable glass effects.
- The standard reflection model represents an observed image as a linear mixture of transmission and reflection layers.
- Direct RGB blending misses glass-dependent effects including roughness-induced blur, thickness-induced ghosting, and reflectance variation.
- The two-stage pipeline trains a structure-guided renderer and combines it with Physics-Grounded Augmentation to synthesize large-scale paired data.
- The renderer uses clean transmission video and composed lineart structure to produce photorealistic reflection videos.
- PGA applies clip-level glass parameters shared across frames to model controllable roughness, thickness, viewing, coverage, and temporal effects.
- PGA primitives include roughness blur, thickness ghosting, static reflections, and planar fusion, with operations tied to optical properties.
4 Diffusion-Based Video Dereflection
The proposed dereflection model treats removal as conditional video restoration: reflected video conditions a pretrained video diffusion backbone to predict clean transmission. The supplied passages identify the reflection-aware latent adaptation stage.
- Given paired videos, the model predicts clean transmission video from reflected input using conditional video generation.
- The adaptation is built on a Wan2.1 video-inpainting paradigm and is designed to localize and remove reflections.
- 4.1 Stage I: Reflection-Aware Latent Adaptation: Stage I adapts the pretrained video diffusion backbone in latent space with LoRA fine-tuning, noisy clean targets, and reflected-video conditioning.
Stage II: One-Step Pixel-Geometric Refinementℒ���
Stage II refines the rough clean estimate from Stage I with one deterministic denoising step, anchoring the decoded output to the clean target through pixel and geometric constraints.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: Stage I provides a rough but coherent clean estimate in a single forward pass, enabling efficient Stage II refinement.This capability emerges from latent diffusion training.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: Stage II uses a noisy latent at the largest noise level and performs one deterministic denoising update conditioned on the reflected input latent.The predicted clean latent is decoded into the pixel-space output.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: Pixel-space reconstruction and SSIM losses constrain color fidelity and local structural consistency.These losses anchor the decoded output to the clean target in image space.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: A frozen LeReS estimator supplies depth consistency supervision to constrain scene geometry without overpenalizing style or texture.The depth target is fixed, and gradients propagate only through the predicted depth.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: Geometric supervision is focused on reflection-corrupted regions identified by a binary residual mask with threshold 12.The focus targets regions where hallucination and structural distortion are most likely.
- Stage II: One-Step Pixel-Geometric Refinementℒ���: The two stages are complementary: Stage I teaches what to remove, while Stage II teaches what to preserve through pixel and geometric constraints.The one-step capability from Stage I makes this refinement efficient.
5 Experiments
Experiments evaluate S2R-Removal and its synthesis pipeline on paired video, real-world video, and image benchmarks, including qualitative, quantitative, and ablation studies. Results show strong video performance, cross-domain image performance, and benefits from physically grounded synthesis and staged refinement.
- 5.1 Experimental Setup: S2R-Bench evaluates paired videos with PSNR, SSIM, and temporal consistency, while real videos receive human scores for removal quality and transmission preservation.S2R-Ref contains 60 paired videos, and S2R-Real contains 50 in-the-wild videos evaluated by 15 participants.
- 5.2 Comparison with State-of-the-Art: S2R-Removal achieves the best PSNR and temporal consistency on S2R-Ref and the highest human scores on S2R-Real for both evaluated aspects.It also provides competitive or superior results on the listed image benchmarks.
- 5.2 Comparison with State-of-the-Art: 87.09 ms/frame makes S2R-Removal approximately 1.67× faster than the referenced comparison while retaining strong dereflection performance.The reported efficiency advantage is attributed to the one-step design.
- 5.2 Comparison with State-of-the-Art: Qualitatively, S2R-Removal suppresses reflections while preserving transmission tone and structure, whereas image-based methods leave residues, flicker, or shift scene appearance.The controlled comparisons use clean ground truth, and video comparisons inspect the 1st, 20th, and 40th frames.
- 5.3 Effectiveness of S2R-Synthesis: 34.13 dB PSNR and 0.9704 SSIM are achieved with full PGA, improving over planar-only synthesis by +1.78 dB and +0.0061 SSIM.The data-generation ablation progressively adds augmentation types and evaluates RDNet on OpenRR-1k.
- 5.4 Ablation Study: 28.84 dB PSNR and 0.8594 SSIM are obtained by the full two-stage removal objective, after intensity supervision and pixel-geometric refinement improve the diffusion-only baseline.Intensity supervision alone raises PSNR from 27.00 dB to 27.71 dB and SSIM from 0.8348 to 0.8430.
6 Conclusion
The paper concludes with a closed-loop framework combining paired reflection synthesis, diffusion-based dereflection, and benchmark evaluation. Its data pipeline uses structure-space physics-grounded controls and learned rendering, while the removal model uses reflection-aware adaptation and geometric refinement.
- 6 Conclusion: The framework combines physics-grounded paired video synthesis, reflection-aware diffusion dereflection, and benchmark evaluation into a closed loop.The authors position these components as a foundation for training and evaluating video dereflection methods.
- 6 Conclusion: The appendix documents training data, PGA controls, benchmark construction, qualitative comparisons, component visualizations, single-image application, and downstream driving-task evaluation.It also identifies method limitations concerning multi-layer reflections and coupled camera-motion geometry.
- 6 Conclusion: S2R-Bench supports full-reference and human perceptual evaluation through paired S2R-Ref videos and in-the-wild S2R-Real videos.The benchmark is designed to assess reconstruction fidelity, temporal consistency, removal quality, and transmission preservation.
- 6 Conclusion: S2R-Synthesis generates paired data by filtering reflection-free videos, applying reflection synthesis, and incorporating supplementary image-derived video clips.The initial Ditto-1M stage yields 11,132 paired clips, while additional synthesis uses HumanVid, UltraVideo, and traditional synthesis.
- 6 Conclusion: The removal training set contains 38,826 image and video samples, including 25,799 physics-grounded synthesized videos and 2,997 traditional-blending videos.Virtual camera motions convert 80% of image samples into video sequences during training.
- 6 Conclusion: PGA models glass-related effects through controllable operations tied to optical properties, including roughness, thickness, reflectance, and structure-space composition.The pipeline extracts transmission and reflection lineart, augments the reflection condition, and fuses it with the transmission condition.
C.1 Per-Primitive Physical Derivation
S2R-Synthesis models reflection effects through structure-space augmentations, including roughness blur, thickness-induced offsets, reflectance changes, partial coverage, and static temporal behavior. These controls reproduce physically expected trends while preserving clip-level temporal coherence and support benchmark construction.
- Physical augmentations: Roughness augmentation applies channel-wise Gaussian convolution to reflection lineart, modeling broader blur from rough glass surfaces.The blur width grows with roughness under the stated microfacet approximation.
- Physical augmentations: Thickness augmentation spatially translates reflection lineart to simulate lateral offsets between reflections from multiple glass interfaces.The displacement is motivated by thickness-dependent refraction under the paraxial approximation.
- Physical augmentations: Reflectance augmentation uses element-wise intensity modulation to represent weak or strong specular reflections and ambient brightness bias.The parameters γ and β control reflection strength and brightness offset, respectively.
- Additional controls: Partial-coverage augmentation masks reflection lineart with randomly generated rectangular regions, while static augmentation repeats one reflection frame across the clip.These operations model spatially varying reflectance and camera-relative stationary reflections.
- Composition and validation: Augmented reflection and transmission lineart are summed into a clipped structural condition, with parameters shared across frames to ensure temporal coherence.The synthesis comparison varies roughness, reflectance, and thickness under controlled conditions.
- Composition and validation: S2R-Synthesis reproduces increasing blur, overlay strength, and ghosting offset as roughness, reflectance, and thickness increase, without a heavyweight renderer.S2R-Bench combines full-reference S2R-Ref evaluation with no-reference S2R-Real human perceptual evaluation.
E Additional Qualitative Comparisons
Additional comparisons show S2R-Removal’s qualitative behavior across real-world videos, prior dereflection methods, general-purpose editing, attention visualizations, stages, and single-image inference. The comparisons emphasize targeted reflection suppression, transmission preservation, and refinement benefits.
- Additional Qualitative Comparisons: S2R-Removal more completely suppresses strong reflections than image-based methods while preserving the original scene appearance across real-world videos.Image-based methods often leave residues or produce temporally inconsistent removal.
- Additional Qualitative Comparisons: S2R-Removal visibly outperforms prior video dereflection methods in reflection suppression and transmission preservation, especially for strong reflections.Quantitative comparison is unavailable because the prior methods release no code.
- Additional Qualitative Comparisons: Compared with Kling O1, S2R-Removal performs more targeted dereflection and better preserves the original video content.The general-purpose editing model may alter global appearance, scene content, or background structure.
- Attention and staged refinement: Residual-derived intensity supervision makes DiT responses more reflection-aware and improves alignment with reflection-corrupted regions.The predicted intensity map captures reflection location and intensity-related structure.
- Attention and staged refinement: Stage II’s one-step pixel-geometric refinement recovers finer appearance details and better preserves scene geometry after Stage I suppression.Stage I may introduce appearance shifts or structural inconsistencies; corresponding gains are reported in Table 3.
- Single-image evaluation: Zoom-based video inference consistently outperforms direct single-image inference on the Real, SIR2, and Nature benchmarks, with zoom-out performing best.The model converts an image into a smooth pseudo-video clip before selecting an output frame.
- Single-image evaluation: Despite being trained for video dereflection, S2R-Removal achieves more effective image reflection removal while better preserving transmission content on public image benchmarks.This qualitative result is consistent with the cross-method quantitative results in Table 1.
H Downstream Benefits of Dereflection
Dereflection improves downstream perception on reflection-contaminated BDD100K videos. Both driving-area segmentation and vehicle detection show higher reported metrics after applying the model.
- Downstream Benefits of Dereflection: Driving-area segmentation improves after dereflection, with YOLOP IoU increasing from 0.823 to 0.835 and mIoU from 0.891 to 0.898.Qualitative examples show more complete and accurate segmentation of the drivable area.
- Downstream Benefits of Dereflection: The downstream qualitative examples associate dereflection with improved segmentation and detection performance.Figure 13 presents representative examples for both tasks.
- Downstream Benefits of Dereflection: Vehicle detection improves after dereflection, with YOLOv5n mAP@0.5 increasing from 0.351 to 0.372 and mAP@0.95 from 0.221 to 0.238.The reported qualitative results indicate fewer reflection-induced missed detections and improved vehicle localization.
I Limitations
The framework remains limited by ambiguous nested reflections and incomplete modeling of camera-motion-dependent reflection geometry. These constraints affect restoration targets and physical realism in complex glass scenes.
- I Limitations: Nested reflections from multiple glass layers may remain as inner-layer residuals or be over-removed because the restoration target is ambiguous.The authors note that this issue is also common to existing dereflection methods.
- I Limitations: The synthesis pipeline uses temporally coherent clip-level controls but does not explicitly model viewpoint-dependent reflection parallax coupled to camera motion.Future work proposes richer physical simulation and larger real-world video benchmarks.