Source-linked AI summary
Reflection Removal through Efficient Adaptation of Diffusion Transformers
Daniyar Zakarin, Thiemo Wandel, Anton Obukhov, Dengxin Dai
TL;DR
Single-image reflection removal is an ill-posed separation problem, and foundation-model adaptation depends on representative training data. The paper combines a pretrained DiT, efficient LoRA adaptation, and physically based synthetic glass data; WindowSeat reports strong benchmark and in-the-wild performance.
Problem
Single-image reflection removal is an ill-posed source-separation problem, and noisy or unrepresentative training data can limit adaptation of foundation models.
Source: §1 · pp. 1–2Method
The paper adapts a pretrained DiT with LoRA and uses a Blender PBR pipeline to synthesize glass reflections for training.
Source: §1 · p. 2, §3.2 · p. 3Results
WindowSeat achieves a 1.56 dB PSNR gain and raises SSIM from 0.930 to 0.940 on zero-shot SIR2 (500), with state-of-the-art performance on benchmarks and in-the-wild images.
Source: §4.2 · p. 6, §5 · p. 8Takeaways & Limitations
The authors conclude that physically grounded synthetic data and lightweight adaptation provide an efficient and scalable framework for reflection removal.
Source: §5 · p. 8WindowSeat removes secondary and higher-order reflections, which may conflict with ground truth or user intent; the authors suggest finer user control over how many reflection layers to remove.
Source: §4.4 · p. 8
Abstract
from arXiv · showhide
We introduce a diffusion-transformer (DiT) framework for single-image reflection removal that leverages the generalization strengths of foundation diffusion models in the restoration setting. Rather than relying on task-specific architectures, we repurpose a pre-trained DiT-based foundation model by conditioning it on reflection-contaminated inputs and guiding it toward clean transmission layers. We systematically analyze existing reflection removal data sources for diversity, scalability, and photorealism. To address the shortage of suitable data, we construct a physically based rendering (PBR) pipeline in Blender, built around the Principled BSDF, to synthesize realistic glass materials and reflection effects. Efficient LoRA-based adaptation of the foundation model, combined with the proposed synthetic data, achieves state-of-the-art performance on in-domain and zero-shot benchmarks. These results demonstrate that pretrained diffusion transformers, when paired with physically grounded data synthesis and efficient adaptation, offer a scalable and high-fidelity solution for reflection removal. Project page: https://hf.co/spaces/huawei-bayerlab/windowseat-reflection-removal-web
1. Introduction
Single-image reflection removal is an ill-posed separation problem that needs both scene understanding and knowledge of image-formation physics. The paper adapts a foundation DiT using diverse photorealistic synthetic data and an efficient training protocol.
Motivation
- Single-image reflection removal is an ill-posed source-separation problem requiring scene understanding and image-formation physics.Source: §1 · p. 1
Research gap
- Prior methods used limited-pretraining task-specific architectures, simplified reflection assumptions, or restricted data sources, contributing to sparse distribution coverage and poor in-the-wild and out-of-distribution performance.Source: §1 · pp. 1–2
Motivation
- Noisy or unrepresentative training data can limit foundation-model adaptation, whereas diverse photorealistic synthetic data provides coverage and controllability for exploiting pretrained priors.Source: §1 · p. 2
Contributions
- The paper combines a scalable PBR data pipeline, efficient foundation-DiT adaptation, and benchmark evaluation in WindowSeat.The protocol supports one-day training on a single consumer GPU and allows future work to swap in improved base models.Source: §1 · p. 2
2. Related Work
Prior reflection-removal work spans controlled-capture inputs, real and synthetic data, and diffusion-based methods. This paper extends physically based synthesis and adapts an image-editing diffusion model into a straightforward feed-forward approach.
Taxonomy of reflection removal
- Prior SIRR work uses controlled-capture side channels or real-image and video datasets, which can be limited in scene coverage and pixel alignment.Pseudo-ground-truth generation is easier to scale but offers limited control over glass properties and data distribution.Source: §2 · p. 2
Alpha blending
- Alpha-blended synthetic datasets are common, but their screen-space mixing models lack the physical realism needed to capture true glass behavior.Source: §2 · p. 2
PBR data and foundation models
- The paper extends PBR synthesis with fixed 3D scenes and varying input images, avoiding per-scene geometric reconstruction while preserving physically faithful light transport.It also describes lightweight LoRA and 4-bit QLoRA adaptation of modern flow-matching image-editing models.Source: §2 · p. 2
Foundation models and SIRR
- WindowSeat adapts an image-editing diffusion model into a feed-forward reflection-removal network, and the resulting model outperforms prior work across the board.Source: §2 · pp. 2–3
PBR data
- The PBR pipeline uses the Principled BSDF in Blender to simulate photorealistic glass effects; IoR changes reflection strength, thickness increases ghosting, and roughness controls scatter and blur.The caption states that screen-space alpha blending cannot faithfully reproduce this simulation.Source: Figure 2 · p. 3
3. Method
The method replaces simple alpha blending with physically based glass rendering, then adapts a pretrained DiT to remove reflections in one pass. Its data pipeline controls glass effects while preserving alignment between contaminated inputs and clean targets.
3.1. Recap: Alpha Blending
- Alpha blending linearly combines transmission and reflection images, but does not capture realistic glass phenomena such as subsurface scattering, internal-reflection ghosting, or high-intensity specular highlights.Source: §3.1 · p. 3
3.2. Physically Based Rendering Pipeline
- The PBR pipeline renders a transmission scene viewed through glass, using HDR environment maps or planar RGB images as reflection sources.HDR sources support light interactions across a wider spectrum, while RGB sources provide varied reflection patterns.Source: §3.2 · p. 3
- The pipeline uses textured mesh planes for efficient rendering and varies glass properties through the Principled BSDF, including reflection strength, blur, attenuation, and ghosting.Ground-truth images are rendered with IoR set to 1.0 and metallic and roughness terms zeroed; thickness and camera-glass spacing are controlled to preserve pixel alignment.Source: §3.2 · pp. 3–4
3.3. One-Step Flow Matching
- The frozen VAE encodes the contaminated input, and the DiT processes two latent token streams with a precomputed text embedding; the auxiliary stream duplicates the input latent.Source: §3.3 · p. 4
- The DiT-predicted latent velocity produces a reflection-free latent in one update, replacing multi-step diffusion sampling with a single forward pass.LoRA adapters are trained in pixel space using PSNR and SSIM losses between the output and clean target.Source: §3.3 · p. 4
- Predicting latent-space velocity rather than the absolute reflection-free latent yields consistently better inpainting in the reported ablation.Figure 4 compares PBR with alpha blending and latent with flow objectives; its caption says PBR data and the flow objective produce more accurate results.Source: Figure 4 · p. 4
4. Experiments
Experiments compare WindowSeat with prior methods on in-domain and zero-shot benchmarks and ablate its architecture, losses, and training data. The strongest reported gains are on SIR2 and Real, while the ablations identify choices that improve robustness and reconstruction.
4.1. Experimental Setup
- Evaluation uses Nature and Real as in-domain datasets and SIR2 as a zero-shot benchmark; the PBR training set contains 25,000 rendered images.The model has about 12.5B parameters; 3.6% are trained and 95.7% quantized to 4-bit, with a reported 21 GB peak GPU memory.Source: §4.1 · pp. 4–5, Table 3 · p. 5
4.2. Comparison with the State of the Art
- 1.56 dB PSNR gain and SSIM 0.930→0.940 on zero-shot SIR2 (500), plus 1.06 dB PSNR gain and SSIM 0.846→0.856 on Real.Table 2’s MS-SSIM and LPIPS comparisons also favor WindowSeat; Figure 5 shows fewer artifacts than competing methods.Source: Table 1 · p. 5, Table 2 · p. 5, §4.2 · p. 6
4.3. Ablation Study
- Using two latent inputs raises peak GPU memory from 19GB to 21GB but improves performance by almost 3 dB on Postcard; duplicating E(B) beats Gaussian noise overall.Gaussian noise slightly increases PSNR on Nature, while the duplicated RGB latent performs better overall.Source: §4.3 · p. 6
- Predicting latent-space velocity vθ rather than the absolute edited latent ẑedit consistently yields better inpainting.The reported example is shown in Figure 4 (right).Source: §4.3 · p. 6
- Using both PSNR and SSIM losses performs best; omitting PSNR is significantly worse in PSNR and SSIM, by up to 7 dB on Wild.Omitting SSIM still yields competitive results on Real, Nature, and Postcard.Source: §4.3 · p. 6, Table 5 · p. 6
- Training without alpha-blended data achieves the best SIR2 results, including a 1.36 dB gain on Postcard; the PBR-trained model removes a reflection that the alpha-blended model leaves.The comparison uses transmission and blended layers sampled from the same COCO dataset.Source: §4.3 · p. 7, Table 6 · p. 8
- PBR-trained models show improved robustness to stronger reflections across test sets rendered with progressively increasing glass IoR.The test set comprises 30 scenes rendered in five settings; Figure 6 compares PSNR, SSIM, MS-SSIM, and LPIPS for PBR- and alpha-blended-data models.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
4.4. In-the-wild Evaluations and Limitations
- WindowSeat removes secondary and higher-order reflections, which may conflict with ground truth or user intent; the authors suggest future control over how many layers are removed.Source: §4.4 · p. 8
5. Conclusion
WindowSeat combines a LoRA-fine-tuned foundation DiT with a high-fidelity PBR pipeline for efficient single-image reflection removal. The authors report state-of-the-art performance on established benchmarks and in-the-wild images, while noting that overshooting can reduce realism.
5. Conclusion
- WindowSeat generalizes across real-world reflection-removal scenarios, including intense reflections and wet or contaminated glass.The authors describe its handling of previously unseen conditions as robust.Source: Figure 7 · p. 8
- Removing secondary reflections can overshoot and reduce realism, as shown in the SIR2 Wild-Scene 010 example.The example's ground truth contains secondary reflections that WindowSeat removes.Source: Figure 8 · p. 8
- WindowSeat achieves state-of-the-art performance on established benchmarks and in-the-wild images using LoRA fine-tuning and a high-fidelity PBR data pipeline.The model can be trained on a single consumer GPU within one day.Source: §5 · p. 8