Source-linked AI summary
Reflection-aware Generative Novel View Synthesis
GeonU Kim, Shin Dong-Yeon, Tae-Hyun Oh
TL;DR
Generative NVS methods often fail to recognize mirrors and exploit their reflected content, limiting reflection-consistent scene reconstruction. Ref-GeNVS estimates mirror planes, forms reflected virtual views, and uses two-stage attention and injection mechanisms without finetuning. It generates novel views that better align with mirror-reflected content and expose global scene structure, though accurate mirror-plane estimation remains difficult for very small or partially visible mirrors.
Problem
Existing generative NVS methods often treat mirror images as ordinary views and fail to exploit reflected content for scene generation.
Method
Ref-GeNVS estimates the mirror plane, reflects input camera poses to create virtual views, and applies Mirror-gated attention with Reflection injection in a two-stage generation process.
Results
Ref-GeNVS synthesizes novel views that better align with mirror-reflected content and expose global scene structure compared to prior approaches.
Takeaways & Limitations
The method uses partial mirror observations to support reflection-consistent scene generation and infer regions beyond an agent’s direct field of view.
Takeaways & Limitations
Accurate mirror-plane estimation is difficult when mirrors are very small, partially visible, or lack reliable geometric cues, which can misalign reflected content.
Abstract
from arXiv · showhide
We propose Ref-GeNVS, a training-free, reflection-aware method for generative novel view synthesis (NVS) in mirror scenes. Existing multi-view diffusion models often fail to recognize the mirror in the scene and cannot exploit reflected content for scene generation. To fix this issue without additional training, our key idea is to treat a mirror image as two complementary views. From input images, we estimate the mirror plane and reflect camera poses to form virtual views. Based on this virtual view setup, we propose a two-stage generation method consisting of Mirror-gated attention and Reflection injection, which enables reflection-consistent NVS by explicitly leveraging reflection relationships in a multi-view diffusion model. Ref-GeNVS inherits the strong generalizability of the multi-view diffusion backbone, while it does not require finetuning. On synthetic and real scenes including mirrors, Ref-GeNVS outperforms recent generative NVS methods by generating reflection-consistent and contextually coherent novel views, revealing scene structure visible only through mirrors. Project page: https://kim-geonu.github.io/Ref-GeNVS/
1 Introduction
Ref-GeNVS addresses the failure of generative NVS methods to use mirror cues by treating mirror images as complementary views and introducing a training-free reflection-aware pipeline.
- Motivation: Existing generative NVS methods typically treat a mirror image as one ordinary view, ignoring structural information in reflections.This can produce disagreement with true mirror content, baked mirror surfaces, and leakage between reflective and non-reflective regions.
- Approach: Ref-GeNVS treats a mirror image as two complementary views to generate the scene reflected on the mirror.The approach is designed as a training-free reflection-aware NVS method.
- Approach: Mirror-gated attention and Reflection injection explicitly leverage reflected cues during generation without model finetuning.These techniques target reflection-consistent novel-view synthesis.
2 Related Work
NeRF- and 3DGS-based NVS methods can achieve strong reconstruction quality but often misinterpret reflective objects because view-dependent appearances violate their assumed multi-view photometric consistency.
- Novel view synthesis: NeRF and 3D Gaussian Splatting are representative baselines for photorealistic novel-view synthesis from calibrated inputs.These approaches use implicit radiance fields or explicit point-based primitives for volume rendering.
- Reflective objects: Reflective objects are often misinterpreted because strong view-dependent appearances violate the multi-view photometric consistency assumed by these methods.
3 Method
Ref-GeNVS generates reflection-consistent novel views through a training-free, two-stage pipeline that treats mirror content as complementary virtual views. It masks mirrors and gates reflected evidence in Stage 1, then injects that evidence to complete mirror surfaces consistently in Stage 2.
- Ref-GeNVS aims to generate target views consistent with mirror-revealed scene content while keeping synthesized mirror surfaces globally consistent with the generated scene.
- Preprocess: Reflected virtual views are formed by reflecting input camera poses across the estimated mirror plane and horizontally flipping the image-mask pairs and corresponding poses.
- Naively adding reflected views creates mirror-recognition errors, non-mirror conditioning leakage, and inconsistency between generated scenes and mirror surfaces.
- Stage 1: Scene generation with Mirror-gated attention: Stage 1 masks mirror regions, adds reflected virtual views, and uses Mirror-gated attention so only mirror-region tokens influence target-view generation.
- Stage 2: Mirror surface generation with Reflection injection: Stage 2 completes masked mirror surfaces through Reflection injection, using reflected-view evidence and Stage 1 outputs to maintain reflection consistency.
- Stage 2: Mirror surface generation with Reflection injection: SDEdit-style guidance anchors non-mirror regions during denoising, stabilizing transitions and enabling natural compositing near mirror boundaries.
4 Experiments
Experiments evaluate Ref-GeNVS on synthetic and real mirror scenes under sparse-input and single-image settings, including reflection-only regions. Results show improved perceptual, pixel-level, qualitative, reconstruction, ablation, and distorted-reflection performance.
- Datasets and protocol: The evaluation uses eight synthetic scenes and the real Mirror-NeRF dataset, with trajectories containing regions visible only through reflection.Synthetic scenes use physically based mirror materials and calibrated camera parameters.
- Sparse-input novel view synthesis: Ref-GeNVS surpasses MVGenMaster and SEVA across DreamSim, CLIP similarity, PSNR, SSIM, and LPIPS in sparse-image NVS.The evaluation combines perceptual metrics with pixel-level measures of geometric consistency.
- Sparse-input novel view synthesis: Ref-GeNVS produces more consistent target views than prior methods, especially in regions visible only through mirrors.Competing methods often misinterpret or ignore reflected regions, producing structural inconsistencies and hallucinated content.
- Single-image novel view synthesis: In the single-image setting, Ref-GeNVS improves DreamSim and CLIP similarity against both the input view and the ground-truth reflected view.These comparisons assess global coherence with the input and semantic alignment with reflected content.
- Ablation study: Ablations show progressively better reflection consistency when adding reflected virtual views, Mirror-gated attention, and two-step generation to SEVA.Mirror-gated attention reduces leakage from non-mirror regions, while two-step generation further aligns the reflected scene with the mirror surface.
- Additional analyses: Ref-GeNVS yields more complete and stable 3D geometry and can handle distorted reflections after warping them into a planar-mirror representation.The distorted-reflection experiment uses a mirror ball without modifying the pipeline.
5 Conclusion
The paper concludes that Ref-GeNVS provides training-free, reflection-aware novel view synthesis by augmenting multi-view diffusion with reflected views. It reports better alignment with reflected content and exposure of global scene structure than prior approaches, while identifying trainable reflection-conditioned models as future work.
- Conclusion: Ref-GeNVS augments a multi-view diffusion backbone with reflected views to generate reflection-consistent novel views without training.The method estimates the mirror plane and reflects camera poses to create complementary viewpoints.
- Conclusion: The method better aligns generated views with mirror-reflected content and exposes global scene structure compared to prior approaches.The conclusion also identifies potential applications in 3D scene generation and embodied agents.
- Future work: Future work includes a trainable framework incorporating mirror detection and reflected-view conditioning during training.The paper notes that naïve training without explicit physical priors is not guaranteed to learn reflection constraints implicitly.
Supplementary Material
The supplementary-material passages identify the authors and their KAIST affiliation.
- Author information: The paper is authored by GeonU Kim, Shin Dong-Yeon, and Tae-Hyun Oh.The listed affiliation is KAIST.
A Implementation details
Mirror-gated attention converts reflected mirror masks into token-level attention gating and implements that gating through padded tensors compatible with FlashAttention.
- Mirror-gated attention: Mirror masks are downsampled and reshaped into token-level binary masks aligned with the attention feature layout.The mask is broadcast to match the transformer attention tensor layout.
- Padding-based masked attention: Padded query, key, and value tensors add a channel that encodes whether tokens lie inside or outside the mirror region.Inside-mirror tokens receive b = 0, while outside tokens receive b = −C, with C set to 104.
- Padding-based masked attention: The fused scaled dot-product attention operator computes the gated attention output using the FlashAttention backend.The resulting output has shape O ∈ R^B×H×N×D after removing the appended padding channel.
- Mirror-gated attention: Mirror-gated attention is applied across cross-view attention layers and reflected-view self-attention to restrict reflected evidence to mirror regions.Boundary pixels around the mirror are additionally inpainted to reduce artifacts from non-mirror regions.
A.2 Stage 2: Reflection injection
Stage 2 restores mirror appearance through guided inpainting while using scene content generated in Stage 1, with preprocessing designed to estimate mirror geometry from sparse inputs.
- A.2 Stage 2: Reflection injection: Stage 2 conditions on original and reflected inputs while excluding the Stage 1 mirror region as a source view.Only the non-mirror portion of the Stage 1 output guides diffusion through SDEdit-style conditioning.
- A.2 Stage 2: Reflection injection: A Gaussian-blurred soft mirror mask smoothly blends the generated mirror region with surrounding image content.The blurred mask is resized to latent resolution before blending.
- Mirror geometry estimation: Mirror masking makes per-pixel 3D regression produce a consistent planar mirror surface across Dust3R, Pi3, and DA3.Without masking, predictions may represent the mirror as either a flat plane or a hole in scene geometry.
- Mirror geometry estimation: Surface-based mirror-plane estimation fits RANSAC planes to mirror surface points and is used for sparse-view geometry estimation.In the pose-free case, predicted 3D points and camera poses provide the inputs for extracting mirror surface points.
- Mirror geometry estimation: For provided poses, Umeyama alignment resolves coordinate-system mismatch before estimating the mirror plane; single-image inputs are calibrated through generated image pairs.The single-image procedure aligns mirror-plane estimation with the generative backbone scale without external calibration.
A.4 Experimental details
Experiments evaluate Ref-GeNVS on real and synthetic mirror scenes using competing methods at their recommended image resolutions on one RTX 6000 Ada GPU.
- A.4 Experimental details: Experiments use the real-world Mirror-NeRF dataset and a synthetic dataset for evaluation.The Mirror-NeRF dataset contains three scenes.
- A.4 Experimental details: MVGenMaster-based methods operate at 512 × 512 resolution, while SEVA-based methods operate at 576 × 576 resolution.Each competing method uses its recommended input resolution.
- A.4 Experimental details: All experiments are conducted on a single NVIDIA RTX 6000 Ada Generation GPU.The hardware setup is shared across the reported experiments.
B Robustness on inaccurate mirror detection
The robustness study corrupts mirror regions before segmentation and compares edge-based with surface-based plane estimation as mask quality declines.
- Robustness evaluation: Random patch corruption before DAM segmentation progressively lowers detected-mask IoU to simulate realistic mirror detection errors.The degraded masks are propagated through preprocessing and both generation stages before evaluation.
- Robustness comparison: The comparison contrasts edge points without masking against masked surface points predicted by per-pixel 3D regression.Both strategies estimate the mirror plane from different geometric evidence.
- Robustness comparison: As mask IoU decreases, both plane-estimation methods degrade, but surface-based estimation remains significantly more robust than edge-based estimation.Figure S3 shows edge-based estimation producing incorrect planes and misaligned reflected content under inaccurate masks.
- Robustness comparison: At mask IoU around 0.6, surface-based estimation still outperforms SEVA.The result supports more stable mirror-plane estimation under imperfect mirror detection.
C Limitations
Ref-GeNVS is limited by mirror-plane estimation when reliable geometric cues are scarce, especially for small, partially visible, or heavily occluded mirrors.
- Mirror-plane estimation can fail when boundary correspondences are scarce or unstable, causing inaccurate reflected-view geometry.
- Small or partially visible mirrors can make point-cloud regression unreliable and produce outputs misaligned with ground truth.
- These geometric failures can misalign reflected content in final generation, despite robustness to realistic mirror-mask errors.
D Additional experimental results
Additional experiments show that Ref-GeNVS produces reflection-consistent sparse-view results, supports qualitative single-image evaluation, improves reflected-region and mirror-surface quality, and incurs higher inference time.
- Sparse images novel view synthesis: Ref-GeNVS generates reflected content consistent with scene structure visible in the mirror, unlike prior sparse-image novel view synthesis methods.The comparison covers both real and synthetic datasets.
- Single image novel view synthesis: Single-image experiments evaluate novel-view trajectories on real-world, MSD, and MMD datasets without ground-truth scene images behind the mirror.The target views use 360-degree orbit camera poses around the scene.
- Region-wise evaluation: Reflection and mirror masks are extracted as ground-truth binary masks from Blender for region-wise evaluation.The masks identify reflected scene regions and the mirror surface.
- Region-wise evaluation: Region-wise PSNR shows a larger performance gap in reflected regions than in the full scene, while mirror-surface generation also improves.Reflected-region PSNR can remain low because large target-to-reflected-view pose gaps create a wide-baseline setting.
- Inference time: 397 seconds per scene versus 130 seconds for SEVA reflects the computational trade-off of Ref-GeNVS’s preprocessing and two-step generation pipeline.Runtimes are measured on an NVIDIA RTX 6000 Ada Generation GPU.