PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRFReN: Neural Radiance Fields with Reflections

Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, Song-Hai Zhang

arXiv:2111.15234v2cs.CVcs.GR

TL;DR

Complex glass and mirror reflections exceed NeRF’s ability to represent view-dependent effects, causing inaccurate depth and rendering. NeRFReN separates transmitted and reflected components into distinct radiance fields, guides their under-constrained decomposition with geometric priors and training strategies, and achieves high-quality synthesis with physically sound depth while supporting interactive handling of difficult mirror cases.

  • Problem

    NeRF handles simple view-dependent effects such as highlights but struggles with complex reflections, producing inaccurate depth and renderings when reflected content violates multi-view consistency.

  • Method

    NeRFReN uses separate transmitted and reflected neural radiance fields, combines their images with a learned reflection fraction, and guides decomposition with architectural choices, geometric priors, and training strategies.

  • Results

    NeRFReN achieves high-quality novel view synthesis and physically sound decomposition and depth estimation results across real-world scenes, including challenging mirror cases.

  • Takeaways & Limitations

    The image-domain combination of transmitted and reflected components relaxes NeRF’s multi-view consistency restriction and supports depth estimation and scene editing applications.

  • Takeaways & Limitations

    The method is limited for curved or multiple non-coplanar reflective surfaces and ignores view-dependent effects of reflected objects and Fresnel-dependent reflection fractions.

Abstract

from arXiv · show

Neural Radiance Fields (NeRF) has achieved unprecedented view synthesis quality using coordinate-based neural scene representations. However, NeRF's view dependency can only handle simple reflections like highlights but cannot deal with complex reflections such as those from glass and mirrors. In these scenarios, NeRF models the virtual image as real geometries which leads to inaccurate depth estimation, and produces blurry renderings when the multi-view consistency is violated as the reflected objects may only be seen under some of the viewpoints. To overcome these issues, we introduce NeRFReN, which is built upon NeRF to model scenes with reflections. Specifically, we propose to split a scene into transmitted and reflected components, and model the two components with separate neural radiance fields. Considering that this decomposition is highly under-constrained, we exploit geometric priors and apply carefully-designed training strategies to achieve reasonable decomposition results. Experiments on various self-captured scenes show that our method achieves high-quality novel view synthesis and physically sound depth estimation results while enabling scene editing applications.

1. Introduction

NeRFReN addresses NeRF’s failures on complex reflections by separating transmitted and reflected scene components and combining their rendered images. Geometric priors and training strategies support decomposition, while interactive reflection masks handle especially challenging cases such as mirrors.

  • Motivation: NeRF handles highlights but struggles with severe glass and mirror reflections, producing inaccurate geometry and view synthesis when multi-view consistency fails.Reflected structures may be modeled at virtual depths, while reflected objects visible from only some viewpoints can cause inaccurate renderings.
  • Method: NeRFReN models transmitted and reflected scene components with separate neural radiance fields and composes their rendered images using a learned reflection fraction.The image formulation is I = I_t + β I_r.
  • Method: Because component decomposition is highly under-constrained, NeRFReN uses assumptions about reflection fractions, transmitted depth smoothness, and reflected geometry simplicity.These assumptions address degenerate solutions in which one component explains the scene, both explain it, or geometry is mixed between components.
  • Results: The method combines specialized network architecture, geometric priors, and warm-up training to obtain reasonable transmitted/reflected decompositions and competitive novel view synthesis.The approach is reported to achieve physically sound decomposition results on several real-world scenes, including challenging mirror cases.
  • Interactive setting: For especially ambiguous scenes such as mirrors, a small number of user-provided reflection masks can produce the correct decomposition.The interactive setting supplements the unsupervised method when its assumptions cannot resolve the ambiguities.
  • Applications: The work also investigates depth estimation and scene editing using the decomposed transmitted and reflected components.These applications extend the representation beyond novel view synthesis.

2. Related Work

Prior work addresses novel view synthesis and reflections through image-based, neural, and rendering-based representations. NeRFReN extends NeRF with separate transmitted and reflected fields for complex reflections, rather than relying on assumptions suited to simpler inverse-rendering settings.

  • Neural scene representations: Image-based rendering commonly uses mesh models reconstructed from captured images to support novel view synthesis.Such approaches represent scene geometry explicitly through reconstructed meshes.
  • Neural scene representations: Coordinate-based neural representations, including NeRF, model scenes as continuous volumetric fields and produce strong results in casually captured scenes.NeRF also handles view-dependent effects such as highlights by conditioning radiance on viewing direction.
  • Reflections in rendering: Rendering systems such as Screen Space Reflection simulate reflections at low cost, while novel-view synthesis with reflections remains challenging.Prior reflection-focused work decomposed images into transmitted and reflected layers with binary masks and piece-wise planar geometry.
  • Reflections in rendering: NeRFReN adopts an image formulation related to prior layer decomposition but uses two neural radiance fields to model transmitted and reflected components.This design targets complex reflections while retaining NeRF’s strengths for novel view synthesis.
  • Reflections in rendering: Unlike NeRF-based inverse-rendering methods built on opaque surfaces, simple BRDFs, and specific lighting models, NeRFReN assumes additive transmitted and reflected composition.The differing assumptions are intended for scenes containing complex reflections.

3. Method

NeRFReN separates transmitted and reflected scene components into distinct radiance fields, then uses geometric priors and warm-up training to make the under-constrained decomposition physically plausible. Its formulation supports reflection-aware rendering and user-supervised handling of difficult reflective scenes.

  • Scene formulation: NeRFReN models transmitted and reflected scene components with separate neural radiance fields and combines their rendered colors additively.A learned reflection fraction attenuates the reflected field before composition.
  • Network architecture: The reflection fraction is predicted from the transmitted branch because it represents the reflective surface’s material property.The transmitted color remains view-direction conditioned for low-frequency effects such as highlights, while the reflected field models stable virtual images.
  • Decomposition challenge: The decomposition is under-constrained because infinitely many component assignments can reproduce training images, including empty, duplicated, or mixed fields.These ambiguities can produce visually plausible renderings while failing to separate reflected radiance from transmitted radiance.
  • Geometric priors: NeRFReN uses an edge-preserving depth-smoothness prior for transmitted geometry and bidirectional depth consistency for reflected geometry.The smoothness term is relaxed across color gradients, while bidirectional consistency favors simple shell-like reflected geometry.
  • Warm-up training: A warm-up strategy gradually increases and then decreases geometric-constraint weights to stabilize training without scene-specific hyperparameter tuning.The method also masks viewing direction early to reduce leakage of reflected content into the transmitted component.
  • Interactive setting: Texture-less reflectors such as mirrors can defeat unsupervised decomposition, but manually labeled reflection-fraction maps enable successful isolation of reflected components in challenging cases.The additional supervision is applied to the estimated reflection fraction and succeeds in several scenarios that existing novel-view-synthesis techniques find difficult.

4. Experiments

On the RFFR scenes, NeRFReN delivers competitive view synthesis with more physically correct depth, while geometric priors and design choices support decomposition and reflection editing.

  • Setup: NeRFReN uses separate transmitted and reflected neural radiance fields as the experimental baseline against NeRF and NeRF-D.The evaluation introduces NeRF-D as NeRF augmented with the proposed depth smoothness constraint.
  • Setup: The RFFR dataset contains six forward-facing scenes with strong glass- and mirror-induced reflections, evaluated on held-out test images.The experiments report qualitative and quantitative results on the test split.
  • Decomposition: NeRFReN produces human-consistent decompositions and high-quality view synthesis, with reflection fraction maps distinguishing reflectors from nonreflectors.Large reflection-fraction values indicate reflectors, while small values indicate nonreflectors; low-frequency highlights remain in the transmitted field.
  • Comparisons: NeRFReN significantly outperforms NeRF in PSNR on art1 and mirror, while remaining slightly better on the other scenes.Table 1 reports comparable performance to NeRF and better performance than NeRF-D across PSNR, SSIM, and LPIPS.
  • Comparisons: Compared with NeRF and NerfingMVS, NeRFReN provides physically correct depth in severe-reflection scenes where mixed geometry or stereo initialization fails.NeRF-D improves depth but produces blurry renderings, while NerfingMVS struggles on complex reflections.
  • Ablations: Both geometric priors are essential for reasonable decomposition, and transmitted-field view dependency prevents highlights from creating erroneous reflected geometry.Removing the priors mixes components, while removing view dependency causes inaccurate depth and redundancies in the reflected component.
  • Applications: NeRFReN supports reflection removal and image-domain reflection substitution while maintaining view consistency.Reflections can be replaced with renderings from another NeRF or another scene representation such as a mesh.

5. Limitations and Conclusion

NeRFReN has scope limitations for reflective geometry and view-dependent effects, but concludes with high-quality synthesis, physically sound decomposition, and editing-related applications.

  • Limitations: NeRFReN is limited in modeling curved reflective surfaces and multiple non-coplanar reflective surfaces.These are identified as major limitations of the method.
  • Limitations: The method does not model view-dependent effects of reflected objects or Fresnel-induced view dependence in reflection fractions.These omissions constrain its treatment of reflection behavior.
  • Conclusion: The paper concludes that NeRFReN combines transmitted and reflected components to support high-quality view synthesis and physically sound decomposition.It also investigates depth estimation, reflection removal, and reflection substitution as potential applications.

Supplementary Material

The supplementary material adds implementation details, limitation and failure-case discussions, LLFF results, ablations, qualitative demonstrations, and video results.

  • Supplementary contents: The supplement covers network and training details, additional limitations and failure cases, LLFF comparisons, further ablations, qualitative demonstrations, and project-webpage videos.These materials are organized across supplementary Sections A.1–A.4 plus video results.

A.1.1. Network Configurations

The NeRFReN network uses fully connected layers with positional encodings for coordinates and viewing directions, while keeping parameter count approximately comparable to NeRF.

  • Architecture: NeRFReN’s detailed architecture uses fully connected layers followed by ReLU activations except at output layers.The architecture is illustrated in Figure A1.
  • Architecture: The network applies positional encodings γ(x) and γ(d) to the input coordinate and viewing direction.The coordinate and direction encodings support the network inputs used by the architecture.
  • Architecture: The design uses approximately the same number of parameters as the original NeRF network.This is an explicit architecture-design constraint.

A.1.2. Warm-Up Training

Warm-up training schedules geometric constraints in stages rather than optimizing them simultaneously, then reduces their weights to prioritize photometric accuracy.

  • The method first increases λd to enforce correct transmitted-component geometry.
  • It then increases λbdc to remove redundancies from the reflected component.
  • Both constraint weights are gradually decreased afterward so training focuses more on accurate renderings.

A.1.3. Training Details

Training uses staged constraint weighting, early transmitted-image supervision, reflection-map smoothness, and separate fine sampling for the two independently modeled components.

  • Constraint weights are scheduled through training, with λd and λbdc increased and later decreased to stabilize optimization and improve renderings.
  • The initial loss keeps the transmitted image close to the input image during the first 1k iterations.
  • A smoothness constraint penalizes neighboring differences in the reflection fraction map β.
  • The fine transmitted and reflected networks use separate fine-sample sets because their components have independent geometries.

A.2. Limitations and Failure Case

The appendix documents failure cases and evaluation behavior, including unstable curved reflections, non-coplanar surfaces, and the effects of warm-up training and user-provided masks.

  • Failure cases: Curved reflective surfaces with unstable virtual images produce inconsistent reflected geometry, causing inaccurate decomposition and rendering.
  • Failure cases: Multiple non-coplanar reflective surfaces are a limitation, although limited viewing angles can make a single reflected field viable in practice.
  • Evaluation: 29.16 average PSNR on LLFF is competitive with NeRF's 29.66 despite the added training priors.
  • Warm-up ablations: Directly applying strong geometric constraints can oversmooth transmitted depth and drive the reflected component toward an empty solution.
  • User guidance: Without reflection masks, the network struggles to distinguish transmitted from reflected geometry; four masks suffice for the mirror scene and one for the TV scene.
Loading 2111.15234v2…