Source-linked AI summary
NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis
Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, Jonathan T. Barron
TL;DR
Relightable 3D reconstruction must recover geometry and materials from images under known lighting while supporting unseen viewpoints and illumination. NeRV uses continuous neural fields plus a learned visibility field to simulate direct and one-bounce indirect light efficiently, outperforming alternatives in complex lighting settings. Its central contribution is making physically based light transport tractable during optimization of a relightable volumetric representation.
Problem
NeRF-like volumetric representations do not separate incident light from material properties, while brute-force visibility computation is intractable beyond controlled single-light settings.
Method
NeRV represents scene properties with neural volumetric fields and trains a visibility MLP alongside density to query visibility and expected termination depth for light transport.
Results
NeRV outperforms all baselines under challenging complex lighting and matches prior work under simple lighting.
Takeaways & Limitations
Neural visibility fields make direct environment illumination and one-bounce indirect illumination tractable during training of continuous relightable scene representations.
Abstract
from arXiv · showhide
We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.
1. Introduction
NeRV targets relightable 3D scene reconstruction from images under known, unconstrained lighting, extending neural volumetric representations beyond fixed-lighting view synthesis. Its neural visibility field makes direct and one-bounce indirect illumination tractable during optimization.
- Motivation: Inferring geometry and materials from images could enable realistic 3D models for recognition, robotics, and graphics applications.The paper emphasizes novel-view rendering from unseen viewpoints and lighting conditions as a central graphics goal.
- Motivation: NeRF synthesizes photorealistic novel views by mapping 3D locations to continuous volume density and color, but does not model material-light interactions.NeRF’s outgoing-light representation ignores how incoming light interacts with surface materials.
- Challenge: Relighting neural volumes is computationally difficult because visibility requires querying density throughout potentially many rays, especially with multiple or environment light sources.Brute-force volumetric light transport becomes intractable for complex illumination and indirect lighting.
- Approach: NeRV trains a visibility MLP alongside the density MLP to query visibility and expected termination depth instead of densely sampling each corresponding ray.The visibility field is supervised for consistency with density samples observed during optimization.
- Approach: NeRV supports training with arbitrary environment lighting and one-bounce indirect illumination, whereas prior relightable NeRF methods were limited to controlled single-point-light settings.The method is designed to retain view synthesis while adding relighting under more complex illumination.
2. Related Work
Related work spans continuous neural scene representations, inverse rendering, and graphics methods for efficient global illumination. NeRV builds on these lines by using a neural visibility field for relighting with changing, unknown geometry.
- Neural scene representations: NeRF reformulates scene reconstruction as a continuous MLP field that represents complex scenes and renders photorealistic novel views.Unlike textured meshes and voxel grids, the representation is continuous and parameterized by an MLP.
- Inverse rendering: Inverse rendering and intrinsic image estimation seek to separate image variation caused by material, lighting, and geometry.Prior approaches address underconstrained inference using learned priors, known geometry, or multiple images under different lighting.
- Relightable neural fields: NeRV requires posed images under different known lighting conditions, placing it in the multi-light-input category of inverse-rendering methods.It extends Neural Reflectance Fields, which used a continuous MLP but required single-point-light illumination because of brute-force visibility computation.
- Relightable neural fields: Latent lighting codes have been used to relight neural representations by conditioning predicted appearance on per-image illumination.This strategy includes NeRF in the Wild and related discretized scene representations.
- Efficient light transport: Neural Visibility Fields are a neural analogue of graphics visibility precomputation, designed for inverse rendering while geometry changes during optimization.The approach connects learned function approximation with precomputation and approximation strategies for efficient global illumination.
3. Method
NeRV extends NeRF with reflectance and neural visibility fields to model direct and indirect light transport efficiently in a continuous volumetric scene representation. The visibility field replaces expensive ray-wise density integrations, enabling relighting under environment lighting and one-bounce indirect illumination during training.
- 3.2. Neural Reflectance Fields: NeRV represents scenes as continuous 3D fields of oriented particles that absorb and reflect external light, rather than NeRF particles that absorb and emit light.The representation includes volumetric density, surface normals, and BRDF-based material properties.
- 3.3. Light Transport via Neural Visibility Fields: A visibility MLP predicts environment visibility and expected termination depth from a 3D location and direction, replacing brute-force density integrals during rendering.The MLP is optimized alongside the shape MLP and trained for consistency with visibility and termination-depth estimates implied by the shape field.
- 3.3. Light Transport via Neural Visibility Fields: Direct illumination combines known environment lighting with predicted visibility, while indirect illumination is computed from reflected incoming radiance and BRDF responses.NeRV decomposes reflected radiance into direct and indirect components and uses predicted visibility and intersection depth to evaluate these terms.
- 3.3. Light Transport via Neural Visibility Fields: NeRV reduces direct-lighting complexity from quadratic to linear in the number of samples per ray and indirect-lighting complexity from cubic to linear.This approximation enables environment-lighting simulation and one-bounce indirect illumination within optimization of the continuous representation.
- 3.4. Rendering: The full rendering procedure composites radiance samples along camera rays using NeRF’s numerical quadrature and the visibility-aware volume-rendering formulation.The training objective combines image-rendering loss with losses that encourage predicted visibility and termination depth to match shape-MLP estimates.
4. Results
NeRV recovers relightable scene representations under complex lighting, where prior methods struggle, while matching prior work under simple single-point lighting. Its evaluations also show benefits from indirect illumination and analytically computed normals.
- Baseline comparisons: Latent-code baselines fail to generalize to lighting conditions unlike those observed during training, whereas NeRV produces high-quality test renderings.The comparison uses the “Ambient+Point” dataset.
- Overall results: NeRV matches prior work under single-point lighting but outperforms all baselines under challenging complex lighting conditions.Performance degrades for Bi et al. and latent-code methods when lighting includes multiple or ambient sources.
- Baseline comparisons: Bi et al. performs comparably to NeRV with a single point source but degrades on “Colorful+Point” and “Ambient+Point” training datasets.Its forward model cannot simulate light from more than one source during training.
- Baseline comparisons: NeRV generally matches or outperforms NLT despite requiring unconstrained known lighting rather than controlled OLAT capture and substantially fewer input images.The NLT training dataset contains eight times as many images as the other datasets.
- Ablation studies: Modeling indirect illumination improves performance and produces realistic interreflection details, including additional brightness inside the Lego bulldozer’s cab.The ablation study evaluates one-bounce indirect illumination on the “Ambient+Point” dataset.
- Ablation studies: Analytic normals produce more accurate estimated surface normals than MLP-predicted normals, although the numerical performance difference is smaller.The two normal variants produce similar renderings.
5. Conclusion
The paper demonstrates relightable neural volumetric representations for scenes with environmental and indirect lighting. It uses a visibility MLP to approximate volume-rendering terms that brute-force sampling makes intractable during training.
- Conclusion: NeRV recovers relightable neural volumetric representations from images captured under environmental and indirect lighting.The method targets arbitrary relighting through learned visibility-field approximation.
- Conclusion: A visibility MLP approximates portions of the volume-rendering integral that would otherwise be intractable to estimate by brute-force sampling during training.The conclusion frames learned function approximation as a way to reduce the computational burden of physically based differentiable rendering.
A. BRDF Parameterization
NeRV uses a standard microfacet BRDF with diffuse albedo, specular roughness, and surface normal parameters, incorporating a Lambert cosine term into the BRDF definition.
- BRDF definition: The reflectance function is the standard microfacet BRDF described by Walter et al., with implementation simplifications from Filament and Unreal Engine.It is defined for 3D location x, incoming direction ωi, and outgoing direction ωo.
- BRDF parameters: The BRDF uses diffuse RGB albedo a, scalar roughness γ, and surface normal n at 3D point x.The parameterization also uses F0 = 0.04 for dielectric materials.
- BRDF definition: The implementation includes the Lambert cosine factor (n · ωi) inside the BRDF to simplify the main-paper equations.The gloss and geometry terms are expressed using the surface normal and incoming and outgoing directions.
- Comparison context: The supplementary results compare NeRV with Neural Light Transport under different capture requirements and recovered-geometry settings.NLT uses controlled laboratory lighting, eight times as many images, and input proxy geometry.
B. Additional Qualitative Results
Additional qualitative results show that NeRV recovers effective relightable 3D representations under complex illumination. Bi et al. and latent-code methods fail in lighting regimes outside their supported or observed conditions.
- Additional qualitative results: NeRV renders convincing novel viewpoints under novel lighting conditions in the additional qualitative comparisons.The comparisons include complex illumination settings.
- Additional qualitative results: Bi et al. fails to recover accurate models when trained with illumination more complex than a single point light.The failure appears in columns 3–6 of the qualitative comparison.
- Additional qualitative results: Latent-code methods such as NeRF+LE and NeRF+Env cannot generalize to lighting conditions unlike those seen during training.These methods model lighting-dependent appearance variation through latent codes.