Source-linked AI summary
NeRD: Neural Reflectance Decomposition from Image Collections
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, Hendrik P. A. Lensch
TL;DR
NeRD tackles the ill-posed decomposition of shape, reflectance, and illumination from image collections, especially when lighting varies or is unconstrained. It uses physically based differentiable rendering with explicit BRDF and illumination representations, then extracts relightable textured meshes; experiments report high-quality relightable assets and improved robustness to illumination changes, shadows, and glossy reflections.
Problem
Separating shape, illumination, and SVBRDF from images is highly ill-posed, while many implicit radiance-field methods support view synthesis but not relighting under varied illumination.
Method
NeRD jointly optimizes shape, spatially varying BRDF parameters, and spherical-Gaussian illumination in a coordinate-based representation using differentiable physical rendering.
Results
NeRD obtains high-quality relightable 3D assets and produces realistic real-time rendering and relighting under arbitrary unseen illuminations via explicit mesh extraction.
Takeaways & Limitations
Explicit reflectance and illumination decomposition makes the representation more robust to illumination changes, cast shadows, and glossy reflections than appearance-based representations.
Takeaways & Limitations
The method does not explicitly model shadowing during optimization, and its spherical-Gaussian environment model can limit high-frequency lighting effects.
Abstract
from arXiv · showhide
Decomposing a scene into its shape, reflectance, and illumination is a challenging but important problem in computer vision and graphics. This problem is inherently more challenging when the illumination is not a single light source under laboratory conditions but is instead an unconstrained environmental illumination. Though recent work has shown that implicit representations can be used to model the radiance field of an object, most of these techniques only enable view synthesis and not relighting. Additionally, evaluating these radiance fields is resource and time-intensive. We propose a neural reflectance decomposition (NeRD) technique that uses physically-based rendering to decompose the scene into spatially varying BRDF material properties. In contrast to existing techniques, our input images can be captured under different illumination conditions. In addition, we also propose techniques to convert the learned reflectance volume into a relightable textured mesh enabling fast real-time rendering with novel illuminations. We demonstrate the potential of the proposed approach with experiments on both synthetic and real datasets, where we are able to obtain high-quality relightable 3D assets from image collections. The datasets and code is available on the project page: https://markboss.me/publication/2021-nerd/
1. Introduction
Inverse rendering seeks shape and spatially varying material properties from images, but separating shape, illumination, and reflectance is highly ill-posed. NeRD addresses this by jointly decomposing these factors in a coordinate-based representation, supporting varied illumination and relightable mesh output.
- Inverse rendering estimates shape and spatially varying BRDF material properties from image collections for applications including view synthesis, relighting, and object insertion.
- The reflected color depends on integrating SVBRDF and incoming illumination over the hemisphere, making shape, material, and lighting difficult to disentangle.A dark region may reflect dark material, missing incident light, or a surface normal facing away from the light.
- Traditional controlled captures and newer practical setups constrain illumination, limiting estimation methods to laboratory conditions or flash photography.Recent practical methods typically assume a single dominant source such as a camera-mounted flash.
- NeRD explicitly decomposes shape, reflectance, and illumination within a NeRF-like representation using spatially varying BRDF parameters and jointly optimized spherical Gaussian lighting.Differentiable rendering minimizes photometric loss for each input image.
- NeRD accepts images captured under different illumination conditions and supports both camera motion and rotating-object captures with known poses and foreground masks.Illumination can be optimized per image for varying lighting or globally for static lighting.
- The learned coordinate-based representation can be converted into a textured 3D mesh with SVBRDF parameters for real-time rendering, relighting, and 3D asset generation.
2. Related Work
Neural scene representations use object-specific volumetric or implicit networks to synthesize novel views. NeRD follows this paradigm while replacing direct radiance modeling with reflectance decomposition and illumination-aware sampling.
- Neural scene representations: Neural scene representations model each scene with a trained volumetric, voxel, or implicit network and enable novel view synthesis through their inherent 3D structure.
- NeRD architecture: Figure 2 traces rays through sampled 3D points, Fourier embeddings, and spherical-Gaussian parameters before alpha-compositing colors using density.
3. Method
NeRD jointly learns shape, BRDF reflectance, and illumination from image collections, using differentiable physically based rendering within a NeRF-like volume. Its architecture handles varying illumination through illumination-conditioned sampling, compressed BRDF representations, density-derived normals, and conversion to a relightable mesh.
- Problem setup: NeRD estimates BRDF parameters, surface normals, density, and spherical-Gaussian environment illumination for each scene volume.The input may contain images captured under fixed or different illumination conditions; the environment uses 24 spherical-Gaussian lobes.
- Sampling network: The sampling network predicts illumination-dependent colors while keeping density illumination-independent, enabling useful volume sampling under varying lighting.It embeds the estimated illumination, compacts the 24 × 7 spherical-Gaussian parameters to 16 dimensions, and combines them with latent features before predicting color.
- Differentiable rendering: Differentiable rendering approximates the rendering equation with 24 spherical-Gaussian evaluations and backpropagates photometric loss to BRDF, normals, and illumination.Diffuse and specular reflectance are evaluated separately, with incoming and outgoing ray directions explicitly represented.
- Surface normal estimation: Surface normals are computed as the normalized negative density gradient, coupling normal estimation to the implicitly represented shape.Photometric optimization therefore updates the density field through the normal computation, while jittered ray samples encourage smoother normals.
- Compressed BRDF space: A two-dimensional scene-level BRDF bottleneck makes similar materials share embeddings, coupling estimates across surface points and improving decomposition robustness.The compressed embedding is used because real-world objects often contain spatially separated regions with similar BRDFs.
- Relightable mesh: After training, NeRD converts the learned reflectance volume into a textured mesh with BRDF parameters, enabling real-time rendering and relighting.The conversion generates a point cloud, computes a mesh and texture atlas, and fills the atlas with BRDF parameters.
4. Results
NeRD is evaluated on synthetic and real scenes for BRDF decomposition, novel-view synthesis, and relighting under varying or unseen illumination. Results show plausible reconstructions and flexible relighting, while decomposition remains affected by inherent ambiguity and illumination-model differences.
- Evaluation: NeRD recovers shape, appearance, and illumination for relighting, evaluated against ground-truth images and BRDF parameters on synthetic data and held-out views on real data.Known validation environment maps are used directly when available.
- BRDF decomposition: Synthetic scenes produce rerenderings very similar to ground truth, although estimated BRDF parameters can differ locally under passive unknown illumination.The deviations reflect decomposition ambiguity and differences between spherical-Gaussian and high-resolution ground-truth environment maps.
- Analysis: Joint optimization of shape and SVBRDF is identified as essential because separately or sequentially recovering scene components can fail in challenging scenes.The authors also report that COLMAP fails to reconstruct plausible geometry for some datasets, undermining downstream material estimation.
- Relighting: Real-world scenes yield plausible BRDFs and visually close relit views, including convincing results from different perspectives and completely novel artificial illuminations.For the gnome, estimated parameters are reported as largely unaffected by shadows, with material and roughness patterns matching the scene.
- Novel-view synthesis: On varying-illumination test views, NeRD mostly matches or surpasses NeRF-A while addressing relighting and more flexible capture conditions.NeRF performs poorly in varying illumination, whereas NeRF-A is a strong baseline but supports a simpler problem.
- Novel-view synthesis: On the Ethiopian Head, NeRD avoids the spurious geometry introduced by NeRF and produces convincing shapes, reflection properties, and novel relighting.NeRF-A improves reconstruction but cannot relight under arbitrary illumination when only one illumination type is observed.
5. Conclusion
NeRD explicitly decomposes shape, reflectance, and illumination, then extracts a textured mesh for realistic real-time rendering and relighting. Experiments show strong reconstruction behavior, while shadow modeling and high-frequency illumination remain limitations.
- 5. Conclusion: NeRD augments coordinate-based radiance fields with explicit BRDF and illumination representations for jointly decomposing shape, reflectance, and lighting.The method links surface normals to shape so photometric loss can update geometry through differentiable rendering.
- 5. Conclusion: The extracted textured mesh supports realistic real-time rendering and relighting under arbitrary unseen illuminations.Mesh extraction converts the neural volume into an explicit asset suitable for rendering beyond the training images.
- 5. Conclusion: NeRD does not explicitly model shadowing during optimization, and its spherical-Gaussian environment model is limiting for high-frequency lighting effects.In static environments with deep crevices, shadows can be baked into diffuse albedo.
- 5. Conclusion: The mesh-generation pipeline samples high-density surface points, removes outliers, reconstructs a surface, unwraps UVs, and bakes BRDF parameters into textures.The implementation uses Poisson reconstruction, optional mesh reduction, and one ray per texel for texture baking.
- 5. Conclusion: On real-world scenes, NeRD preserves details and reflective properties, reduces baked-in highlight artifacts, and handles varying-illumination cases where NeRF fails.Its diffuse estimates are more consistent than Li et al.’s, and specular-color estimation supports rendering metals correctly.