Source-linked AI summary
NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, Jonathan T. Barron
TL;DR
Recovering full 3D shape and spatially varying reflectance from multi-view images under one unknown illumination is underconstrained. NeRFactor distills NeRF geometry into a surface representation and jointly estimates geometry, reflectance, visibility, and lighting using re-rendering and priors. It recovers convincing relightable models for synthetic and real scenes and reports performance exceeding classic and deep-learning baselines across tasks.
Problem
Recovering geometry and material properties for novel-view rendering under unknown illumination is a longstanding, fundamentally underconstrained problem.
Method
NeRFactor distills NeRF volumetric geometry into a surface representation, then jointly optimizes geometry, reflectance, visibility, and illumination with smoothness and learned BRDF priors.
Results
NeRFactor recovers convincing 3D models for free-viewpoint relighting under one unknown illumination and outperforms classic and deep-learning state of the art across various tasks.
Takeaways & Limitations
Explicit visibility supports shadow removal from albedo estimation and realistic soft or hard shadows under arbitrary lighting, alongside material editing and novel-view synthesis.
Takeaways & Limitations
The method limits light probes to 16 × 32, models only single-bounce direct illumination, and can fail with particularly poor NeRF geometry.
Abstract
from arXiv · showhide
We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.
1 INTRODUCTION
NeRFactor recovers relightable 3D shape and reflectance from multi-view images captured under one unknown illumination. It uses NeRF-derived surface initialization, joint factor optimization, visibility modeling, smoothness priors, and a learned BRDF prior.
- NeRFactor targets the underconstrained recovery of geometry and material properties for arbitrary-view rendering under novel lighting.
- The method distills NeRF geometry into a hard-surface representation and models normals and visibility as spatially smooth continuous MLP functions.Shading is computed at the expected termination depth rather than throughout the volume.
- Its factorization uses re-rendering loss, spatial smoothness constraints, and a data-driven BRDF prior learned from real measured BRDFs.
- NeRFactor jointly estimates surface normals, light visibility, albedo, spatially varying BRDFs, and environment lighting from multi-view images under one unknown condition.The recovered representation supports novel viewpoints under arbitrary lighting.
- Explicit visibility separates shadows from albedo and enables synthesis of realistic soft or hard shadows under arbitrary novel lighting.
- NeRFactor assumes hard surfaces with one intersection per ray, direct illumination only, and achromatic specular reflectance, excluding scattering, transparency, translucency, and metallic materials.
2 RELATED WORK
Prior inverse-rendering methods address the underconstrained problem through strong learned priors or additional observations. NeRFactor instead combines neural geometry with unknown single-illumination factorization and explicit visibility for full 3D relighting.
- Inverse rendering factorizes observed appearance into geometry, material properties, and lighting, but the general problem is severely underconstrained.
- Single-image methods infer plausible factors using strong priors, but do not recover full 3D representations viewable from arbitrary viewpoints.
- Full-3D relighting methods commonly require additional observations such as scanned geometry, proxy models, silhouettes, or multiple known lighting conditions.
- NeRFactor uses initial geometry estimated from NeRF to recover factored models for challenging objects with highly reflective surfaces and detailed geometry.
- Material-acquisition methods traditionally estimate BRDFs using known planar geometry and complex controlled camera and lighting setups.
- Neural geometry methods represent continuous 3D properties with MLPs mapping coordinates to quantities such as density, occupancy, or signed distance.
- Compared with concurrent relighting approaches, NeRFactor handles one unknown illumination, models visibility and shadows, and uses a learned rather than analytic BRDF.
3 METHOD
NeRFactor represents shape and spatially varying reflectance as neural fields, initializing surface geometry and visibility by distilling an optimized NeRF. It jointly refines these fields with reconstruction, smoothness, and learned BRDF priors to explain views under unknown illumination.
- Model representation: NeRFactor parameterizes surface normals, light visibility, albedo, and BRDF reflectance as 3D fields that explain multi-view images captured under one unknown illumination.
- Geometry initialization: The method distills an optimized NeRF into expected surface locations, normals, and directional visibility, then re-parameterizes them with MLPs for joint refinement.MVS geometry can also initialize the model.
- Geometry initialization: Surface rendering is made efficient by evaluating outgoing radiance at each ray’s expected termination surface instead of sampling every point along the ray.
- Geometry refinement: Normal and visibility MLPs are regularized toward NeRF estimates and spatial smoothness while also reproducing observed appearance, reducing noisy geometry artifacts.Normal smoothness uses displaced surface locations, and visibility smoothness is applied across spatial locations for a fixed light direction.
- Geometry refinement: The method pretrains geometry fields before joint optimization so albedo or BRDF fields do not explain shadows as painted reflectance variation.Without geometry pretraining, the model predicts worse albedo with shading residuals.
- Reflectance modeling: NeRFactor combines Lambertian albedo with a learned non-diffuse BRDF prior trained on measured real-world reflectances, rather than relying only on an unrestricted analytic parameterization.The learned prior encourages plausible reflectance functions under the single unknown lighting condition.
4 RESULTS & APPLICATIONS
NeRFactor demonstrates high-quality geometry, appearance factorization, novel-view relighting, MVS initialization, and material editing across synthetic and real scenes. Smoothness constraints improve relighting geometry, while explicit visibility supports realistic shadows and shading-free albedo.
- High-quality geometry: Smoothness constraints remove noisy geometry artifacts that remain after directly deriving or jointly optimizing NeRF geometry, producing relightable surface normals and visibility.Without spatial smoothness, harsh OLAT lighting reveals geometry-induced artifacts; with smoothness, the estimates resemble ground truth.
- Joint estimation of shape, reflectance, and lighting: NeRFactor recovers smooth surface normals, ambient occlusion, clean albedo, and spatially varying BRDFs that reflect different materials across synthetic scenes.Normals are close to ground truth except in very high-frequency regions, while albedo contains little shadowing and BRDF latent codes vary spatially.
- Free-viewpoint relighting: NeRFactor synthesizes novel views under OLAT and arbitrary probes with realistic specularities, hard shadows, and soft shadows.Its direct-illumination model can make ground-truth global-illumination images appear brighter, but visibility modeling enables cast-shadow synthesis.
- Free-viewpoint relighting: Real-world captures are factorized into lighting, normals, visibility, albedo, and BRDF codes, enabling relighting with realistic specularities and shadows.The estimated flower albedo is shading-free, and the fully 3D factorization supports novel viewpoints.
- MVS initialization and material editing: MVS initialization yields denoised geometry that preserves details and supports realistic view synthesis, relighting, and shadows.Higher-quality geometry retains geometric detail while removing MVS noise; NeRFactor also supports editing albedo and non-diffuse BRDFs before re-rendering.
5 EVALUATION STUDIES
The evaluation studies ablate major components and compare NeRFactor with classic and deep learning-based methods for appearance factorization and free-viewpoint relighting. Results show that its learned BRDF prior, geometry pretraining, and smoothness constraints support performance beyond alternatives and baselines.
- Ablation Studies: The learned BRDF variant outperforms the analytic BRDF alternative overall, although the analytic model performs well across all tasks.Removing smoothness on predicted roughness was necessary for the analytic variant, which could otherwise converge to maximum roughness everywhere and sometimes produce noisy renderings.
- Ablation Studies: Geometry pretraining improves albedo estimation and overall performance by preventing the albedo model from explaining shadows.Training the normal and visibility networks from scratch produces worse albedo with shading residuals.
- Ablation Studies: Removing smoothness constraints preserves view synthesis quality but hurts albedo estimation and relighting, producing noisy estimates insufficient for relighting.This contrasts with view synthesis, where NeRF can achieve high quality without smoothness constraints.
- Ablation Studies: Using NeRF’s unrefined shape leaves noisy normals and visibility that produce artifacts in final renderings.The reflectance remains smooth because of the full model’s smoothness priors, but geometry-related noise persists in rendering.
- Baseline Comparisons: The improved Oxholm and Nishino baseline struggles with shadow residuals and spatially varying BRDFs despite receiving ground-truth illumination, while NeRFactor models visibility explicitly.The baseline’s albedo residuals negatively affect relighting, and its non-spatially-varying BRDF misestimates materials with different spatial properties.
- Baseline Comparisons: NeRFactor outperforms Philip et al. [2019] + Masks in both PSNR and SSIM under the reported point-light relighting comparison.The masking provides a more generous comparison by removing the baseline’s yellow background fog with ground-truth object masks.
6 LIMITATIONS
NeRFactor has limitations from its light-probe resolution, direct-illumination approximation, and dependence on sufficiently good NeRF or MVS geometry initialization.
- Light and illumination limits: The 16 × 32 light-probe resolution may be insufficient for very hard shadows or very high-frequency BRDFs.Very high-frequency illumination can leave specularity or shadow residuals in albedo estimation.
- Light and illumination limits: NeRFactor models only single-bounce direct illumination, so it does not properly account for indirect illumination effects.
- Geometry initialization: NeRFactor can fail when NeRF estimates particularly poor geometry that does not affect view synthesis.Real-world scenes contained faraway incorrect floating geometry that was invisible from input cameras but cast shadows on objects.
7 CONCLUSION
The conclusion presents NeRFactor as a prior-driven method for recovering shape, reflectance, and lighting from multi-view images under unknown illumination, enabling relighting, novel views, and material editing.
- Conclusion: NeRFactor recovers shape and reflectance from multi-view images and camera poses captured under one unknown illumination condition.The method addresses an ill-posed problem using spatial smoothness constraints and a data-driven prior on real-world BRDFs.
- Conclusion: The recovered factorization supports point-light or light-probe relighting, arbitrary viewpoints, and editing of the object’s albedo and reflectance.
- Conclusion: NeRFactor produces high-quality geometry for relighting and view synthesis, convincing albedo and spatially varying BRDFs, and lighting estimates reflecting dominant light sources.
A IMPLEMENTATION DETAILS
Implementation uses staged training: scene-specific NeRF optimization, MERL-based BRDF pretraining, geometry pretraining, and final joint optimization with MLP-based components.
- Implementation: The implementation uses TensorFlow 2 and the Adam optimizer with default hyperparameters.
- Training pipeline: NeRFactor first optimizes a NeRF per scene and trains a BRDF MLP on MERL, then freezes both during final joint optimization.NeRF supplies shape initialization, while the BRDF MLP supplies a latent space of real-world BRDFs.
- Network architecture: The introduced MLPs use four layers with 128 hidden units and a skip connection from the input to the second layer.Positional encoding uses 10 levels for 3D locations and 4 levels for directions.
- Training cost: NeRF training takes 2,000 epochs and 6–8 hours on four NVIDIA TITAN RTX GPUs, while initial normal and visibility computation takes 30 minutes per view.Geometry pretraining takes 200 epochs and around 20 minutes on a TITAN RTX.
B DATA
The study combines multi-view object images and camera poses with measured BRDFs and captured light probes. It evaluates synthetic and real scenes under varied illumination and imaging setups.
- The dataset uses multi-view images with corresponding camera poses, real-world measured BRDFs, and captured light probes.
- Synthetic Blender scenes are relit with natural illuminations from real light-probe images to create more natural input conditions.
- Real mobile-phone scenes contain close to 100 images each, with camera poses obtained through COLMAP structure-from-motion.
- DTU-MVS scenes provide around 50 multi-view images and corresponding camera poses, alongside Poisson-reconstructed surfaces.
- The measured BRDFs come from MERL, which contains 100 real-world BRDFs measured using a conventional gonioreflectometer.
C ADDITIONAL EVALUATION STUDIES
Additional studies test albedo consistency across illumination conditions, visualize component ablations, and compare NeRFactor with SIRFS.
- The evaluation examines albedo consistency across different input lighting conditions and supplements quantitative ablations with qualitative visualizations.
- Four drastically different illuminations applied to ficus are used to estimate albedo consistency from separate multi-view image sets.
- Pairwise PSNR ≥34.7 dB indicates similar albedo predictions across the four input illuminations.
C.2 Qualitative Ablation Studies
Qualitative ablations show that NeRFactor’s geometry refinement, smoothness regularization, initialization, and learned BRDF prior each address distinct reconstruction problems. The comparison with SIRFS highlights the role of multi-view, full-3D estimation.
- C.2 Qualitative Ablation Studies: Fixing NeRF geometry while removing normal and visibility MLPs leaves geometry too noisy for relighting.
- C.2 Qualitative Ablation Studies: Removing smoothness regularization produces noisy geometry and albedo, while training normal and visibility MLPs from scratch can make albedo explain shading and shadows.
- C.2 Qualitative Ablation Studies: Replacing the learned BRDF with a roughness-predicting MLP causes maximum roughness everywhere or spatially non-smooth reflectance.
- C.3 More Baseline Comparisons: NeRFactor estimates normals, albedo, shading-related properties, BRDFs, and visibility in full 3D, whereas SIRFS uses one image and does not model shadowing.
- C.3 More Baseline Comparisons: SIRFS produces reasonable albedo but inaccurate surface normals and cannot render arbitrary viewpoints or synthesize shadows during relighting.