Source-linked AI summary
Extracting Triangular 3D Models, Materials, and Lighting From Images
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, Sanja Fidler
TL;DR
Automating 3D content creation from images remains difficult because photogrammetry involves a complex multi-stage pipeline with errors that propagate between stages. This paper jointly optimizes topology, spatially varying materials, and environment lighting to extract explicit triangle meshes, achieving state-of-the-art-level view synthesis and material factorization with compatibility across modern 3D engines and modeling tools.
Problem
Photogrammetry requires a complex multi-stage pipeline, while efficient high-frequency environment-lighting approximations remain costly for interactive applications.
Method
The method jointly optimizes unknown-topology triangle meshes, spatially varying materials, and HDR environment lighting using differentiable tetrahedral surfaces and a split-sum lighting approximation.
Results
The extracted models achieve state-of-the-art-level view synthesis and material factorization while supporting scene editing, material decomposition, and high-quality view interpolation at interactive rates.
Takeaways & Limitations
The explicit representation is directly compatible with modern 3D engines and modeling tools, simplifying artist workflows and supporting broad scene-editing applications.
Takeaways & Limitations
The method performs less well on limited-view, low-view-count, or varying-illumination scans than recent neural implicit methods because it assumes constant lighting and uses a smoother shape representation.
Abstract
from arXiv · showhide
We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike recent multi-view reconstruction approaches, which typically produce entangled 3D representations encoded in neural networks, we output triangle meshes with spatially-varying materials and environment lighting that can be deployed in any traditional graphics engine unmodified. We leverage recent work in differentiable rendering, coordinate-based networks to compactly represent volumetric texturing, alongside differentiable marching tetrahedrons to enable gradient-based optimization directly on the surface mesh. Finally, we introduce a differentiable formulation of the split sum approximation of environment lighting to efficiently recover all-frequency lighting. Experiments show our extracted models used in advanced scene editing, material decomposition, and high quality view interpolation, all running at interactive rates in triangle-based renderers (rasterizers and path tracers). Project website: https://nvlabs.github.io/nvdiffrec/ .
1. Introduction
The introduction frames automated 3D content creation as an inverse-rendering problem and presents an efficient method that jointly extracts triangular geometry, spatially varying materials, and lighting from multiview images. Unlike neural-rendering approaches that entangle these components [45], the extracted models support editing, material decomposition, and high-quality view interpolation at interactive rates.
- Motivation: Automating 3D modeling can reduce production costs and enable faster, more diverse content creation, addressing the mostly manual and technically demanding nature of 3D content creation.Photogrammetry assists by converting multiple object photographs into 3D models, and game studios use it to build detailed virtual landscapes.
- Problem formulation: The authors formulate reconstruction as inverse rendering, jointly optimizing steps according to rendered-image agreement with captured input images.
- Motivation: Unlike neural-rendering methods that produce high-quality novel views but entangle geometry, materials, and lighting in neural networks [45], this approach extracts deployable triangular models.
- Contribution: The method extracts triangular meshes of unknown topology, spatially varying materials, and lighting from multiview images without an initial 3D geometry guess.It assumes one unknown environment-lighting condition, known camera poses, and object masks, as in prior work [5].
- Results: The reconstructed models support scene editing, material decomposition, and high-quality view interpolation at interactive rates in rasterizers and path tracers.
2. Related Work
Prior reconstruction methods use correspondences, voxels, or neural implicit representations, but face matching, memory, geometric, and computational limitations. This work instead optimizes an explicit mesh, materials, and lighting end-to-end from 2D image loss, extending differentiable topology optimization and remaining compatible with existing graphics tools.
- Classical Reconstruction: Classical methods rely on image correspondences or voxel grids, making them vulnerable to matching errors or cubic memory costs.Correspondence-based methods fuse estimated depth maps into point clouds and sometimes meshes, while voxel methods estimate occupancy and color per voxel.
- Neural Implicit Representations: Neural volumetric methods achieve strong novel-view synthesis but suffer geometric ambiguity, while converting implicit surfaces to meshes adds unoptimized error.NeRF-style methods compute radiance by ray marching through neurally encoded 5D light fields; surface-to-mesh conversion introduces additional error after optimization.
- Explicit Surface Representations: Explicit mesh methods address topology directly, but most assume fixed topology; DMTet [60] removes that constraint yet focuses on 3D supervision.This work extends DMTet [60] to 2D supervision using differentiable rendering to jointly optimize topology.
- Materials and Lighting: Radiometric reconstruction has progressed from specialized capture setups to neural and differentiable-rendering methods, but some light-transport models incur very high computational cost.BTF and SVBRDF methods require special configurations, lighting patterns, or complex capture; Bi et al. [4] and NeRV [64] model light transport for effects such as shadows.
- Positioning of This Work: Unlike neural reconstruction methods that can be slow and disconnected from established tools, this work outputs explicit meshes, BRDF parameters, and HDR-probe lighting compatible with existing renderers.The method jointly optimizes topology, materials, and environment lighting from 2D supervision, using differentiable marching tetrahedrons, volumetric texturing, and a differentiable split-sum approximation.
3. Our Approach
The approach jointly optimizes triangle-mesh topology, spatially varying materials, and environment lighting from multi-view images using differentiable rendering. It combines DMTet-based mesh optimization, compact volumetric material representations, and a differentiable split-sum lighting model.
- Optimization Task: Shape, materials, and lighting are jointly optimized by minimizing image, mask, and regularization losses through differentiable rendering and Adam [33].The renderer uses physically based high-dynamic-range shading, with Limage defined on tone-mapped colors and Lmask providing mask supervision.
- Assumptions: The deferred differentiable rasterizer supports diffuse Lambertian and isotropic GGX shading for dielectrics and metals, but excludes reflections, refractions, translucency, and shadows during optimization.These assumptions arise from using deferred shading and direct lighting for performance reasons.
- 3.1 Topology Optimization: DMTet directly optimizes triangle meshes during reconstruction, avoiding Marching Cubes discretization and decimation errors at moderate triangle counts.NeRF [45] and NeuS [67] can provide high-quality view interpolation, but Marching Cubes introduces significant quality loss at low triangle counts.
- 3.2 Spatially-Varying Shading Model: World-space volumetric texturing maintains smooth mappings during topology changes, while a positional-encoding MLP compactly represents material parameters near the surface.The resulting kd, korm, and n values form 2D textures compatible with standard 3D tools and game engines.
- 3.3 Environment Lighting: The method learns all-frequency environment lighting through a differentiable split-sum approximation with trainable cube-map texels and prefiltered mip levels.The split-sum model uses two texture lookups, standard GGX parameterization, and supports relighting with point and area lights.
4. Experiments
Experiments show that the explicit mesh-material-lighting representation supports relighting, editing, simulation, and competitive reconstruction across synthetic and real-world datasets. It also improves lighting approximation efficiency, while material reconstruction remains sensitive to global-illumination effects and novel-view quality is slightly lower than specialized methods.
- Relighting: Our method outperforms NeRFactor in all reported relighting and albedo metrics, producing more detailed results despite artifacts from rasterizer training versus global-illumination inference.Errors are concentrated in strongly shadowed or color-bleeding regions, where material and geometry quality suffer.
- Scene editing and simulation: Explicit triangle meshes enable advanced editing and soft-body simulation in off-the-shelf tools, including relighting, accurate shadows, and scene illumination in a Cornell box.These capabilities are difficult for neural volumetric representations, which generally support only rudimentary relighting and simple scene edits.
- Synthetic datasets: On the eight-scene NeRF realistic synthetic dataset, our method consistently performs on par with NeRF, with better quality in some scenes.The margin is smaller for perceptual metrics such as SSIM and LPIPS; DRUMS and SHIP are failure cases under significant reflections, refractions, caustics, and limited direct-lighting assumptions.
- View interpolation: Despite slightly lower novel-view quality than specialized view-interpolation methods, our approach produces high-quality interpolation; NeRFactor is 4.21 dB below the NeRF baseline, consistent with NeRD’s reported 4.17 dB reduction.The explicit factorization into shape, materials, and lighting explains the tradeoff relative to methods specialized for view interpolation.
- Real-world datasets: On real-world NeRD scans with inconsistent camera poses and masks, our method yields sharper silhouettes and significantly higher image quality than NeRF and NeRD.The mesh representation enforces opaque geometry, avoiding the floating geometry and holes that affect the competing volumetric methods.
- Lighting approximation: The split-sum lighting approximation captures environment lighting more faithfully across all frequencies than spherical Gaussians, while reducing optimization time 5× versus SG with 128 lobes.At inference, it requires only two texture lookups and has lower runtime cost than the SG model.
5. Limitations and Conclusions
The method achieves state-of-the-art-parallel view synthesis and material factorization with explicit triangle meshes, materials, and environment lighting compatible with modern 3D tools. Its main limitations are simplified shading without global illumination or shadows, reliance on alpha masks, static lighting and camera poses, and high training resource demands.
- Limitations: Simplified shading omits global illumination and shadows, accelerating optimization but limiting material extraction and relighting.The authors identify differentiable path tracing as a route for lifting this limitation in future work.
- Limitations: The system relies on alpha masks and additionally assumes static lighting and fixed camera poses during reconstruction.The method is described as robust to corrupted masks, but the passage indicates that mask handling could still be improved.
- Conclusions: Figure 11 states that the split sum approximation can capture high-frequency lighting when compared with a 128-lobe Spherical Gaussian approximation.The comparison uses 256 path-traced training images and reports arithmetic-mean image metrics over 16 novel test views.
- Limitations: Training requires high compute resources and substantial memory, while deepfakes are the only nefarious use case identified by the authors.The authors state that deepfakes are common to scene reconstruction methods and do not foresee other nefarious uses.
- Conclusions: The method reports results on par with state-of-the-art for view synthesis and material factorization using explicit triangle meshes with materials and environment lighting.This representation is directly compatible with modern 3D engines and modeling tools, supporting applications and simpler artist workflows.
6. Supplemental Material
The supplemental material provides additional results, ablations, and implementation details, including novel use cases, geometric-quality evaluation, per-scene statistics, visual examples, and efficient split-sum pre-integration.
- Supplement overview: Sections 7–9 add novel use cases, further evaluations, and implementation details for the method.The described use cases are automatic level-of-detail creation from images and appearance-aware model extraction; implementation includes efficient split-sum pre-integration.
- Section 7: Section 7 presents automatic level-of-detail creation from images and appearance-aware model extraction.
- Section 8: Section 8 reports geometric-quality evaluation, additional per-scene statistics, and visual examples.
- Section 9: Section 9 provides implementation details, including efficient split-sum pre-integration.
7. Novel applications
The paper extends image-based reconstruction to novel applications: learning topology and lighting for level-of-detail creation, extracting NeRFs into engine-compatible meshes, and jointly recovering topology and materials in challenging scenes.
- Novel applications: The paper generalizes appearance-driven level-of-detail creation [25] from fixed topology and known lighting to jointly learning topology and lighting directly from rendered images.The demonstration uses 256 path-traced views with masks and poses.
- Novel applications: The method extracts NeRFs into triangle meshes with textured PBR materials compatible with traditional 3D engines through NeRF, Marching Cubes, and differentiable rendering.It trains a NeRF, extracts its mesh with Marching Cubes, then finetunes geometry and learns 2D textures using image supervision, with DMTet topology optimization disabled.
- Novel applications: DMTet jointly learns challenging topology and materials, including highly specular models under high-frequency lighting, while preserving geometric detail and material quality.The demonstration uses known environment lighting and many views; Figure 14 compares approximations from 5000 observations with reference models, starting from a 128^3 tetrahedral grid.
8. Results
The results evaluate relighting, material separation, view interpolation, mesh geometry, and segmentation robustness. The method produces sharper novel views than NeuS and competitive geometry with fewer triangles, but performs worse on limited-view or varying-illumination scans.
- Relighting and materials: Relighting and material-separation evaluations provide per-scene results and visual comparisons with NeRFactor, while simplified shading omits strong-shadow and global-illumination effects.Relighting metrics average eight validation views under eight light probes; material comparisons include albedo, kd, and normals.
- View interpolation: Our method consistently produces greater detail and sharpness than NeuS on the KNOB, DAMICORNIS, and CERBERUS view-interpolation datasets.PhySG struggles with complex geometry in the NeRF synthetic scenes, while Figure 18 compares against NeRF and NeuS under known masks and camera poses.
- Mesh reconstruction: Our meshes use considerably fewer triangles than marching-cubes extractions while remaining competitive in Chamfer loss, although transparent scenes are challenging.The evaluation emphasizes image quality because tangent-space normal maps capture high-frequency appearance detail at modest triangle counts.
- Mesh reconstruction: Chamfer results vary with shape: neural implicit reconstruction performs well on organic forms, whereas NeRF attains the lowest loss on the CERBERUS robot.Figure 19 reports Chamfer L1 on synthetic datasets with increasing geometric complexity and compares NeRF, NeuS, and the explicit method.
- Robustness: Reconstruction quality decreases gracefully as segmentation masks deteriorate, and the system remains stable even under large synthetic corruptions.The corruption experiment is a stress test because every mask is degraded, unlike typical segmentation errors that are localized.
- Limitations: On scans with limited viewpoints, few images, or varying illumination, the method performs worse than NeuS, Unisurf, and IDR.The paper attributes this weakness to smoother neural-implicit shape regularization and the method’s physically based shading assumption of constant lighting.
9. Implementation
The implementation optimizes a 128-resolution tetrahedral grid with initialized geometry, textures, and lighting using scheduled Adam training, while regularizers suppress artifacts and stabilize materials, lighting, and mesh refinement.
- Implementation: A 128-resolution tetrahedral grid provides 1.53M tetrahedra and 277k vertices, with each tetrahedron generating up to two Marching Tetrahedral triangles.Per-vertex SDF values are randomly initialized so approximately 10% begin inside, while vertex offsets start at zero.
- Implementation: Training uses Adam with an exponential learning-rate decay from 1.0 to 0.1 over 5000 iterations, followed by 5000 iterations of locked-topology geometry and texture finetuning.Batches contain eight images rendered at native dataset resolutions, typically 512×512 to 1024×1024 pixels.
- Losses: The renderer uses tone-mapped L1 image loss, neutral-light regularization, NeRFactor-style material smoothness [79], and a trainable visibility term approximating ambient occlusion.The visibility term is stored in the otherwise unused o-channel of the korm specular-lobe texture and does not model directional visibility.
- SDF Regularizer: SDF regularization removes random internal faces and, in DTU experiments, visibility-based BCE regularization removes floaters behind surfaces unseen in training views.The proposed edge loss penalizes SDF sign changes and is reported to remove internal structures more efficiently than Liao et al..
- Lighting: Incoming lighting is represented by a 6×512×512 cube map whose mip levels are average-pooled and convolved with GGX filters using roughness-dependent bounds.Cube-map gradient computation uses GPU scatter operations with non-blocking atomic additions because filter footprints can cross cube edges and corners.
10. Scene Credits
The scene credits acknowledge contributed models, datasets, and rendered assets from named creators and repositories, with permissions and usage licenses specified. Smithsonian 3D assets are cited as.
- Model credits: Credited assets include Mori Knob, Cerberus, Damicornis, Saxophone, Jackson, and Spot models, supplied by named creators or NVIDIA under stated permissions or public-domain terms.Mori Knob is by Yasotoshi Mori; Cerberus is used with NVIDIA’s permission; Smithsonian 3D models are courtesy of the repository; Spot is by Keenan Crane.
- Dataset credits: The NeRD datasets moldGold-Cape and ethiopianHead are distributed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International.
- Dataset credits: NeRF and NeRFactor datasets contain renders from modified Blender models on BlendSwap, including chair by 1DInc and drums by bryanajones, with the chair marked CC-0.