PaperlayerView exact revision on arXiv

Source-linked AI summary

Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising

Jon Hasselgren, Nikolai Hofmann, Jacob Munkberg

arXiv:2206.03380v2cs.GRcs.CV

TL;DR

Simplified renderers make physically plausible decomposition difficult, while Monte Carlo rendering introduces noise that challenges gradient-based inverse rendering. The paper combines ray-traced direct illumination with multiple importance sampling and differentiable denoising to jointly reconstruct explicit geometry, materials, and lighting. It reports improved material-light reconstruction over previous work while remaining competitive in optimization time, with limitations including direct-illumination scope and computational intensity.

  • Problem

    Simplified shading models make it harder to plausibly disentangle shape, material, and lighting, while Monte Carlo noise makes gradient-based optimization challenging.

  • Method

    The pipeline jointly optimizes explicit triangle meshes, PBR materials, and environment lighting using ray-traced Monte Carlo direct illumination, multiple importance sampling, and differentiable denoising.

  • Results

    The physically based renderer improves material and light reconstruction over previous work while remaining competitive in optimization time.

  • Takeaways & Limitations

    Denoising makes low-sample gradient-based inverse rendering practical and can save computational resources during reconstruction and later fine-tuning.

  • Takeaways & Limitations

    The approach is restricted to direct illumination, omits specular chains, uses biased rendering, relies on foreground segmentation, and requires a high-end GPU.

Abstract

from arXiv · show

Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials & lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This substantially improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.

1 Introduction

Existing reconstruction methods often use simplified shading, making physically plausible separation of shape, materials, and lighting difficult. This paper combines physically based Monte Carlo rendering with variance-reduction techniques to improve decomposition and relighting.

  • Differentiable rendering supports high-quality multi-view reconstruction using neural fields, signed distance fields, or triangle meshes.
  • Simplified shading models and baked appearance make plausible shape, material, and lighting disentanglement harder.
  • Monte Carlo rendering provides more accurate light simulation but introduces noise that makes gradient-based optimization difficult and can require intractably many samples.
  • The method reconstructs explicit triangle meshes, PBR materials, and environment lighting from multi-view images for compatibility with DCC tools and game engines.
  • Compared with previous methods, the formulation primarily improves material and light separation.
  • Multiple importance sampling, differentiable denoisers, and decomposition into lighting components reduce variance and improve material-light separation.

2 Previous Work

Previous work reconstructs scenes with implicit or explicit representations and estimates materials and lighting through differentiable rendering. Related methods commonly approximate illumination or use learned irradiance, while this system extends that line with Monte Carlo rendering and denoising.

  • Neural methods for multi-view reconstruction: NeRF-style methods use volumetric neural light fields for novel-view synthesis, but volume-rendering ambiguity can reduce geometric quality.
  • BRDF and lighting estimation: BRDF and lighting estimation methods range from specialized capture setups to neural prediction and photometric-loss-based differentiable rendering.
  • BRDF and lighting estimation: Intrinsic decomposition methods estimate shape, materials, and lighting using spherical Gaussians, pre-filtered approximations, low-resolution environment maps, or neural indirect illumination.
  • BRDF and lighting estimation: Some related approaches account for shadows by splitting optimization into passes and locking geometry before sampling the shadow term.
  • Image denoisers: Cross-bilateral filters traditionally require scene-specific manual adjustments, whereas newer neural denoisers avoid manual tuning through dataset training.
  • Image denoisers: The proposed pipeline directly incorporates differentiable versions of both traditional and neural denoisers.

3 System

The system jointly optimizes shape, spatially varying materials, and environment lighting from multi-view images using an explicit mesh extracted from an SDF and rendered with a differentiable physically based renderer. Monte Carlo illumination, denoising, and variance-aware optimization address the noise that otherwise impedes reconstruction.

  • 3 System: The task jointly optimizes shape, material, and environment lighting from multi-view images with known foreground masks and camera poses.
  • 3 System: Geometry is represented by a signed distance field and converted into a triangular mesh with DMTet for differentiable optimization.
  • 3 System: The renderer uses Disney's PBR model, combining diffuse shading with an isotropic specular GGX lobe and a tangent-space normal map.
  • 3 System: A differentiable denoiser reduces variance in the noisy Monte Carlo render, and gradients are optimized with respect to both scene parameters and denoiser parameters.
  • 3.1 Direct Illumination: Monte Carlo direct illumination supports all-frequency lighting with shadows by sampling incident directions and tracing visibility rays.
  • 3.1 Direct Illumination: Low-sample Monte Carlo variance makes gradient optimization hard, motivating variance-reduction techniques for complex geometry, lighting, and spatially varying BSDFs.
  • 3.1 Direct Illumination: Shadow-ray visibility gradients are detached because diffuse-scattering gradients were observed to be negligible relative to primary-visibility gradients in the multi-view setting.

4 Variance Reduction

The pipeline combines multiple importance sampling with differentiable denoising to reduce Monte Carlo noise and make physically based inverse rendering practical at low sample counts.

  • Multiple Importance Sampling: Multiple importance sampling combines light, cosine, and GGX importance sampling to reduce variance across materials.The sampling distributions are recomputed each optimization iteration because material and lighting parameters change.
  • Denoising: Denoising improves both forward image quality and gradient estimation by filtering noisy renders and sharing gradients across neighboring pixels.Denoisers trade reduced variance for increased bias, but empirically improve convergence at lower sample counts.
  • Lighting Decomposition: Separating diffuse and specular lighting, including demodulated diffuse lighting, enables selective denoising and regularization without smearing texture detail.The final image composes diffuse albedo with denoised diffuse lighting and adds denoised specular lighting.
  • Empirical Effects: The most complex Porsche scene failed to converge at 8 spp without denoising, highlighting the difficulty of noisy optimization.The ablation compares denoising algorithms across three scenes of increasing complexity using averaged PSNR over 200 novel views.
  • Empirical Effects: At low sample counts, denoising achieves results at 8 spp comparable to 32 spp without denoising, while its benefit diminishes at higher counts.Denoising also produced higher-quality light probes at both low and high sample counts.
  • Denoiser Optimization: Jointly optimizing denoiser parameters can bake features into denoiser weights; locked-weight denoisers performed better than fine-tuned alternatives.OIDN produced color shifts, while a more constrained hierarchical kernel predictor improved results but still lagged behind locked weights.

5 Priors

The method uses smoothness and lighting-specific priors to disentangle materials from lighting, addressing the tendency to bake shadows into albedo textures.

  • Material Priors: Smoothness priors are applied to albedo, specular, and normal-map textures, but are insufficient for high-frequency lighting and sharp shadows.The albedo prior compares diffuse albedo values at nearby world-space surface positions.
  • Lighting Regularization: A novel regularizer compares monochrome demodulated lighting with the reference image to encourage shading detail to remain represented by lighting.This discourages encoding shadows in material textures when high-intensity, localized environment lighting is harder to optimize.
  • Lighting Regularization: The regularizer uses average RGB luminance for rendered lighting and the HSV-value maximum for the reference image as an approximate demodulation.The luminance operator preserves differentiability, while HSV-value ignores chrominance and cannot distinguish shadows from darker materials.

6 Experiments

Experiments show that the method improves material and lighting separation across challenging synthetic and real-photo reconstruction settings, while denoising improves detail at equal sample counts. The approach produces editable explicit meshes and remains competitive in optimization time, although view interpolation is slightly worse than NVDIFFREC.

  • Synthetic relighting: The method outperforms previous work in material reconstruction on synthetic relighting datasets, including the more challenging NeRF setting.NeRF uses higher-frequency lighting and global illumination than NeRFactor, making it substantially more difficult.
  • Real photographs: Real-photo experiments produce editable triangle meshes with convincing material editing and relighting in Blender.The method also decomposes geometry, material parameters, and environment light from real photographs.
  • Denoising: Denoising improves visual quality and environment-lighting detail at equal sample counts, especially around specular highlights and cockpit occlusions.The Porsche study compares optimization with and without denoising.
  • View interpolation: Material and lighting regularization slightly degrades view interpolation, leaving scores slightly below NVDIFFREC despite considerably better separation.The paper contrasts this trade-off with neural light-fields, which excel at view interpolation.
  • Compute resources: Optimization uses 128+ rays per pixel with 5000×2 iterations and takes approximately 4 hours on an A6000 under the reported settings.Ray tracing costs more than pre-filtered environment-light approaches, while runtime scales linearly with sample count.

7 Conclusions

The paper concludes that differentiable Monte Carlo rendering with variance reduction is practical for explicit triangular-mesh reconstruction. It improves material and light reconstruction while remaining competitive in optimization time.

  • 7 Conclusions: Differentiable Monte Carlo rendering combined with variance-reduction techniques is practical for multi-view reconstruction of explicit triangular 3D models.The conclusion emphasizes applicability to object reconstruction using explicit geometry.
  • 7 Conclusions: The physically based renderer improves material and light reconstruction over previous work while remaining competitive in optimization time.Hardware-accelerated ray tracing and differentiable image denoisers support this runtime comparison.

A Optimization and Regularization

The pipeline combines physically based HDR rendering with regularization and staged optimization to stabilize joint reconstruction of geometry, materials, and lighting. These measures address noisy gradients, topology changes, and ambiguities between texture, normals, shadows, and illumination.

  • Image loss: The image loss applies L1 error to tonemapped colors, making optimization robust to the renderer’s high-dynamic-range outputs.The tone map transforms linear radiance x using T(x) = Γ(log(x + 1)).
  • Regularizers: Smoothness priors regularize albedo, diffuse and specular parameters, occlusion, and geometric surface normals to guide decomposition under constant lighting.The method relies on several priors because multi-view images use constant lighting.
  • Regularizers: Regularizing geometric surface normals with small spatial perturbations encourages smoother geometry, particularly during early training.The perturbation uses ϵ ∼ N(0, σ = 0.01).
  • Regularizers: Normal mapping can incorrectly turn environment lighting into a color dictionary, so the method regularizes micro-geometry normal perturbations toward the unperturbed tangent-space normal.This addresses decorrelation between geometric surfaces and texture-based normal perturbations.
  • Regularizers: A monochrome loss on demodulated diffuse and specular lighting improves lighting detail, especially under high-frequency lighting, despite the inability to demodulate the reference image.The luminance and HSV-value operators compare lighting terms with the target image after tonemapping.
  • Staged optimization: Blending the shadow term over the first 1750 iterations improves early convergence when random-topology changes would otherwise create spiky, noisy gradients and poor geometry.The denoiser footprint is also ramped up to avoid interfering with early topology optimization.

B Denoising

The method uses bilateral and neural denoisers during low-sample-count optimization, gradually introducing their spatial influence. Denoising supports reconstruction, but jointly fine-tuning a denoiser can overfit scene features into its weights.

  • Bilateral denoising: The bilateral denoiser combines spatial, depth, and surface-normal components to filter rendered colors while preserving relevant edges.Its spatial footprint σ increases from 1e-4 to 2.0 over the first 1750 iterations.
  • Evaluation: Figure 13 compares denoising algorithms across three increasingly complex scenes using averaged PSNR over 200 novel views.The most complex scene fails to converge at 8 spp without denoising.
  • Evaluation: Figure 14 isolates denoising during HDR environment-light optimization with fixed geometry and materials, reporting light probes and average PSNR.The evaluation uses 8 spp for probes and high-sample-count rendering without denoising for the PSNR measurements.
  • Evaluation: Figure 15 evaluates denoising while jointly optimizing geometry, materials, and environment lighting on the NeRFactor Hotdog scene.The reconstruction results are shown at 32 spp, with PSNR averaged over 200 novel views.
  • Limitations: Fine-tuning denoisers during scene optimization can bake features into network weights instead of scene parameters, negatively affecting reconstruction quality.The paper studies this effect alongside isolated light-probe and joint scene optimization.

C Visibility Gradients from Shadows

Shadow visibility gradients help some targeted inverse-rendering problems but offer less clear benefits for joint multi-view optimization. They can also increase noise and degrade geometry reconstruction quality.

  • Joint optimization: In joint shape, material, and environment-light optimization, shadow-ray gradients typically have negligible impact relative to primary-visibility gradients.The comparison uses view-interpolation scores over validation sets.
  • Joint optimization: Shadow visibility gradients sometimes increase noise levels and degrade reconstruction quality.Figure 17 visualizes slightly noisier geometry reconstructions through surface normals.

D Sampling

The sampling ablations favor multiple importance sampling and correlated forward/backward samples for stable low-sample optimization. Geometry quality remains similar to NVDIFFREC overall, while the evaluation covers relighting and view interpolation across synthetic datasets.

  • Importance sampling: Multiple importance sampling with the balance heuristic consistently produces high-quality reconstruction across diffuse, plastic, and metal materials.The comparison includes cosine, BSDF, light-probe, and multiple importance sampling.
  • Gradient sampling: Reusing the same random seeds in forward and backward rendering drastically reduces noise and improves convergence at low sample counts.This introduces bias but aligns gradients with the exact parameters contributing to the forward rendering.
  • Geometry: The extracted meshes have geometric quality similar to NVDIFFREC overall, with Lego as an outlier because normal smoothness is difficult for its holes and sharp edges.Lower Chamfer L1 scores indicate better geometry quality.
  • Evaluation: Relighting evaluation reports per-scene quality over eight novel views under multiple low- and high-resolution light probes.NeRFactor uses eight 32 × 16 probes, while NeRF and the paper’s dataset use four 2048 × 1024 probes.
  • Evaluation: View-interpolation quality is measured as the arithmetic average over 200 test images for each scene.Training sets contain 100 random-view images for NeRFactor and NeRF, and larger sets for Apollo, Porsche, and Roller.

G Scene Credits

This section credits the sources for several 3D models and datasets and presents relighting examples across NeRFactor and NeRF scenes under varied light probes.

  • Scene Credits: The credits identify model, scene, and dataset sources including Mori Knob, Bob, Rollercoaster, Porsche, Apollo, Damicornis, Family, Character, and Gold Cape.The listed sources include LDraw, the Smithsonian 3D repository, Tanks&Temples, BlendedMVS, and NeRD, with stated licenses.
  • Relighting Examples: Figures 21–23 show Hotdog, Ficus, and Lego scenes relit under six of eight low-frequency test-set light probes.These examples use the NeRFactor synthetic dataset.
  • Relighting Examples: Figures 24–26 show Hotdog, Materials, and Microphone scenes relit under four high-frequency light probes.The Hotdog example uses Poly Haven probes, while the scenes are from the NeRF synthetic dataset.
  • Shading-Term Breakdown: Figure 27 compares shading terms with and without the light regularizer across complex and simple-visibility meshes.For Hotdog and Lego, shadows are more accurately baked into lighting terms; for Chair, Materials, and Microphone, the regularizer may over-contrast light probes but produces smoother kd textures.
Loading 2206.03380v2…