Source-linked AI summary
Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering
Fujun Luan, Shuang Zhao, Kavita Bala, Zhao Dong
TL;DR
Recovering shape and appearance from images remains challenging, while prior approaches often treat geometry and reflectance separately. This paper unifies their optimization with differentiable rendering and reports state-of-the-art reconstruction quality. Its scope is limited by mesh-topology changes and opaque-material modeling.
Problem
Shape and appearance recovery from images is challenging, and prior approaches often treat geometry and reflectance separately or require geometry to be predetermined.
Source: §1 · pp. 1–2, §4.3 · p. 10Method
A unified analysis-by-synthesis pipeline jointly refines geometry and reflectance using a Monte Carlo differentiable renderer, coarse-to-fine optimization, and geometry and reflectance priors.
Source: §1 · p. 2, §5 · p. 10Results
The method achieves state-of-the-art reconstruction quality, with examples spanning smooth and detailed geometries and reflectance.
Source: §1 · p. 2, §4.2 · p. 10Takeaways & Limitations
Unbiased geometric gradients and regularized coarse-to-fine optimization support robust reconstruction of object shape and reflectance.
Source: §5 · p. 10The method has difficulty with complex mesh-topology changes and is limited to opaque materials, excluding appearances such as anisotropic reflection or subsurface scattering.
Source: §4.3 · p. 10
Abstract
from arXiv · showhide
Reconstructing the shape and appearance of real-world objects using measured 2D images has been a long-standing problem in computer vision. In this paper, we introduce a new analysis-by-synthesis technique capable of producing high-quality reconstructions through robust coarse-to-fine optimization and physics-based differentiable rendering. Unlike most previous methods that handle geometry and reflectance largely separately, our method unifies the optimization of both by leveraging image gradients with respect to both object reflectance and geometry. To obtain physically accurate gradient estimates, we develop a new GPU-based Monte Carlo differentiable renderer leveraging recent advances in differentiable rendering theory to offer unbiased gradients while enjoying better performance than existing tools like PyTorch3D and redner. To further improve robustness, we utilize several shape and material priors as well as a coarse-to-fine optimization strategy to reconstruct geometry. We demonstrate that our technique can produce reconstructions with higher quality than previous methods such as COLMAP and Kinect Fusion.
1. Introduction
The paper addresses shape and reflectance recovery from images, a problem often handled through separate methods with limitations for glossy objects or unknown geometry. It unifies their optimization using differentiable rendering, with coarse-to-fine optimization and geometry and reflectance priors.
1. Introduction
- Monte Carlo edge sampling provides unbiased and consistent gradient estimates in a renderer specialized for collocated configurations.The gradients support optimization of mesh vertex positions.Source: §1 · p. 2
- The method jointly optimizes object geometry and spatially varying reflectance through an analysis-by-synthesis pipeline.It uses rendering-loss gradients with respect to both geometry and reflectance.Source: §1 · p. 2
- Coarse-to-fine optimization and geometry and reflectance priors are used to make reconstructions more robust.Source: §1 · p. 2
- The paper evaluates individual pipeline steps and demonstrates the technique on synthetic and real examples.The stated motivation includes practical, high-quality reconstruction using inexpensive handheld acquisition setups.Source: §1 · p. 2
2. Related work
Prior work addresses shape, reflectance, and their joint recovery through different acquisition and optimization strategies. The paper positions unified analysis-by-synthesis optimization against these approaches and highlights geometry-gradient estimation as a challenge in differentiable rendering.
2. Related work
- MVS reconstructs geometry from feature correspondences across multiple views, while Shape from Shading and Photometric Stereo infer surface information from image intensities or changing illumination.Shape from Shading has difficulties with illumination changes, non-diffuse reflectance, and textureless surfaces; Photometric Stereo typically recovers no reflectance beyond diffuse albedo.Source: §2 · p. 2
- Traditional SVBRDF acquisition often uses dense measurements, while newer sparse-input approaches do not readily generalize to complex shapes.Some methods estimate plausible SVBRDFs for flat objects using few images.Source: §2 · p. 2
- Prior joint-estimation methods use varied strategies, while many rely on MVS or alternate shape and reflectance optimization, offering little to no guarantee on reconstruction quality.Examples include plane sweeping, apparent-normal optimization, alternating optimization, a hand-held sensor rig, and separate geometry and reflectance stages.Source: §2 · p. 2
- Monte Carlo edge sampling provides unbiased geometry-gradient estimates, whereas approximated gradient methods can introduce inaccuracies that degrade results.Source: §2 · pp. 2–3
3. Our method
The method jointly fits mesh geometry and spatially varying Disney-BRDF reflectance to calibrated images through analysis-by-synthesis. It combines collocated-light differentiable rendering with regularized, coarse-to-fine optimization.
3.1. Forward and differentiable rendering
- Geometry gradients combine differentiated pixel-interior integrals with boundary integrals over projected silhouette discontinuities, estimated using Monte Carlo edge sampling.The renderer handles interior and boundary contributions in separate passes and provides unbiased gradient estimates.Source: Figure 2 · p. 3, §3.1 · pp. 4–5
3. Our method
- Mesh vertices and UVs plus Disney-BRDF diffuse albedo, specular albedo, and roughness maps are jointly optimized to match calibrated images under collocated lighting.Source: §3.1 · p. 3, §3 · p. 3, §3.2 · p. 5
3.2. Analysis-by-synthesis optimization
- Adam minimizes rendered-versus-target image loss plus regularization, starting from an initial geometry and constant-valued reflectance textures.Initial geometry can come from MVS or Kinect Fusion.Source: §3.2 · pp. 5–6
3.3. Regularization
- Material regularization correlates diffuse and specular albedos and smooths roughness, while mesh terms penalize Laplacian variation, adjacent-face normal differences, and long edges.Source: §3.3 · p. 7
3.4. Improving robustness
- To address fixed-topology sensitivity and update-induced mesh artifacts, staged remeshing and upsampling are paired with El Topo vertex motion that prevents self-intersections.Source: §3.4 · pp. 7–8
4. Results
The evaluations report cleaner reconstructions than COLMAP, Kinect Fusion, and Nam et al., alongside detailed geometry and reflectance recovery from synthetic and real images. Renderer comparisons, optimization studies, and stated failure cases qualify those results.
4. Results
- The implementation uses 500–1000 iterations and takes 0.5–2 hours per example.The renderer uses C++ with CUDA 11 and OptiX 7.1; the optimization pipeline is implemented in PyTorch.Source: §4 · p. 8
4.1. Evaluations and comparisons
- The specialized renderer outperforms SoftRas, PyTorch3D, Mitsuba 2, and Redner without biased gradients, while faster Nvdiffrast uses approximated gradients.In identical shape-optimization setups, biased gradients produced artifacts or blurry geometry.Source: §4.1 · p. 8
- Removing the Laplacian loss yields bumpy geometry, removing normal and edge-length losses yields artifacts, and omitting coarse-to-fine optimization traps the shape in a local optimum.Source: §4.1 · p. 8
- Without diffuse/specular correlation, diffuse colors are baked into specular albedo; without roughness smoothing, noisy roughness creates highlight artifacts.Source: §4.1 · p. 8
- With 10 input images, reconstruction is highly under-constrained, whereas 50 or more produce novel-view renderings that closely match ground truth.Source: §4.1 · p. 8
- Against COLMAP and Kinect Fusion, the method produces higher-quality results from crude synthetic initial geometry; against Nam et al., it recovers cleaner geometry and appearance.Nam et al.’s reconstructions show bumpy geometry and inaccurate highlights.Source: §4.1 · pp. 8–9, Figure 8 · p. 9
4.2. Reconstruction results
- Reconstructions recover smooth and detailed geometry from 50 synthetic input images and detailed geometry and reflectance from 100 real photographs per example.Detailed structures emerge directly in the mesh, without normal or displacement mapping.Source: §4.2 · pp. 9–10
4.3. Discussion and Analysis
- The authors attribute quality gains to edge-sampling gradients, manifold-preserving surface evolution, and coarse-to-fine optimization with regularization; complex topology changes and non-opaque appearances remain limitations.The simplified Disney BRDF does not handle anisotropic reflection or subsurface scattering.Source: §4.3 · p. 10
5. Conclusion
The paper presents a unified pipeline that jointly refines object geometry and reflectance, using a custom Monte Carlo differentiable renderer and coarse-to-fine optimization with priors. The authors also identify limits involving capture configuration, mesh topology, and input-image count.
5. Conclusion
- The custom renderer produces unbiased geometric gradients and outperforms several existing tools, while the pipeline uses coarse-to-fine optimization and geometry and reflectance priors.The renderer’s geometric gradients are described as crucial for high-quality reconstructions.Source: §5 · p. 10
- The technique is specialized to a collocated camera and point light, and can have difficulty capturing materials with strong retroreflection.The authors state that generalizing to other configurations would be useful future work.Source: §5 · p. 10
- Mesh topology refinement relies on remeshing between coarse-to-fine stages; robust and flexible topology optimization remains future work.Source: §5 · p. 10
- More advanced geometry or appearance regularization may enable high-quality reconstructions from fewer input images.Source: §5 · p. 10