Source-linked AI summary
SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization
Yue Jiang, Dantong Ji, Zhizhong Han, Matthias Zwicker
TL;DR
Image-based 3D shape optimization requires recovering scene geometry through an inverse rendering process, but practical optimization needs effective parameterizations and gradients. SDFDiff addresses this with differentiable ray-casting of signed distance fields, multi-resolution optimization, and deep-learning integration, demonstrating multi-view reconstruction and state-of-the-art single-view reconstruction results.
Problem
Inverse rendering seeks scene parameters that reproduce input images, but practical use requires a suitable powerful image-formation parameterization and effective optimization gradients.
Method
SDFDiff ray-casts signed distance fields in a differentiable renderer, uses multi-resolution optimization, and integrates with deep-learning frameworks for image-based shape optimization.
Results
The approach supports multi-view and unsupervised single-view 3D reconstruction, more robustly performs multi-view reconstruction than a state-of-the-art point-based differentiable renderer, and achieves state-of-the-art single-view reconstruction results.
Takeaways & Limitations
SDFs provide arbitrary-topology, continuous watertight geometry that supports reconstruction applications and integration with learning methods without 3D supervision.
Takeaways & Limitations
The current multi-view shading model is insufficient for inverse rendering from real camera images because it omits shadows, interreflections, texture, non-diffuse surfaces, and complex illumination.
Abstract
from arXiv · showhide
We propose SDFDiff, a novel approach for image-based shape optimization using differentiable rendering of 3D shapes represented by signed distance functions (SDFs). Compared to other representations, SDFs have the advantage that they can represent shapes with arbitrary topology, and that they guarantee watertight surfaces. We apply our approach to the problem of multi-view 3D reconstruction, where we achieve high reconstruction quality and can capture complex topology of 3D objects. In addition, we employ a multi-resolution strategy to obtain a robust optimization algorithm. We further demonstrate that our SDF-based differentiable renderer can be integrated with deep learning models, which opens up options for learning approaches on 3D objects without 3D supervision. In particular, we apply our method to single-view 3D reconstruction and achieve state-of-the-art results.
1. Introduction
SDFDiff applies differentiable rendering of signed distance fields to image-based 3D shape optimization, exploiting SDFs’ flexible topology and watertight surfaces. It supports multi-view reconstruction and learning-based single-view reconstruction without 3D supervision.
- SDFs represent arbitrary topologies while inherently producing continuous, watertight surfaces.These properties distinguish them from fixed-topology meshes and discontinuous point clouds.
- SDFDiff is a differentiable renderer based on ray-casting SDFs and integrated with a deep learning framework.The integration enables neural networks to address ill-posed inverse problems using image-based optimization.
- The paper introduces SDFDiff, multi-resolution optimization, and applications spanning multi-view and learning-based single-view reconstruction.
- Multi-resolution gradient descent reliably converges to high-quality multi-view reconstructions with detailed geometry and complex topology, even from few views.
- The renderer trains deep neural networks for single-view 3D reconstruction without 3D supervision.The approach recovers accurate 3D shapes with arbitrary topology.
2. Related Work
Prior work uses multiple geometry representations and differentiable rendering strategies for 3D reconstruction. SDFDiff addresses limitations involving topology, surface continuity, resolution, and dependence on 3D supervision.
- SDFs represent arbitrary topologies and watertight surfaces, unlike meshes with constrained topology and point clouds without continuous surfaces.
- Most learning-based 3D reconstruction methods require 3D supervision, while unsupervised alternatives are often limited to specific lighting or orientation settings.
- Differentiable renderers enable 3D reconstruction without 3D supervision by optimizing neural networks using images as training data.
- Voxel-based differentiable renderers are limited to low-resolution voxel grids.
- Sphere-tracing scene representation networks focus on novel-view synthesis rather than view-independent watertight 3D surface reconstruction.
3. Overview
The paper formulates image-based shape optimization as minimizing the discrepancy between target and rendered images. SDFDiff renders SDF geometry and computes derivatives for gradient-based optimization.
- A renderer maps scene parameters Θ to an image, while optimization seeks parameters that reproduce a desired target image.
- The inverse-rendering problem is reformulated as energy minimization using an image loss between target and rendered images.
- The paper focuses on shape optimization and assumes illumination, camera pose, and surface appearance are known.
- The SDF-based renderer takes SDFs and camera pose as inputs, renders an image, and uses automatic differentiation to compute derivatives.
4. Differentiable SDF Rendering
SDFDiff combines sphere tracing with differentiable local shading for SDF surfaces. This avoids differentiating the full tracing process while providing gradients with respect to local SDF samples.
- Discrete SDFs are sampled on regular grids, trilinearly interpolated, and rendered by sphere tracing to locate the zero-level-set surface.
- Only local computations involving eight nearby SDF samples need differentiation; sphere tracing itself need not be differentiable.
- Sphere tracing advances each ray by the interpolated SDF distance, preventing boundary crossing while permitting large steps toward the surface.
- Differentiable shading computes pixel color from local SDF samples, light and camera parameters, the intersection point, and surface normal.
- The approximate intersection p(d0, ..., d7) = s + trilinear(d0, ..., d7; s)v is conservative and cannot cross the true ray intersection.
- An exact intersection would require solving a cubic root problem, whereas the simpler approximation was found more robust in practice.
- The surface normal is obtained from SDF gradients computed by finite differencing and trilinear interpolation.
- The implementation uses CUDA for SDF ray casting and PyTorch automatic differentiation for GPU-based neural-network integration.
5. Multi-view 3D Reconstruction
SDFDiff reconstructs 3D shapes from multiple views by optimizing SDF values with differentiable rendering, known scene parameters, and a coarse-to-fine multi-resolution scheme. It produces detailed reconstructions, with robustness supported by comparisons and parameter studies.
- 5. Multi-view 3D Reconstruction: The method optimizes an SDF from 26 synthetically rendered views with known camera poses, illumination, and surface appearance, initializing the shape as a sphere.The camera poses point from the bounding box’s faces, edges, and vertices toward its center.
- 5.1. Energy Function: SDFDiff minimizes image differences while regularizing SDF values so their spatial gradient has unit magnitude.The image loss is summed over target views, and gradients with respect to SDF values enable optimization.
- 5.2. Iterative Optimization: ADAM optimization selects one view per step, while the multi-resolution scheme progressively increases the SDF grid from 8^3 to 64^3.Grid refinement uses trilinear interpolation, and rendering resolution is matched to the SDF resolution.
- 5.3. Experimental Results: SDFDiff reconstructs geometric details more accurately than DSS and obtains better results than SMVS using only 26 views, although the SMVS comparison is not entirely fair.SMVS uses 1000 views and estimates camera parameters, albedo, and illumination, whereas SDFDiff assumes these are known.
- 5.3. Experimental Results: SDFDiff achieves smaller symmetric Hausdorff distances than the other two approaches for the evaluated torus, bunny, and dragon reconstructions.The distances are reported relative to bounding-box size.
- 5.4. Parameter Study: Lower initial resolution and more multi-resolution stages improve reconstruction quality, while image resolution has little effect and noisy inputs preserve robustness.Single-resolution optimization fails to reconstruct the object successfully under the reported settings.
6. Learning-based Single-view Reconstruction
The single-view system combines a two-stage SDF reconstruction network with image, SDF-validity, and geometry losses. It reconstructs detailed and topologically complex objects, achieving higher 3D IoU than competing methods in most categories.
- Network: The network uses an Encoder-Decoder for coarse SDFs followed by a refiner that improves 3D reconstruction quality.It is trained on all dataset shapes simultaneously.
- Loss Function: The loss combines image reconstruction, SDF-validity, and geometry-smoothing terms, applied locally near predicted surfaces.A distance-based mask suppresses these regularization losses farther from the zero level-set.
- Training Process: Training proceeds sequentially through the Encoder-Decoder, refiner, and jointly trained network, without multi-resolution optimization.The stated sequence first trains the Encoder-Decoder, then fixes it while training the refiner, and finally trains all parts together.
- Qualitative Evaluation: The method reconstructs detailed objects and complicated topologies that a template-mesh baseline cannot capture for chairs.SoftRasterizer relies on a spherical-topology template mesh.
- Quantitative Evaluation: Higher 3D IoU scores than the state-of-the-art approaches are achieved in most of the 13 ShapeNet categories.The authors also report finer details across the categories.
7. Discussion and Limitations
SDFs support arbitrary topology and continuous watertight surfaces, and SDFDiff achieves robust multi-view and state-of-the-art single-view reconstruction results. The current multi-view shading model remains insufficient for inverse rendering from real camera images, while the learning approach outputs discrete grid SDFs.
- Advantages: SDFs represent arbitrary topologies rather than inheriting the topology of a template mesh.This supports shape optimization when the target topology is unknown or complex.
- Advantages: SDFs inherently produce continuous watertight surfaces, unlike point clouds that require post-processing for watertight output.The paper connects this representation property to multi-view and single-view reconstruction applications.
- Results: The approach more robustly performs multi-view reconstruction than a state-of-the-art point-based differentiable renderer and achieves state-of-the-art single-view results.These are the paper’s reported outcomes across its two reconstruction settings.
- Limitations: The current shading model is insufficient for inverse rendering from real camera images because it omits shadows, interreflections, texture, non-diffuse surfaces, and complex illumination.The ray-tracing renderer could be extended to include these effects.
- Limitations: The deep learning approach outputs a discrete SDF on a 3D grid, while continuous neural SDFs could be more memory efficient but computationally expensive.Evaluating a continuous network at every ray-marching step may be too expensive for unsupervised differentiable reconstruction.
8. Conclusion
SDFDiff uses signed-distance-function differentiable rendering for watertight 3D geometry and demonstrates applications in multi-view and neural-network-based single-view reconstruction, including state-of-the-art shape reconstruction.
- SDFDiff renders watertight 3D geometry represented by signed distance functions and applies the approach to multi-view and unsupervised single-view reconstruction.
- Figure 8 presents single-view reconstruction results for airplanes, chairs, and benches.
- The method achieves state-of-the-art results in shape reconstruction.