DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Shaohui Liu; Yinda Zhang; Songyou Peng; Boxin Shi; Marc Pollefeys; Zhaopeng Cui

Source-linked AI summary

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui

arXiv:1911.13225v2Nov 29, 2019cs.CVcs.GR

TL;DR

Neural implicit SDF rendering requires many expensive network queries, limiting differentiable inverse graphics. The paper introduces an optimized differentiable sphere tracer with efficient forward and backward passes, and uses it to recover accurate 3D shapes from varied partial observations. The resulting geometry-based optimization generalizes across datasets and remains robust to imperfect or sparse inputs.

Problem
Rendering neural implicit SDFs is computationally and memory intensive because surface location and bounds are unavailable and each ray query requires a neural-network forward pass.
Method
The method combines sphere tracing with coarse-to-fine and aggressive marching, safe convergence criteria, and approximated gradient back-propagation for fully differentiable rendering.
Results
The renderer produces accurate 3D shapes from single depth and multi-view observations, with improved generalization across datasets and robustness to sparse or imperfect inputs.
Takeaways & Limitations
Geometry-based latent optimization connects high-capacity neural implicit shape priors with partial 2D observations without requiring additional training.

Abstract

from arXiv · show

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward passes of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backwards to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises.

1. Introduction

The paper develops an efficient, fully differentiable renderer for neural implicit signed distance functions, enabling inverse-graphics optimization of 3D shape from 2D observations. It addresses the computational and memory costs of sphere tracing while supporting accurate, generalizable shape prediction.

Inverse graphics seeks the visual structure that best explains observations, requiring an accurate differentiable renderer for optimizing 3D geometry.
Continuous implicit SDFs encode accurate geometry but lack the explicit surface location and bounds available in discretized representations.
The proposed renderer maps neural implicit SDFs from latent codes to depth, normals, silhouettes, and other 2D observations from arbitrary viewpoints.
The renderer enables latent-code optimization from multi-view images and single depth images without additional training, with accurate shapes and improved generalization across datasets.The approach uses a pretrained generative model such as DeepSDF and optimizes its latent code at test time.
Coarse-to-fine marching, aggressive step updates, safe convergence criteria, and approximated gradients reduce rendering computation and memory usage.The method optimizes both forward rendering and backward propagation for commodity-level GPUs.

2. Related Work

Prior work studies discretized 3D representations and geometry-based shape prediction, while sphere tracing provides a conventional way to render signed distance fields. These lines of work motivate continuous implicit representations and ray-based rendering.

3D Representation for Shape Learning: Voxel-based shape representations encode occupancy or signed distance values, extending 2D neural operations into 3D geometry.
Sphere tracing marches a ray from each pixel using the current signed distance as its step size until the surface is reached.
3D Shape Prediction: Multi-view stereo methods reconstruct shapes geometrically but can degrade on texture-less regions without prior knowledge.

3. Differentiable Sphere Tracing

The method renders neural implicit SDFs with sphere tracing while reducing the computation and memory costs that make naive tracing impractical. It produces differentiable depth, normals, silhouettes, and other observations for inverse optimization.

3.1. Preliminaries - Sphere Tracing: Sphere tracing marches each camera ray by queried signed-distance values until it reaches or sufficiently approaches the surface.Each step requires evaluating the implicit SDF at the current ray position.
3.1. Preliminaries - Sphere Tracing: Neural-network SDFs make naive sphere tracing prohibitive because every ray query requires a network forward pass and back-propagation graph.A VGA rendering can require millions of network queries.
3.2. Efficient Forward Propagation: The forward pass combines coarse-to-fine tracing, aggressive marching with α = 1.5, and a convergence criterion that stops when the SDF magnitude is below ϵ.Rays begin at one-quarter resolution, split into four after every three steps, and then continue at full resolution.
3.3. Rendering 2D Observations: The traced distance is converted into differentiable depth and surface normals, while a minimum absolute SDF value along each ray provides a differentiable silhouette.Normals use the normalized gradient of the implicit SDF, approximated by neighboring samples.
3.3. Rendering 2D Observations: The renderer can jointly produce additional spatially varying properties, including color, semantics, and material-related observations.These properties are represented as implicit functions and rendered with the SDF.
3.4. Approximated Gradient Back-Propagation: For back-propagation, the method ignores residual-term gradients and uses an approximation that retains the first gradient term because higher-order gradients empirically have less optimization impact.This reduces memory demands while supporting optimization of the latent shape code.

4. Experiments and Results

The experiments evaluate rendering efficiency, differentiability, quality, and geometry-based shape prediction from depth and multi-view images. Results show tractable rendering, effective gradient propagation, accurate completion under sparse observations, and strong generalization.

Rendering Efficiency: 887K query steps render a 512 × 512 image within 0.99s on a single NVIDIA GTX-1080Ti.The optimized backward pass reduces memory usage, while synchronized inference, aggressive marching, and coarse-to-fine rendering accelerate computation.
Back-Propagation Effectiveness: Random initial camera poses can be optimized toward ground-truth poses using gradients from depth and binary silhouette observations.The renderer propagates image-plane errors to camera extrinsic parameters.
Rendering Other Properties: The renderer can produce color images at arbitrary resolution, viewpoints, and illumination from implicit functions encoding signed distance and color.The experiments use a 512-dim latent code learned from textured meshes; semantic and material properties can also be rendered.
3D Shape Prediction from Single Depth Image: With sparse depth observations, the method consistently recovers accurate shapes and outperforms DeepSDF even when DeepSDF receives ground-truth normals.Available silhouettes further improve robustness against sparsity.
3D Shape Prediction from Multiple Images: On PMO’s synthetic test set, the method achieves comparable results from random initialization despite using a DeepSDF decoder not trained or fine-tuned on that set.On real data, it produces higher-quality, structurally correct shapes while PMO produces rough shapes.

5. Conclusion

The paper presents differentiable sphere tracing for neural implicit SDFs, enabling geometric reasoning for 3D shape prediction from single and multiple views. Experiments report accurate shapes, strong generalization, and robustness to imperfect or partial observations.

Differentiable sphere tracing renders depth maps, normals, and silhouettes from neural implicit signed distance functions.
On real-world multi-view images, the method produces more reasonable results with correct structure, while preserving surface details on the PMO test set.
The geometry-based optimization produces more accurate 3D shapes than state-of-the-art methods.
The method generalizes well to new datasets and remains robust to imperfect or partial observations.

A.1. Benefits of Aggressive Marching

Aggressive marching accelerates sphere tracing under shallow ray-surface angles and improves training supervision by sampling both sides of the surface. Differentiable silhouette rendering and carefully chosen convergence thresholds support efficient optimization, with a speed-quality trade-off in threshold selection.

Aggressive Marching: Aggressive marching uses a step size α times the current SDF, with α set to 1.5 by default, to accelerate convergence and stabilize training.
Faster Convergence: 52 to 33 convergence steps: with d = 1.0, ϵ = 5 × 10−5, and θ = 10 degrees, aggressive marching reduces the minimum steps required.
Stable Training: Aggressive marching samples locations behind the surface, providing positive and negative SDF supervision and enabling convergence from both sides.
Convergence Criteria: The convergence threshold ϵ bounds the intersection error, but large values increase rendering noise while small values increase queries and non-convergence risk.
Convergence Criteria: A practical threshold example uses f = 60mm, dmin = 10cm, S = 32mm, and R = 512 to obtain ϵ ≈ 0.5 × 10−4m (0.05mm).
Differentiable Silhouette: Differentiable silhouettes use the minimum absolute SDF along each ray, subtract ϵ for a soft silhouette, and back-propagate errors toward the nearest surface.

A.4. Drawbacks

The aggressive tracing strategy can create artifacts on very thin surfaces, while one center ray per pixel leaves the renderer susceptible to aliasing.

Thin surfaces may develop small holes when one aggressive marching step crosses two surfaces.
The aggressive strategy can leave the ray front end classified as outside the shape, causing continued marching and artifacts in the back part of objects.
One ray shot from each pixel center introduces a potential aliasing effect.

B.1. Network Architecture

The geometry network follows DeepSDF's architecture, using nine fully connected layers with 512-dimensional hidden layers; texture re-rendering uses a matching texture network.

The geometry network consists of 9 fully connected layers, with dimension 512 for each hidden layer.
For texture re-rendering, shape and texture codes are concatenated and passed to a texture network with the same architecture.
Both shape and texture codes have dimension 256.

B.2. Implementation of Dynamic Synchronized Inference

The implementation synchronizes unfinished rays and reduces memory-intensive operations to make differentiable rendering practical. It also defines image-plane losses and reports rendering demonstrations across viewpoints and lighting.

Dynamic synchronized inference: Unfinished camera rays are concatenated and processed in batches, then mapped back to the original image resolution after each sphere-tracing step.Each ray is checked for convergence or exit from the unit sphere, with a binary flag tracking its status.
Memory-efficient backpropagation: Minimum absolute SDF queries are selected using detached computation and then re-evaluated to avoid unaffordable computational-graph memory consumption.Query locations are saved globally, enabling the minimum K queries per ray to receive gradients in a second feedforward pass.
Image-plane losses: Depth and normal losses are computed over the rendered foreground, while multiview photometric error uses pixels visible in both views.Visibility is determined from the difference between reprojected and source-view depth.
Image-plane losses: Rendered silhouettes use the minimum absolute query minus ϵ, with negative foreground and positive background values supervised by a dedicated loss.The formulation makes silhouette rendering differentiable rather than relying on nondifferentiable termination checks.
Experimental configuration: Experiments use Adam with an initial learning rate of 1e-2, sample 30k or 10k points for Chamfer evaluation, and optimize latent codes under dataset-specific settings.Shape completion runs for 100 iterations, while multiview experiments use different view counts and image resolutions for synthetic and real-world data.
Demonstrations: Supplementary demonstrations render high-resolution depth, normals, silhouettes, and RGB images under varied lighting and camera viewpoints.Additional qualitative results include texture rerendering and multiview reconstruction.

D. Additional Experimental Results

Additional experiments compare the method with DeepSDF and PMO on sparse-depth completion and multiview shape prediction. The results show improved qualitative structure, while failures remain under insufficient visual or geometric information.

Shape completion: Under sparse input depth, the optimized mesh visibly outperforms DeepSDF and produces fewer holes, especially from the depth-capture viewpoint.Perspective cameras and online error computation are used instead of DeepSDF’s fixed sampling strategy.
Shape completion: Silhouette supervision allows reasonable completion even when the input depth is extremely sparse, while occluded regions benefit from the pretrained shape prior.The paper notes that completion becomes ill-posed when the camera view provides insufficient information.
Multiview shape prediction: On the PMO test set, random initialization produces more visually satisfactory 3D shapes than PMO, whose outputs remain low-resolution with limited vertices.PMO also fails to generate reasonable shapes in most cases when random initialization is used.
Multiview shape prediction: On real-world chair images, random initialization again yields visually superior outputs to PMO, although insufficient texture causes failures.Incorrect similarity-transform estimation is another reported failure condition.
Texture rerendering: Texture rerendering demonstrations show high-resolution outputs across different resolutions, viewpoints, and illumination conditions.The supplementary material also includes related video demonstrations.