PaperlayerView exact revision on arXiv

Source-linked AI summary

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, Zhaopeng Cui

arXiv:1911.13225v2cs.CVcs.GR

TL;DR

Rendering neural implicit SDFs for inverse graphics is computationally demanding, while shape optimization requires differentiable rendering. This paper introduces differentiable sphere tracing and reports accurate shape prediction with strong generalization and robustness to imperfect or partial observations.

  • Problem

    Neural implicit SDF rendering requires many costly network queries, while inverse-graphics-based 3D reasoning needs an efficient, differentiable renderer.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    The paper uses differentiable sphere tracing to render neural implicit SDFs into 2D observations and propagate losses back to optimize 3D shape codes.

    Source: §1 · pp. 1–2

  • Results

    The geometry-based optimization produces more accurate shapes than prior learning-based approaches, generalizes to new datasets, and handles imperfect or partial observations.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The approach supports test-time 3D shape inference from a single depth image or multi-view color images without additional training.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function. Due to the nature of the implicit function, the rendering process requires tremendous function queries, which is particularly problematic when the function is represented as a neural network. We optimize both the forward and backward passes of our rendering layer to make it run efficiently with affordable memory consumption on a commodity graphics card. Our rendering method is fully differentiable such that losses can be directly computed on the rendered 2D observations, and the gradients can be propagated backwards to optimize the 3D geometry. We show that our rendering method can effectively reconstruct accurate 3D shapes from various inputs, such as sparse depth and multi-view images, through inverse optimization. With the geometry based reasoning, our 3D shape prediction methods show excellent generalization capability and robustness against various noises.

1. Introduction

The introduction identifies the lack of differentiable rendering for neural implicit signed distance functions (SDFs) and presents a renderer that supports 2D-supervised inverse optimization of 3D shape.

1. Introduction

  • Inverse-graphics reasoning seeks the visual structure that best explains observations, and 3D geometric reasoning requires an efficient differentiable renderer to simulate observations and propagate their errors.The passage situates this approach in early 3D geometry work.Source: §1 · p. 1
  • Continuous implicit SDFs can encode detailed geometry at arbitrary points, but prior discretized representations lose geometric detail and no differentiable renderer for learning-based continuous SDFs existed.The passage identifies signed-distance-field representation [35] and contrasts it with discretized representations.Source: §1 · p. 1
  • The proposed renderer differentiably produces depth, normals, silhouettes, and other observations from neural implicit SDFs, enabling 2D losses to optimize shape and infer geometry from images or depth.With a pre-trained generative model, inference searches latent-code space for a shape whose renderings match the observations; the authors report accurate predictions across datasets without finetuning and better generalization than purely learning-based approaches.Source: §1 · pp. 1–2
  • The method makes neural SDF rendering tractable by optimizing sphere tracing’s forward and backward passes, reducing queries, computation, and memory while maintaining resolution.It uses coarse-to-fine rendering, faster ray marching, safe convergence criteria, and an approximate gradient reported to have negligible training impact; sphere tracing otherwise requires repeated neural-network queries and recursive computation graphs.Source: §1 · p. 2

2. Related Work

Related work spans 3D shape representations, differentiable rendering across representations, and prediction from 2D observations. This work targets differentiable rendering for implicit SDFs and shape prediction from depth and multi-view imagery.

3D Representation for Shape Learning

  • Implicit continuous SDFs and occupancy functions offer high-frequency detail and high resolution, motivating an efficient differentiable renderer for SDF-based shape learning [35] [30] [4].Voxels have high memory overhead and relatively low resolution; meshes typically have fixed, simple topology, while point clouds lack topology information and make mesh generation non-trivial [53, 6, 47, 40, 13, 56, 9, 46, 48, 12, 23, 21, 38, 39].Source: §2 · p. 2

Differentiable Rendering

  • Prior differentiable renderers address meshes, point clouds and voxels [18] [32], but not continuous implicit functions; related alternatives target occupancy or use an LSTM without clear geometric meaning [27] [45].Liu et al. [27] make approximations and do not exploit implicit-SDF rendering efficiency, while the LSTM renderer [45] is limited to low-resolution images by memory consumption; this method renders implicit-SDF geometry at high resolution and can be applied without training to existing deep learning models.Source: §2 · pp. 2–3

3D Shape Prediction

  • With neural shape priors [35], this renderer predicts accurate 3D shapes by geometric optimization from random codes using depth or multi-view images.Geometry-based multi-view stereo degrades on textureless regions, single-image systems rely heavily on priors and generalize weakly [6] [11], while learned multi-view methods use cross-view constraints [16] [17] [50]; depth-image input is also studied as shape completion [9] [8].Source: §2 · p. 3

3. Differentiable Sphere Tracing

The section introduces a differentiable sphere-tracing renderer for neural implicit SDFs, optimizing its forward and backward passes so rendered observations can supervise 3D geometry. It presents efficient ray marching, ways to convert traced rays into 2D observations, and an approximate gradient method for optimization.

3.1. Preliminaries - Sphere Tracing

  • For each pixel, sphere tracing [14] marches a camera ray by queried signed-distance steps until it hits or approaches the surface.The method updates distance along the ray by the current SDF value and checks convergence against a threshold.Source: §3.1 · p. 3, §3 · p. 3

3.2. Efficient Forward Propagation

  • To reduce neural-network queries, rays start at the unit-sphere intersection, refine coarse-to-fine, use 1.5× distance steps, and stop when neighboring pixels are no longer confused.A dynamic unfinished-ray mask avoids further marching converged rays; the approach addresses millions of queries at VGA resolution.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3. Rendering 2D Observations

  • Traced ray distances yield depth and surface points; finite-difference SDF gradients give normals, while minimum absolute ray SDF produces a differentiable silhouette approximation.Implicit functions can also jointly render color and semantics with the SDF.Source: §3.3 · pp. 4–5

3.4. Approximated Gradient Back-Propagation

  • Because unrolling repeated DeepSDF [35] queries costs substantial memory, back-propagation ignores residual and higher-order gradients and supervises K near-surface samples to optimize shape codes from partial observations.For each sample, the gradient uses only the first term of the stated gradient equation.Source: §3.4 · p. 5

4. Experiments and Results

The experiments evaluate the renderer’s efficiency and gradient propagation, then use a fixed pretrained DeepSDF decoder to optimize 3D shapes from depth or multi-view image observations. They report shape-prediction results, generalization, and rendering of properties beyond depth.

4.1. Rendering Efficiency and Quality

  • At 512 × 512 resolution, the accelerated renderer uses 887K query steps and runs in 0.99s with a maximum of 50 tracing steps; graph pruning reduces memory use, and acceleration does not impair optimization.Dynamic synchronized inference, aggressive marching, and coarse-to-fine rendering all speed up rendering; query steps increase only slightly when the maximum tracing step is raised to 100.Source: §4.1 · p. 5, Figure 4 · p. 5, Figure 5 · p. 5
  • Image-plane losses successfully back-propagate to shape and camera pose: shape loss tracks Chamfer distance, and a random initial camera pose can be optimized toward ground truth.The camera optimization uses depth and binary silhouette observations with a known shape.Source: §4.1 · pp. 5–6, Figure 6 · p. 6
  • The signed-distance stopping threshold affects boundary quality: large thresholds dilate shapes and small thresholds can leave geometry incomplete; the experiments use 5 × 10^-5.The authors report this threshold produces complete shapes with details up to image resolution.Source: §4.1 · p. 6, Figure 7 · p. 6
  • The renderer can produce color images at arbitrary resolutions, viewpoints, and illumination from a 512-dimensional latent code, and can also render properties such as segmentation and material.The compact code is presented as potentially useful for model compression compared with meshes containing vertices, triangles, and texture maps.Source: §4.1 · p. 6, Figure 8 · p. 6

4.2. 3D Shape Prediction

  • Shape prediction uses a fixed pretrained DeepSDF decoder and test-time optimization of its latent code against 2D observations, without additional training.The experiments target generalization while maintaining shape quality.Source: §4.2 · p. 6

4.2.1 3D Shape Prediction from Single Depth Image

  • With sparse depth, our method consistently outperforms DeepSDF even when DeepSDF receives ground-truth normals; adding silhouette supervision further improves accuracy and robustness.For dense depth, the method also performs better than DeepSDF with depth-only supervision, which the authors attribute to adaptive sampling near boundaries and thin structures.Source: Table 2 · p. 7, §4.2.1 · p. 7

4.2.2 3D Shape Prediction from Multiple Images

  • On PMO’s synthetic test set, our method achieves comparable Chamfer Distance from random initialization, although its DeepSDF decoder was not trained or fine-tuned on PMO data.The comparison uses Chamfer Distance in the gt→pred direction on 50 randomly selected PMO test instances.Source: Table 3 · p. 7, §4.2.2 · p. 7
  • Our method generalizes to images with different focal lengths and is unaffected by noisy latent-code initialization, while PMO degrades; on real data, our shapes have higher quality and more correct structures.Both methods perform worse on the challenging real dataset; the method also benefits from good initialization.Source: §4.2.2 · p. 8, Figure 10 · p. 8

5. Conclusion · Appendix

The paper presents differentiable sphere tracing for rendering observations from neural implicit signed distance functions, enabling geometry-based 3D shape prediction from single and multiple views. Experiments support improved accuracy, generalization, and robustness; the appendix adds renderer analysis, implementation details, and qualitative results.

5. Conclusion

  • The proposed differentiable sphere tracing renderer renders depth maps, normals, and silhouettes from neural-network-parameterized implicit signed distance functions.The conclusion describes the renderer as enabling 2D observations from implicit signed distance functions parameterized as a neural network.Source: §5 · p. 8
  • On the PMO test set, the method maintains surface details, while PMO may not effectively optimize shapes; on real images, it produces more reasonable shapes with correct structure.Source: Figure 11 · p. 8, Figure 12 · p. 8
  • The geometry-based optimization algorithm produces more accurate 3D shapes than SOTA, generalizes well to new datasets, and is robust to imperfect or partial observations.The conclusion describes these as findings from extensive experiments.Source: §5 · p. 8
  • The renderer enables geometric reasoning in 3D shape prediction from both single and multiple views, alongside a high-capacity 3D neural representation.Source: §5 · p. 8
  • Promising directions for the renderer include self-supervised learning, joint recovery of other properties with geometry, and neural image rendering.Source: §5 · p. 8

Appendix

  • The appendix provides detailed analysis of the proposed renderer, implementation details, and additional qualitative results.Source: Appendix · p. 11

A. More Analysis on the Design of Differentiable Sphere Tracing … A.3. Differentiable Rendering of Silhouette

The analysis explains how aggressive marching improves convergence and training supervision, how to choose a convergence threshold, and how raywise SDF queries enable differentiable silhouette rendering.

A.1. Benefits of Aggressive Marching

  • With α=1.5, aggressive marching reduces the minimum convergence steps from 52 to 33 for d=1.0, ϵ=5 × 10−5, and θ=10 degrees.The method uses a step size α times the current SDF, with α set to 1.5 by default.Source: §A.1 · p. 11
  • Aggressive marching samples both sides of the surface and more points near it, providing positive and negative SDF supervision that stabilizes training.Unlike regular marching, it can pass through the surface and bounce between inside and outside regions.Source: §A.1 · p. 11, Figure 14 · p. 11

A.2. Convergence Criteria

  • The convergence threshold ϵ bounds rendering noise, but setting it too large increases depth error while setting it too small increases queries and risks un-converged pixels being treated as background.With a fixed maximum tracing-step count, those background pixels can cause erosion.Source: §A.2 · pp. 11–12
  • The proposed threshold makes neighboring ray-front balls nonintersecting; for f=60mm, dmin=10cm, S=32mm, and R=512, ϵ≈0.5 × 10−4m (0.05mm).Source: §A.2 · p. 12, Figure 15 · p. 12

A.3. Differentiable Rendering of Silhouette

  • The soft silhouette is each ray’s minimum absolute SDF query minus ϵ, and thresholding its sign yields the binary foreground or background silhouette.The rendered silhouette is fully differentiable with respect to the implicit SDF.Source: §A.3 · p. 12, Figure 16 · p. 12
  • A silhouette loss against the ground-truth mask back-propagates through the rendered silhouette, while minimum-absolute queries optimize the nearest surface along each ray for smooth shape optimization.The soft silhouette is differentiable with respect to the implicit SDF and camera extrinsic parameters.Source: §A.3 · p. 12
  • For rays missing the unit sphere, the method assigns a soft silhouette equal to the ray’s distance from the origin minus 1.0, preserving differentiability and sign-based background classification.Source: §A.3 · p. 12

A.4. Drawbacks

Aggressive sphere tracing can create artifacts on very thin geometry, and the renderer is also subject to pixel-level aliasing. Anti-aliasing strategies can be applied to mitigate aliasing.

A.4. Drawbacks

  • Aggressive marching can skip both surfaces of very thin geometry, causing rays to march indefinitely and produce artifacts such as small holes.Figure 18 illustrates small artifacts on thin surface areas.Source: §A.4 · p. 12, Figure 18 · p. 13
  • The renderer can exhibit aliasing because it shoots only one ray from each pixel center, but established anti-aliasing strategies can mitigate it.Source: §A.4 · p. 13

B. Implementation Details · B.1. Network Architecture

The implementation follows DeepSDF’s 9-layer, 512-dimensional network architecture; texture re-rendering uses a matching network conditioned on concatenated shape and texture codes. The authors report that advanced sphere tracing enables high-resolution images with limited time overhead despite the heavier architecture.

B.1. Network Architecture

  • The geometry network follows DeepSDF [35], using 9 fully connected layers with 512 dimensions per hidden layer.Source: §B.1 · p. 13
  • For texture re-rendering, concatenated shape and texture codes are fed into a texture network with the same architecture as the geometry network.Source: §B.1 · p. 13
  • Both the shape code and texture code have dimension 256.Source: §B.1 · p. 13
  • DeepSDF’s [35] architecture is heavier than the [45] backbone, which has 4 layers with 256 dimensions each.Source: §B.1 · p. 13
  • The authors report that advanced sphere-tracing strategies produce high-resolution images with limited time-consumption overhead.Source: §B.1 · p. 13

B.2. Implementation of Dynamic Synchronized Inference … C. Rendering Demos

The supplementary sections explain efficient differentiable rendering, image-plane losses, experiment settings, and demonstrations of the renderer’s outputs.

B.2. Implementation of Dynamic Synchronized Inference

  • Each tracing step batches unfinished camera rays for feedforward computation, then maps results back and deactivates rays that converge or leave the unit sphere.A binary flag tracks each ray’s status.Source: §B.2 · p. 13
  • To limit memory use, the renderer minimizes query locations over a detached graph, then re-evaluates the minimum K queries with gradients attached.This enables rendering 2048 × 2048 images on a single GTX-1080Ti.Source: §B.2 · p. 13

B.3. Depth / Normal / Silhouette / Color Loss

  • Depth and normal losses use rendered foreground pixels, photometric error uses commonly visible pixels, and silhouette loss enforces negative foreground and positive background values.Visibility is determined by reprojected-versus-rendered depth, with ∆2 d < 0.001 treated as visible; depth and photometric losses use L1, and normal loss uses negative dot product [10].Source: §B.3 · p. 13

B.4. More Details and Hyperparameters

  • Experiments use PyTorch and Adam at learning rate 1e-2, with K=3 for completion, K=1 for multiview reconstruction, and at most 100 marching steps.Chamfer Distance uses 30k sampled points for depth completion and 10k for multiview reconstruction, is scaled by 1000, and is reported only gt→pred for multiview; α ranges from 1.2 to 1.8, as smaller values slow convergence and larger values can cause holes in thin areas.Source: §B.4 · p. 13
  • Shape completion initializes the latent code to zero and optimizes for 100 iterations, weighting depth, silhouette, and shape-code regularization losses by 10.0, 1.0, and 1.0, respectively [35].Source: §B.4 · pp. 13–14
  • For synthetic PMO [25] reconstruction, each iteration samples and warps eight uniformly distributed views, compares each to its closest next view, and trains on 112 × 112 images for 20 epochs.With 72 views per object, the procedure runs nine iterations per epoch.Source: §B.4 · p. 14
  • For real-world reconstruction, the method uses 20–30 views downsampled to 96 × 128, selects six views per iteration, and jointly optimizes shape code and similarity transformation.The transformation is usually accurate after one or two epochs; photometric loss weight is 5.0 for synthetic and real-world experiments.Source: §B.4 · p. 14

C. Rendering Demos

  • The supplementary video demonstrates high-resolution depth, normals, silhouettes, and RGB rendering across lighting and viewpoints, plus multiview reconstruction and an enlarged texture re-rendering example.The enlarged texture re-rendering demo appears in Fig. 19.Source: C. Rendering Demos · p. 14

D. Additional Experimental Results · D.1. Qualitative Comparison for Shape Completion · D.2. Qualitative Comparison for Multi-view Shape Prediction

The additional qualitative results show stronger shape completion and multi-view shape prediction than the compared baselines, while identifying conditions where the method can fail.

D.1. Qualitative Comparison for Shape Completion

  • For sparse-depth shape completion, our optimized meshes look better than DeepSDF [35], with fewer holes through perspective rendering and online error for inverse optimization, especially from the input view.DeepSDF uses fixed sampling; the comparisons cover sofa, plane, and table inputs at different sparsity levels.Source: §D.1 · p. 14, Figure 20 · p. 16, Figure 21 · p. 17
  • With silhouette information, our method works reasonably well at extreme input-depth sparsity; unlike DeepSDF [35], it does not use surface normals during optimization.DeepSDF is given normals predicted from dense ground-truth depth to make its sampling feasible; normals could potentially further improve our method.Source: §D.1 · p. 14
  • A pretrained shape prior helps our method generate reasonable occluded regions, but completion can fail when the camera view provides insufficient information and the problem is ill-posed.Source: §D.1 · p. 14

D.2. Qualitative Comparison for Multi-view Shape Prediction

  • On the PMO test set, our method produces visually satisfactory shapes from random initializations, while PMO [25] yields low-resolution shapes even with pretrained initialization and often fails from random codes.PMO’s pretrained encoder initialization comes from its training set; its low resolution is attributed to a limited number of vertices.Source: §D.2 · p. 14, Figure 23 · p. 19
  • In PMO comparisons, random-code results can converge to rather similar shapes, particularly for chairs and planes in the second column.Source: §D.2 · p. 14, Figure 23 · p. 19
  • On the real-world multiview chair dataset [5], our method with random initialization produces more reasonable results than PMO [25] while jointly optimizing shape code and similarity transformation.It can fail when foreground or background texture provides insufficient photometric cues, or when similarity transformation is estimated incorrectly.Source: §D.2 · p. 14, Figure 24 · p. 20

Related papers

Loading 1911.13225v2…