PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove

arXiv:1901.05103v1cs.CV

TL;DR

Deep 3D shape learning is constrained by the complexity of volumetric networks and the irregularity of mesh representations. DeepSDF learns continuous signed distance fields for shape representation and completion, outperforming benchmarked methods across both tasks while using substantially less memory.

  • Problem

    3D learning methods face limited quality, flexibility, and fidelity because volumetric networks are costly and mesh representations have variable vertices and topology.

  • Method

    DeepSDF learns continuous shape-conditioned signed distance fields with neural networks, representing surfaces as zero-level sets and learning shape embeddings through an auto-decoder.

  • Results

    DeepSDF significantly outperforms benchmarked methods across shape representation and completion while representing complex topologies and using significantly less memory than prior state-of-the-art methods.

  • Takeaways & Limitations

    DeepSDF provides an efficient continuous representation supporting complex shapes, closed surfaces, high-quality surface normals, and latent shape-space interpolation.

  • Takeaways & Limitations

    DeepSDF assumes canonical poses, so in-the-wild completion requires explicit SE(3) optimization that increases inference time.

Abstract

from arXiv · show

Computer graphics, 3D computer vision and robotics communities have produced multiple approaches to representing 3D geometry for rendering and reconstruction. These provide trade-offs across fidelity, efficiency and compression capabilities. In this work, we introduce DeepSDF, a learned continuous Signed Distance Function (SDF) representation of a class of shapes that enables high quality shape representation, interpolation and completion from partial and noisy 3D input data. DeepSDF, like its classical counterpart, represents a shape's surface by a continuous volumetric field: the magnitude of a point in the field represents the distance to the surface boundary and the sign indicates whether the region is inside (-) or outside (+) of the shape, hence our representation implicitly encodes a shape's boundary as the zero-level-set of the learned function while explicitly representing the classification of space as being part of the shapes interior or not. While classical SDF's both in analytical or discretized voxel form typically represent the surface of a single shape, DeepSDF can represent an entire class of shapes. Furthermore, we show state-of-the-art performance for learned 3D shape representation and completion while reducing the model size by an order of magnitude compared with previous work.

1. Introduction · 2. Related Work · 3. Modeling SDFs with Neural Networks

DeepSDF introduces a continuous, learned SDF representation that models shape surfaces as zero-level sets of neural networks, addressing limitations of voxel, mesh, and point-based representations. The method supports generative shape modeling and completion while enabling expressive surfaces with complex topologies.

  • 1. Introduction: The approach addresses the computational growth of 3D convolutional networks and the training difficulties of meshes with unknown vertices and arbitrary topology.These challenges have limited the quality, flexibility, and fidelity of deep-learning approaches for 3D data.
  • 1. Introduction: DeepSDF learns a continuous generative SDF field instead of discretizing the field into a regular voxel grid.The representation is described as efficient, expressive, and fully continuous.
  • 1. Introduction: DeepSDF formulates generative shape-conditioned modeling with a continuous implicit surface, uses a probabilistic auto-decoder, and targets shape modeling and completion.The models produce high-quality continuous surfaces with complex topologies and state-of-the-art quantitative results for reconstruction and completion.
  • 2.1. Representations for 3D Shape Learning: Point-, mesh-, and voxel-based representations support 3D learning but have limitations for continuous surfaces with complex topologies, while voxel SDFs are memory-expensive.Mesh quality depends on parameterization and cutting strategies, whereas voxel-based SDFs inherit the cost of discrete grids.
  • 2.2. Representation Learning Techniques: Related representation-learning work includes GANs and decoder-only auto-decoders trained with self-reconstruction loss.These methods have been studied for applications including noise reduction, missing-measurement completion, and fault detection.
  • 2.3. Shape Completion: Classical shape-completion methods fit implicit surfaces for individual shapes, whereas recent data-driven methods encode partial observations and predict complete volumetric shapes.Inputs include occupancy voxels, discrete SDF voxels, depth maps, RGB images, and point clouds.
  • 3. Modeling SDFs with Neural Networks: DeepSDF represents a watertight surface as the zero isosurface of a neural SDF whose sign indicates whether queried points lie inside or outside.The surface can be rendered by raycasting or rasterizing a mesh extracted with Marching Cubes.
  • 3. Modeling SDFs with Neural Networks: Deep neural networks directly regress continuous SDF values from point samples, but approximation precision is limited by sample count and network capacity.The clamp parameter δ trades near-surface detail concentration against larger safe ray-tracing steps.

4. Learning the Latent Space of Shapes

DeepSDF conditions a single continuous SDF network on latent shape codes, representing multiple shapes through a shared decoder and a low-dimensional embedding. Its encoder-less auto-decoder formulation optimizes codes with decoder parameters, supports arbitrary partial observations, and uses probabilistic regularization for generalization.

  • Latent shape representation: A latent vector z is concatenated with query location x so one neural network can model multiple SDFs and represent each surface as fθ(z, x)=0.The resulting shape can be discretized for visualization using raycasting or Marching Cubes.
  • Auto-decoder formulation: The auto-decoder assigns a latent code to each training shape and jointly optimizes those codes with decoder weights, then estimates an optimal code while keeping weights fixed at inference.Unlike an auto-encoder, it directly accepts latent vectors rather than producing them with an encoder.
  • Probabilistic formulation: A zero-mean multivariate-Gaussian prior with spherical covariance σ^2I regularizes codes, helping produce a compact shape manifold and converge to good solutions.The probabilistic formulation also introduces latent-space regularization for improved generalization.
  • Partial observation handling: DeepSDF handles SDF samples of arbitrary size and distribution because gradients with respect to z are computed separately for each sample, enabling partial observations such as depth maps.This avoids requiring test inputs to resemble the training data or preparing partial-shape training examples.

5. Data Preparation

The data preparation pipeline constructs SDF samples from complete synthetic meshes by normalizing each mesh, sampling spatial and surface points, and computing signed distances from nearest oriented surface points.

  • Mesh and sample preparation: Training uses complete synthetic object meshes, such as ShapeNet, to prepare 3D points with corresponding SDF values.SDF values can be computed through distance transforms for watertight real or synthetic shapes.
  • Mesh and sample preparation: Each mesh is normalized to a unit sphere, and 500,000 spatial points are sampled for training.
  • Surface sampling and SDF computation: Virtual cameras densely sample oriented surface points; meshes with too many problematic double-sided triangles are discarded before nearest-surface SDF computation.Surface normals are oriented toward the cameras, and each spatial point receives its SDF from the closest sampled surface point.

6. Results

DeepSDF is evaluated for known-shape representation, unseen-shape reconstruction, partial-shape completion, and latent-space interpolation. Across these experiments, it outperforms or improves on competing representations while producing continuous, interpretable shape embeddings.

  • Experimental overview: The results evaluate DeepSDF’s ability to represent training shapes, reconstruct unseen shapes, complete partial observations, and learn a smooth shape-embedding space.These experiments test geometric detail, generalization, shape priors, and latent-space structure.
  • Known-shape representation: DeepSDF significantly beats OGN and AtlasNet in Chamfer distance when representing known car shapes from restricted-size latent codes.The comparison uses 30,000 points for Chamfer distance; the EMD difference is smaller because 500 points capture less precision.
  • Unknown-shape representation: DeepSDF significantly outperforms AtlasNet across many shape classes and metrics when encoding unknown test-set shapes.AtlasNet handles mostly consistent topologies without holes better than classes such as chairs, where it misses fine details.
  • Shape completion: DeepSDF completes partial shapes by optimizing a latent code from SDF samples and rendering the complete shape using decoder-encoded priors.The scheme is tested on single-view depth observations, which must be provided in the canonical shape frame of reference.
  • Shape completion: Compared with a recent volumetric completion method, DeepSDF produces more visually pleasing and accurate reconstructions from partial observations.The paper reports quantitative and qualitative completion results and also demonstrates completion from noisy point clouds.
  • Latent-space interpolation: Interpolating pairs of latent vectors yields meaningful shapes and common interpretable features, such as chair arms, that vary linearly in the latent space.These results support the completeness and continuity of the learned shape embedding.

7. Conclusion & Future Work

DeepSDF outperforms applicable benchmarked methods for shape representation and completion while representing complex topologies, closed surfaces, and high-quality surface normals. Its main limitations are inference-time optimization for completion and the assumption of canonical pose.

  • DeepSDF significantly outperforms applicable benchmarked methods across shape representation and completion tasks.
  • DeepSDF represents complex topologies and closed surfaces while providing high-quality surface normals.
  • Shape completion takes considerably more inference time because auto-decoding explicitly optimizes over the latent vector.
  • DeepSDF represents complex shapes without discretization errors using significantly less memory than previous state-of-the-art results.These results are shown in Table 1.
  • DeepSDF currently assumes that models are in a canonical pose.

Supplementary … D. DeepSDF Network Design Decisions

The supplementary material extends DeepSDF with noisy-depth shape-completion experiments, network architecture details, and studies of design decisions affecting SDF regression accuracy and scalability.

  • Supplementary: The supplementary material presents quantitative and qualitative results, technical details, and experiments on DeepSDF design tradeoffs.It covers noisy-depth shape completion, architecture details, network design decisions, and comparisons with auto-encoder variants.
  • B. Shape Completion from Noisy Depth Maps: The noisy-depth experiment evaluates completion from partial point clouds generated by consumer depth-camera noise.The simulation follows Kinect V1-style noise by adding zero-mean Gaussian noise to inverse-depth representations of ground-truth depth images.
  • B. Shape Completion from Noisy Depth Maps: Noise standard deviations of 0.01, 0.02, 0.03, and 0.05 are applied to normalized ShapeNet plane models from Dai et al.’s benchmark test set.Because target shapes are normalized to a unit sphere, the inserted noise is described as significant.
  • B. Shape Completion from Noisy Depth Maps: DeepSDF completion Chamfer distance deteriorates approximately linearly as Gaussian noise increases, while the raw perturbed point cloud’s Chamfer distance increases superlinearly.The comparison is made against the ground-truth shape or depth map, respectively.
  • C. Network Architecture: The main-paper DeepSDF architecture uses 8 fully connected layers with weight normalization.Intermediate vectors use ReLU activation and 0.2 dropout, except at the final layer, with a skip connection at the fourth layer.
  • C. Network Architecture: Figure 10 reports mean and median completion Chamfer distances over 85 plane completions as functions of inverse-depth Gaussian noise.The figure also contrasts these completion errors with distances between perturbed and original depth points.
  • D. DeepSDF Network Design Decisions: The design-decision study examines system parameters that affect SDF regression accuracy and provides insight into algorithmic tradeoffs and scalability.The supplied passage states the study’s purpose but does not report specific parameter outcomes.

D.1. Effect of Network Depth on Regression Accuracy · D.2. Effect of Truncation Distance on Regression Accuracy · E. Comparison with Variational and Standard Auto-encoders on MNIST

DeepSDF benefits from latent-vector skip connections and moderate network depth, while larger truncation distances modestly reduce surface accuracy. On MNIST, the auto-decoder provides broadly effective latent-code optimization and reconstructions comparable to or better than auto-encoder alternatives.

  • D.1. Effect of Network Depth on Regression Accuracy: Latent-vector skip connections let training error continue decreasing with depth, whereas the network without skips saturates quickly at 4 layers.The skip-connected input is concatenated to intermediate feature vectors, such as the 4th and 8th layers of a 12-layer network.
  • D.2. Effect of Truncation Distance on Regression Accuracy: Increasing truncation distance from 0.05 to 1.0 moderately worsens surface representation accuracy, as measured by increasing Chamfer distance.The truncation distance controls how far from the surface the network learns a metric SDF.
  • E. Comparison with Variational and Standard Auto-encoders on MNIST: On MNIST, all methods can represent digits in 2D latent spaces, while AD produces a more condensed space and AD- and VAE-learned spaces generate mostly realistic random samples.The comparison uses binary cross-entropy with decoders matched to the same structure and theoretical capacity.
  • E. Comparison with Variational and Standard Auto-encoders on MNIST: AE decode diverges in all tested latent-code dimensions, whereas AD works well across all tested dimensions and VAE encoder optimization improves for higher-dimensional codes.The test error for VAE and AE using learned encoders is consistently minimized across latent-code dimensions.
  • E. Comparison with Variational and Standard Auto-encoders on MNIST: Optimization-based AE decoding performs poorly because the learned AE latent space contains many bad local minima, unlike reconstructions obtained with learned encoders.VAE and AE produce qualitatively good reconstructions when their learned encoders are used.
  • E. Comparison with Variational and Standard Auto-encoders on MNIST: AD reconstructions more closely resemble the actual test digits than VAE decode reconstructions and are qualitatively on par with end-to-end-trained VAE and AE.VAE decode reconstructions are, for the most part, qualitatively close to the originals.

F. Data Preparation Details · G. Training and Testing Details

DeepSDF preparation normalizes meshes, renders them from virtual cameras, and densely samples signed-distance points near surfaces. Training uses small latent initialization, balanced signs, specified optimization settings, and separate latent-code sizes for reconstruction and completion.

  • F. Data Preparation Details: Each shape is normalized to fit a unit sphere with margin, using sphere radius 1/1.03, then rendered from 100 virtual cameras on the sphere.Surface points are gathered by backprojecting depth pixels from the virtual renderings.
  • F. Data Preparation Details: Around 250,000 triangle-area-weighted surface points are sampled and perturbed with mean-zero Gaussian noise of variances 0.0025 and 0.00025.Each surface point produces two spatial samples through perturbation along all xyz axes.
  • F. Data Preparation Details: For approximately 25,000 uniformly sampled points, SDF distances are measured using a KD-tree and signs are determined from normal-vector dot products.The sign uses the dot product between the normal and the vector difference.
  • G. Training and Testing Details: Latent vectors are initialized from N(0, 0.012) to keep similar shapes from diverging in latent space.The authors identify small initialization as important for training.
  • G. Training and Testing Details: Each gradient-descent batch contains half positive and half negative SDF samples for both training and testing.Balancing positive and negative samples is described as crucial.
  • G. Training and Testing Details: The decoder learning rate is 1e-5 * B, latent-vector learning rate is 1e-3, regularization is σ = 10−2, and each shape contributes 16384 SDF samples per batch.Models were trained on 8 Nvidia GPUs for approximately 8 hours over 1000 epochs; latent sizes were 256 for reconstruction and 128 for completion.

H. Full Derivation of Auto-decoder-based DeepSDF Formulation

The auto-decoder DeepSDF formulation treats shape codes and decoder parameters as jointly optimized variables under a probabilistic model. It derives training and inference objectives by factorizing SDF likelihoods and combining reconstruction with latent-code regularization.

  • Training formulation: Each training shape is paired with a latent code initialized from N(0, 0.0012), and the codes are jointly optimized with decoder parameters θ.The auto-decoder has no encoder; each code zi is associated with training data Xi.
  • Likelihood decomposition: Assuming conditional independence of sampled SDF values given a shape code decomposes each shape likelihood into location-conditioned terms pθ(sj|zi; xj).The sampling location xj parameterizes each individual SDF likelihood.
  • Network likelihood: The likelihood is modeled by a fully connected network prediction ˜sj = fθ(zi, xj), with deviations penalized by a loss function; practice uses the clamped L1 cost.An L2 loss corresponds to assuming Gaussian noise on SDF values, whereas the implementation uses the clamped L1 cost.
  • Training objective: A zero-mean multivariate-Gaussian prior with spherical covariance σ2I yields a joint objective minimizing reconstruction and latent-code regularization over θ and all shape codes.More complex priors could also be assumed.
  • Inference: At inference, fixed decoder parameters θ enable MAP estimation of a latent code z from SDF samples of one underlying shape.The factor 1/σ2 balances reconstruction and regularization in the inference objective.

I. Details on Quantitative Evaluations … I.1.2 OGN

The quantitative evaluations convert DeepSDF outputs into meshes at 512^3 resolution, while qualitative renderings can raycast directly against the continuous SDF. DeepSDF and OGN use matched training data and surface-voxel point-cloud comparisons for evaluation.

  • I.1.1 DeepSDF: DeepSDF models were converted into meshes with Marching Cubes at 512^3 resolution for quantitative evaluation.This conversion was used to evaluate the reconstructed geometry as a mesh.
  • I.1.1 DeepSDF: Qualitative renderings often raycast directly against the continuous SDF instead of using finite-resolution Marching Cubes meshes.Direct raycasting can avoid some artifacts produced by Marching Cubes at finite resolution.
  • I.1.1 DeepSDF: DeepSDF experiments representing known or unknown shapes were trained on ShapeNet v2.The training dataset applied to representation experiments was ShapeNet v2.
  • I.1.1 DeepSDF: Shape completion experiments used ShapeNet v1 to match 3D-EPN.The dataset choice was made for consistency with 3D-EPN.
  • I.1.2 OGN: The OGN “shape from id” decoder was trained for 300,000 steps on the same car training set used for DeepSDF.This setup matched OGN’s car training data to DeepSDF’s.
  • I.1.2 OGN: OGN point-based metrics compared groundtruth 256-voxel training data with generated 256-voxel outputs.Both voxel representations were converted into point clouds before metric computation.
  • I.1.2 OGN: For OGN, both voxel grids were converted into point clouds containing only surface voxels, with one point at each voxel center.Surface voxels were defined using connectivity to at least one of six directions.

I.1.3 AtlasNet · I.1.4 3D-EPN · I.2. Metrics

The evaluation compares separately trained AtlasNet models and processed 3D-EPN completion outputs, using point-based and mesh-specific metrics. These metrics differ in applicability, correspondence requirements, sampling protocols, and treatment of mesh geometry and normals.

  • I.1.3 AtlasNet: AtlasNet models were retrained separately for each evaluation using the authors’ code, default parameters, and DeepSDF-matched class splits.The resulting model quality appeared comparable to the original paper’s models.
  • I.1.3 AtlasNet: AtlasNet evaluation sampled evenly from mesh surfaces rather than using only generated vertices, constraining triangle connectivity across potential holes.The vertex-only computation could leave triangles unconstrained and fail to reflect such connections in Chamfer distance.
  • I.1.4 3D-EPN: 3D-EPN completion results were taken in voxelized distance-function format and converted to final meshes by extracting isosurfaces in MATLAB.The extraction followed the procedure described in the 3D-EPN paper.
  • I.2. Metrics: Chamfer and Earth Mover’s metrics apply to points, meshes sampled as points, and voxels sampled as surface-voxel centers, while three additional metrics target meshes.The mesh-specific metrics are mesh accuracy, mesh completion, and mesh cosine similarity.
  • I.2. Metrics: 30,000 points per set were used for normalized, symmetric Chamfer distance, computed efficiently with a KD-tree.Chamfer distance sums nearest-neighbor distances between two point sets; the reported formulation sums both directions.
  • I.2. Metrics: Earth Mover’s distance forms a one-to-one correspondence between equal-sized point sets and favors similarly even point distributions more than Chamfer distance.Unlike Chamfer distance, it requires a bijection between the evaluated points.
  • I.2. Metrics: Mesh accuracy uses 1,000 generated-surface points and measures the minimum distance containing 90% of them relative to the full ground-truth mesh.Mesh completion instead measures the fraction of 1,000 ground-truth points within ∆=0.01 of the generated mesh, while mesh cosine similarity averages nearest-face normal agreement using 2,500 ground-truth points.

J. Additional Results · J.1. Representing Unseen Objects · J.2. Shape Completions

Additional experiments evaluate DeepSDF on unseen test objects and shape completion, showing successful representation through gradient-based code recovery and high-quality completion on synthetic ShapeNet depth images.

  • J.1. Representing Unseen Objects: DeepSDF is evaluated for representing unseen test objects using trained models.These additional results are presented in Figures 20 and 21.
  • J.1. Representing Unseen Objects: The unseen-object comparison adds mesh completion and mesh cosine similarity metrics.These metrics supplement the methods comparison reported in Table 5.
  • J.1. Representing Unseen Objects: DeepSDF’s success indicates that high-quality shapes similar to test shapes exist in its embedding space.This conclusion is drawn from the unseen-object representation results.
  • J.1. Representing Unseen Objects: The latent codes for unseen shapes can be found through simple gradient descent.Gradient-based code recovery is identified as an implication of the task’s success.
  • J.2. Shape Completions: Additional shape completion experiments use unperturbed depth images from the synthetic ShapeNet dataset.The results are shown in Figure 22.
  • J.2. Shape Completions: These ShapeNet experiments demonstrate the quality of the auto-decoder learning scheme.The completion results provide additional evidence for the learning scheme.
  • J.2. Shape Completions: The shape completion results also demonstrate the quality of DeepSDF’s new shape representation.This conclusion is reported for completions from synthetic ShapeNet depth images.
Loading 1901.05103v1…