PaperlayerView exact revision on arXiv

Source-linked AI summary

SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images

Chen-Hsuan Lin, Chaoyang Wang, Simon Lucey

arXiv:2010.10505v1cs.CVcs.AIcs.LG

TL;DR

Paired image-shape data are difficult to obtain, and silhouette-supervised reconstruction has generally required multiple annotated views of each object. SDF-SRN learns signed distance functions from single-view images and silhouette distance transforms using differentiable rendering, and reports strong results on synthetic and real-world images. The authors conclude that it can learn compelling reconstructions from static image collections without multi-view supervision.

  • Problem

    Learning dense 3D reconstruction from images lacks practical supervision because paired 3D shapes are difficult to obtain and silhouette-based methods have required multiple views per object.

    Source: §1 · pp. 1–2

  • Method

    SDF-SRN learns implicit signed distance functions from single-view images using silhouette distance transforms and differentiable rendering.

    Source: §1 · p. 2

  • Results

    SDF-SRN significantly outperforms state-of-the-art methods on single-view ShapeNet and PASCAL3D+ without 3D supervision or external 2.5D/3D priors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SDF-SRN demonstrates compelling 3D reconstruction from static image collections without associated multi-view supervision.

    Source: §5 · p. 9

    Obtaining camera poses from practical supervision sources such as 2D keypoints is left to future work.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Dense 3D object reconstruction from a single image has recently witnessed remarkable advances, but supervising neural networks with ground-truth 3D shapes is impractical due to the laborious process of creating paired image-shape datasets. Recent efforts have turned to learning 3D reconstruction without 3D supervision from RGB images with annotated 2D silhouettes, dramatically reducing the cost and effort of annotation. These techniques, however, remain impractical as they still require multi-view annotations of the same object instance during training. As a result, most experimental efforts to date have been limited to synthetic datasets. In this paper, we address this issue and propose SDF-SRN, an approach that requires only a single view of objects at training time, offering greater utility for real-world scenarios. SDF-SRN learns implicit 3D shape representations to handle arbitrary shape topologies that may exist in the datasets. To this end, we derive a novel differentiable rendering formulation for learning signed distance functions (SDF) from 2D silhouettes. Our method outperforms the state of the art under challenging single-view supervision settings on both synthetic and real-world datasets.

1 Introduction

SDF-SRN learns dense 3D reconstruction from single-view images and silhouettes, addressing the difficulty of obtaining 3D shape supervision and multi-view annotations. It uses a signed-distance representation and differentiable rendering, and reports strong results under single-view supervision.

1 Introduction

  • Dense 3D reconstruction without direct 3D supervision remains difficult because paired image-shape data are scarce and synthetic-to-real domain gaps limit practical use.The paper identifies abundant, practical supervision for real-world images as an unresolved need.Source: §1 · p. 1
  • Prior silhouette-supervised methods still require multiple annotated views of each object, while voxel and mesh representations have limitations that motivate implicit representations.Voxels are inefficient for representing sparse surfaces, and meshes are constrained by fixed templates.Source: §1 · pp. 1–2
  • The method learns signed distance functions from images using silhouette distance transforms and builds differentiable rendering on Scene Representation Networks.The distance transform supplies geometric supervision from all image pixels.Source: §1 · p. 2
  • SDF-SRN significantly outperforms state-of-the-art 3D reconstruction methods on ShapeNet trained with single-view supervision, and on natural images from PASCAL3D+ without 3D supervision or externally pretrained 2.5D/3D priors.It also learns high-quality 3D reconstructions from real-world static images with single-view supervision.Source: §1 · p. 2

2 Related Work

Prior work learns 3D shapes from 2D silhouettes, often using discretized representations and multiple views. Implicit representations can model complex topologies, while differentiable neural rendering offers an efficient route to globally consistent 3D geometry.

2 Related Work

  • Silhouette-supervised reconstruction methods use visual-hull projection with voxels, point clouds, or meshes, but typically require multiple views of each object.A prior method began learning from static image collections by jointly optimizing mesh textures and shape deformations.Source: §2 · p. 2
  • Implicit representations model complex topologies at arbitrary resolutions, but earlier unsupervised methods used binary occupancy or required pretrained priors or discretized SDFs.SDF-SRN instead learns deep SDFs without pretrained priors by connecting them geometrically to 2D silhouettes through distance transforms.Source: §2 · p. 2
  • Scene Representation Networks learn ray-tracing steps efficiently, supporting recovery of globally consistent 3D geometry rather than view-dependent depth predictions.Earlier 3D-aware representations encode voxelized features at image pixel locations but do not expose explicit underlying 3D geometric structure.Source: §2 · p. 3

3 Approach

SDF-SRN represents a shape as the zero level set of a continuous implicit function and uses silhouette distance transforms to supervise its geometry. Its rendering and training framework combines learned ray marching, surface-constrained depth, and image-conditioned function parameters.

3 Approach

  • The shape is a continuous implicit function whose zero level set defines a dense 3D surface.The function is an MLP, continuous almost everywhere by construction.Source: §3 · p. 3

3.1 Learning 3D Signed Distance Functions from 2D Silhouettes

  • Distance-transform maps spread silhouette geometry across all image pixels, providing richer shape supervision than binary occupancy maps.The maps contain the same information as binary masks, but distribute it across pixels rather than only where occupancy changes.Source: §3.1 · p. 3
  • For exterior pixels, back-projected distance-transform circles define free-space cones that yield lower bounds on SDF values along rays.The bound is the radius of a sphere centered on the ray and inscribed in the cone; for orthographic cameras it equals D(u).Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Rendering Implicit 3D Surfaces

  • Learned ray marching predicts depths, then sign constraints bracket the nearest surface zero-crossing for differentiable bisection.The final traced point is encouraged to be inside the shape and the preceding point in free space.Source: §3.2 · p. 5

3.3 Implementation

  • An image encoder predicts the implicit-function parameters, and end-to-end training combines SDF, RGB, ray, and eikonal losses.Training uses RGB images, silhouettes, and camera poses; the eikonal penalty encourages a unit-norm SDF gradient.Source: §3.3 · p. 6

4 Experiments

SDF-SRN outperforms the baselines on ShapeNet without multi-view associations and on PASCAL3D+ natural images, while recovering shapes and topologies under single-view supervision.

4.1 ShapeNet Objects

  • Without multi-view associations, SDF-SRN outperforms SoftRas and DVR by a wide margin on ShapeNet and even surpasses depth-supervised DVR.The comparisons use independently treated rendered images, rather than known associations between views of the same CAD model.Source: Figure 4 · p. 7, §4.1 · p. 7
  • With one randomly viewed image per CAD model, SDF-SRN outperforms current 3D-unsupervised methods using 500 training images, improves with more data, and recovers shape topologies.The single-view results use the ShapeNet chair category.Source: Table 2 · p. 7, Figure 5 · p. 7, §4.1 · p. 8
  • ShapeNet chair ablations identify differentiable rendering, importance weighting, and positional encoding as useful, while known-pose latent-code optimization further improves test-time reconstruction.The paper links rendering to resolving shape ambiguity, weighting to finer silhouette alignment, and positional encoding to high-frequency detail.Source: Table 3 · p. 8, §4.1 · p. 8

4.2 Natural Images

  • PASCAL3D+ tests real images with scarce examples, unassociated object instances, and varied textures and lighting; the method assumes weak-perspective cameras and uses bounding boxes to normalize images.The dataset includes airplane, car, and chair images.Source: §4.2 · p. 8
  • Obtaining camera poses from practical supervision sources such as 2D keypoints is left for future work.The authors note this could support learning from larger, easier-to-annotate datasets.Source: §4.2 · p. 8
  • On PASCAL3D+, SDF-SRN consistently outperforms CMR and DVR, though car-side concavity remains an ambiguity.The comparison is trained from scratch on images and silhouettes without pretrained 2.5D/3D shape priors.Source: §4.2 · p. 8, Table 4 · p. 9, Figure 6 · p. 9

5 Conclusion

SDF-SRN learns dense 3D reconstruction from static image collections by learning SDF shape representations from distance-transformed silhouettes with improved differentiable rendering. The conclusion presents larger real-world datasets with practical supervision as a future direction.

5 Conclusion

  • SDF-SRN learns dense 3D reconstruction from static image collections using SDF shape representations learned from distance-transformed silhouettes.The framework uses improved differentiable rendering.Source: §5 · p. 9
  • The framework learns from single-view images without associated multi-view supervision and produces compelling reconstructions on challenging natural images.The conclusion emphasizes that the method does not require knowledge of underlying shape structure or topology.Source: §5 · p. 9
  • The authors identify larger real-world datasets with more practical supervision as a direction for future reconstruction research.Figure 7 depicts color and surface-normal predictions from the reconstructions.Source: §5 · p. 9, Figure 7 · p. 9

Broader Impact

SDF-SRN is presented as a tool for dense geometric reasoning from real-world images, with possible applications spanning transportation, robotics, medicine, and 3D content creation.

Broader Impact

  • Dense geometry from real-world images could support vehicle extent estimation, object manipulation and avoidance, and interpretation of 2D medical imagery in 3D.The passage describes applications involving autonomous vehicles, robots, drones, and health professionals.Source: Broader Impact · p. 9
  • In 3D content creation, the framework could simplify model and animation production and reduce time and money costs in applications such as virtual reality.Source: Broader Impact · p. 9
  • The authors acknowledge potential misuse but judge the possible societal and economic benefits to outweigh the risks.Source: Broader Impact · p. 9

A Derivation of the Proposition

The derivation bounds the SDF at a 3D point using free space implied by a silhouette distance-transform circle. Under orthographic cameras, the bound equals the circle’s radius.

A.1 Formal Proof

  • A distance-transform circle around pixel u lies outside the silhouette, so its back-projected cone is free space and bounds the SDF by the radius of an inscribed sphere.The derivation finds the bound using the sphere’s tangent point on the cone.Source: §A.1 · p. 10
  • The optimal circle parameter depends only on normalized image coordinates, not the queried depth or the variable depth of the tangent point.Substituting it reduces the bound calculation to a linear problem with a solution satisfying the non-negativity constraint.Source: §A.1 · pp. 10–11

A.2 Orthographic cameras

  • Under orthographic cameras, the lower bound b(z; u) equals D(u), the distance-transform value, regardless of queried depth.The back-projected cone approaches an unbounded cylinder as the camera center recedes to negative infinity.Source: §A.2 · p. 11

B Dataset

This section provides further details on the datasets used in the experiments.

  • The section provides additional details about the datasets used in the experiments.Source: B Dataset · p. 11

B.1 ShapeNet [4]

The ShapeNet experiments use a specified split and evaluation protocol, mesh-scanner point clouds as ground truth, and image-color augmentation.

  • The ShapeNet split from Yan et al. uses the intersection of ShapeNet v1 and v2 models because nearly half of the v1 car models were removed in v2.Validation is used for hyperparameter tuning and test for evaluation.Source: §B.1 · p. 11
  • Ground-truth point clouds from Kato et al. include interior details that 3D-unsupervised methods cannot recover, so the experiments use Groueix et al. virtual-mesh-scanner point clouds instead.The point clouds are renormalized to fit a zero-centered unit cube.Source: §B.1 · p. 11
  • ShapeNet single-view augmentation perturbs brightness, contrast, and saturation by ±20% and hue across its full range, without changing image scale.Source: §B.1 · p. 11

B.2 PASCAL3D+ [49]

The PASCAL3D+ evaluation uses ImageNet images, rescales objects using bounding boxes, and registers predictions to ground-truth point clouds.

  • Evaluation uses the ImageNet subset because it has less object occlusion than PASCAL VOC, while occlusion handling remains open for all methods and requires appropriate object-scale normalization.Source: §B.2 · p. 12
  • Natural images are centered and rescaled from tight bounding boxes to 64 × 64 canonical images, with a car-specific height rule and ±20% image-scale augmentation.Source: §B.2 · p. 12
  • Dataset camera parameters are used only for evaluation; predictions are rigidly registered to rescaled ground truth with 50 iterations of ICP to address possible depth-ambiguity misalignment.Source: §B.2 · p. 12

C Architectural and Training Details

SDF-SRN combines shared implicit-function networks, an image encoder, and a depth-update LSTM. Training details include sphere-SDF pretraining, positional encoding, and supplementary reconstruction visualizations.

C Architectural and Training Details

  • The implicit functions share a four-layer, 128-unit MLP with layer normalization and ReLU; shallow heads predict SDF and RGB values.The encoder uses ResNet-18 and dataset-specific fully connected layers, while an LSTM predicts depth updates.Source: C Architectural and Training Details · p. 12
  • Pretraining the hypernetwork to predict a zero-centered sphere (r = 0.5) uses 10K sampled points for 2000 iterations and facilitates early convergence.The authors report that this step helps avoid degenerate solutions.Source: C Architectural and Training Details · pp. 12–13
  • Positional encoding maps 3D inputs to higher dimensions to encourage recovery of high-frequency shape components; the implementation uses L = 6.The encoding is applied to inputs of the implicit functions f, g, and h.Source: C Architectural and Training Details · p. 13

D Supplementary Video

E Additional Results

Additional qualitative comparisons cover ShapeNet multi-view supervision and natural PASCAL3D+ images. SDF-SRN captures category-level semantics and varied topologies in the ShapeNet comparisons.

E Additional Results

  • In ShapeNet multi-view experiments with images treated independently, SDF-SRN captures meaningful within-category shape semantics, while SoftRas and DVR suffer from missing viewpoint correspondences.The comparisons cover airplanes, cars, and chairs.Source: E Additional Results · p. 13
  • SDF-SRN captures varied shape topologies in the ShapeNet qualitative results.Source: E Additional Results · p. 13
  • Additional qualitative results on natural PASCAL3D+ images show airplanes, cars, and chairs.Source: E Additional Results · p. 13
Loading 2010.10505v1…