Source-linked AI summary
Unsupervised Learning of 3D Structure from Images
Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, Nicolas Heess
TL;DR
The paper addresses recovering 3D structure from ambiguous 2D observations by introducing generative models and probabilistic inference for 3D representations. It reports high-quality samples, likelihood benchmarks, conditional inference from images, and end-to-end unsupervised learning from 2D images.
Problem
Recovering underlying 3D structure from 2D observations is difficult because the task is ill-posed and inference is intractable.
Method
The paper develops generative models over 3D volumes and meshes, uses probabilistic and conditional inference, and trains models end-to-end with differentiable rendering from 2D images.
Results
The models produce high-quality, multimodal samples, establish likelihood benchmarks across three datasets, and generate plausible 3D shapes from single 2D views.
Takeaways & Limitations
The results demonstrate the feasibility of learning to infer 3D representations from 2D images in a purely unsupervised manner.
Takeaways & Limitations
Volumes create high-dimensional modeling and computational challenges, while standard mesh parameterizations restrict the range of shapes they can represent.
Abstract
from arXiv · showhide
A key goal of computer vision is to recover the underlying 3D structure from 2D observations of the world. In this paper we learn strong deep generative models of 3D structures, and recover these structures from 3D and 2D images via probabilistic inference. We demonstrate high-quality samples and report log-likelihoods on several datasets, including ShapeNet [2], and establish the first benchmarks in the literature. We also show how these models and their inference networks can be trained end-to-end from 2D images. This demonstrates for the first time the feasibility of learning to infer 3D representations of the world in a purely unsupervised manner.
1 Introduction
The paper targets recovery of 3D structure from ambiguous 2D observations, where inference is difficult because many 3D explanations can fit one image and 3D data is scarce. It introduces generative models that learn plausible 3D structures and infer them probabilistically from images.
- Recovering the 3D structure underlying 2D camera observations is a central computer-vision goal.
- A single 2D scene projection is ambiguous because infinitely many 3D structures can produce the same observation.The ambiguity arises from unknown object, camera, and lighting attributes and from self-occlusion.
- Statistical models are needed to distinguish likely 3D structures, but inference remains intractable for mapping pixels, matching views, and handling multimodal representations.
- Ground-truth 3D data is difficult and expensive to collect, limiting dataset size and scope.
- The paper introduces generative models of volumes and meshes and recovers 3D structures from 2D images through probabilistic inference.
2 Conditional Generative Models
The framework models 3D structures with low-dimensional latent codes, generates a volume or mesh, and projects it into the observed-data domain. Conditional inference networks and representation-specific projection operators support training from volumes or images.
- Conditional Generative Models: Given an observed volume or image x and context c, the model infers a 3D representation h using low-dimensional shape and pose codes z.
- 2.1 Architectures: The generative process sequentially transforms independent Gaussian latent variables into refinements of a hidden 3D canvas h_T.The final canvas is treated as the model’s 3D representation before projection to observed data.
- 2.1 Architectures: Inference networks are trained conditionally on context, with volumetric updates using spatial transformers and mesh updates using fully connected networks.
- 2.1 Architectures: A projection operator acts as a renderer, mapping the latent 3D representation into the training data’s domain, such as a volume or image.
- 2.1 Architectures: The framework uses volume-to-volume, volume-to-image neural, or mesh-to-image OpenGL projection according to the representation and available training data.The volume-to-image operator is learned, whereas the OpenGL mesh renderer is fixed.
- 2.1 Architectures: Black-box mesh renderers require gradient estimation, for which the framework uses a REINFORCE-based scheme.
3 Experiments
The experiments evaluate generative 3D models across volumetric datasets and ShapeNet, testing generation, completion, conditional inference, likelihoods, and learning 3D structure from images.
- 3.1 Generating volumes: The model generates sharp, diverse 3D volumes across Primitives, MNIST3D, and ShapeNet while capturing translations, rotations, and fine shape details.ShapeNet generations retain fine details despite greater complexity.
- 3.2 Probabilistic volume completion and denoising: An inference-generation Markov chain completes large missing regions of 3D volumes with high precision.The method alternates inference and generation after occluding half of a volume.
- 3.3 Conditional volume generation: Class-conditional models generate sharp, varied ShapeNet samples that capture plausible object rotations across the evaluated classes.The model conditions generation on a one-hot class encoding.
- 3.4 Performance benchmarking: Increasing generation steps improves likelihoods substantially, while additional context views generally provide smaller gains across the three datasets.The benchmark reports unconditional and three-view likelihood bounds for Primitives, MNIST3D, and ShapeNet, and the generative model outperforms the deterministic baseline even with one view.
- 3.5 Multi-view training: Multi-view training encourages a consistent volumetric representation that supports arbitrary camera rotations from 2D observations.The model uses three fixed context views to reproduce ten simultaneous random views.
- 3.6 Single-view training: A mesh-based model trained with a black-box renderer reconstructs images accurately and infers object extents visible from unobserved camera angles.The result implies correct inference of both mesh and camera in the reported experiment.
4 Discussion
The paper demonstrates efficient probabilistic inference of 3D structure from 2D images, including varying interpretations that reflect ambiguity and reconstructions that recover unseen object extents. It also establishes feasibility of purely unsupervised 3D representation learning through differentiable rendering.
- Conditional training enables efficient inference of posterior distributions over 3D structures given 2D images.
- End-to-end training from 2D images through differentiable renderers demonstrates the feasibility of purely unsupervised 3D representation learning.
- Volume and mesh representations involve a trade-off between volumetric flexibility and mesh compactness, rendering compatibility, and shape expressiveness.Volumes face high-dimensional modeling and computational challenges, whereas standard mesh parameterizations may restrict the shapes they capture.
- Single-view inference produces plausible, varying 3D interpretations that satisfy the observed projection.
- With three depth-image context views, the model generates depth images from ten different viewpoints using an inferred abstract 3D representation.
- Mesh-based reconstruction accurately reproduces observations and infers object extents outside the visible view when rendered from rotated cameras.
A.1 Supplementary related work
The paper situates volumetric models as extensions of deep generative image models and contrasts them with compact mesh representations that better match rendering engines.
- The model extends state-of-the-art deep generative image modeling ideas to volumetric data.
- Volumetric models can combine more easily with continuous projection operators because they use smooth internal representations.
- Mesh representations provide a more compact yet rich space and, with OpenGL, can capture rendering physics more accurately.
A.2 Inference model
The inference model uses an autoregressive structured posterior whose parameters are produced by a deep network, and optimizes a variational objective jointly with the generative model.
- The posterior approximation is autoregressive, modeling q(z_t|z_<t, x, c) conditioned on earlier latent variables, the observation, and context.
- At each inference step, a read operation extracts r_t from x and the previous state, then z_t is sampled from a parameterized Gaussian.
- Inference states use the generative model’s state-transition function, while φ contains the read, mean, and variance parameters.
- The variational loss combines expected reconstruction error with a sum of KL divergences between stepwise posterior and prior distributions.
- The variational and model parameters are optimized jointly by stochastic gradient descent.
A.3 Volumetric Spatial Transformers
Volumetric Spatial Transformers extend spatial attention from images to volumes by applying separable three-dimensional convolutional kernels representing an affine grid transformation.
- Spatial Transformers process an image x using parameters h to produce a transformed output ST(x,h).
- Volumetric Spatial Transformers analogously process an input volume x with parameters h to produce VST(x,h).
- The volumetric operation uses the tensor outer product of three one-dimensional kernels followed by convolution.
- The kernels implement a simple affine transformation of a uniformly covering three-dimensional grid of points.
A.4 Learnable 3D →2D projection operators
The projection operators act as learnable cameras, transforming volumetric canvases before combining 3D and 2D convolutions.
- Learnable projection operators apply an affine transformation to the volumetric canvas cT using a Spatial Transformer.
- The transformed volume is processed with a combination of 3D and 2D convolutions.
- Multiple instances of the projection operator share parameters.
A.5 Stochastic Gradient Estimators for Expectations of Black-Box Functions
The method uses a multi-sample REINFORCE estimator to reduce learning-signal variance when optimizing renders of inferred meshes.
- For each image, the estimator samples K inferred-mesh realizations using small Gaussian noise and computes a render for each.
- Each sample’s learning-signal variance is reduced with a baseline computed from the other K −1 samples.
- The authors report that the estimator works well in practice even for relatively high settings.
A.6 Unconditional generation
The paper demonstrates volumetric generation, completion, and view-conditional 3D inference, with an MCMC procedure whose idealized kernel targets the correct conditional marginal but is approximate in practice.
- A.6 Unconditional generation: The model generates unconditional 3D volumes, with examples shown for Primitives and MNIST3D.
- A.6 Unconditional generation: Volume completion alternates inference and generation to impute missing regions in 3D volumes.
- A.6 Unconditional generation: Class-conditional volumetric generation is demonstrated across all 40 ShapeNet classes.
- A.6 Unconditional generation: View-conditional volumetric models produce samples for Primitives, MNIST3D, and ShapeNet, with varying plausible interpretations of each 2D image.
- A.10 Volume completion with MCMC: The completion procedure fixes observed entries xo and uses a Markov chain over missing entries xu.
- A.10 Volume completion with MCMC: With a sufficiently accurate recognition model, the ideal kernel has p(xu|xo) as its stationary distribution and generates samples from that correct marginal.
- A.10 Volume completion with MCMC: Using the approximate kernel means the stationary distribution need not exactly equal p(xu|xo), although an L1 error bound is provided.