Source-linked AI summary
3D Shape Induction from 2D Views of Multiple Objects
Matheus Gadelha, Subhransu Maji, Rui Wang
TL;DR
The paper asks how to learn distributions of 3D shapes from 2D views of multiple objects without 3D, viewpoint, or annotation data. PrGANs add a differentiable projection module to a GAN so generated shapes are trained through their rendered images, producing useful 3D models and unsupervised shape and viewpoint predictions across several categories.
Problem
The paper investigates learning generative models of 3D shapes from 2D images of unknown objects and viewpoints without 3D or viewpoint information.
Method
PrGANs generate voxelized 3D shapes and viewpoints, render them through a differentiable projection module, and adversarially match the resulting images to real 2D views.
Results
PrGANs produce reasonable 3D shapes across chairs, airplanes, and cars, perform well compared with GANs trained on 3D data, and support unsupervised shape and viewpoint prediction.
Takeaways & Limitations
The learned representation supports novel-view generation, interpolation between shapes, and unsupervised inference of 3D shape and viewpoint from a single image.
Takeaways & Limitations
View-based silhouettes cannot reveal structures hidden by occlusion, voxel grids constrain resolution, and binary silhouettes omit geometric detail from shading.
Abstract
from arXiv · showhide
In this paper we investigate the problem of inducing a distribution over three-dimensional structures given two-dimensional views of multiple objects taken from unknown viewpoints. Our approach called "projective generative adversarial networks" (PrGANs) trains a deep generative model of 3D shapes whose projections match the distributions of the input 2D views. The addition of a projection module allows us to infer the underlying 3D shape distribution without using any 3D, viewpoint information, or annotation during the learning phase. We show that our approach produces 3D shapes of comparable quality to GANs trained on 3D data for a number of shape categories including chairs, airplanes, and cars. Experiments also show that the disentangled representation of 2D shapes into geometry and viewpoint leads to a good generative model of 2D shapes. The key advantage is that our model allows us to predict 3D, viewpoint, and generate novel views from an input image in a completely unsupervised manner.
1. Introduction
The paper addresses unsupervised learning of 3D shape distributions from 2D views of multiple objects with unknown instances and viewpoints. It proposes PrGANs, which match projections of generated 3D shapes to input-image distributions.
- The central problem is learning a generative model of 3D shapes from images of unknown objects taken from unknown views.
- Unknown object identities, viewpoints, and instance counts make structure-from-motion and visual-hull approaches difficult to apply.
- PrGANs augment a GAN generator with a differentiable projection module so generated 3D shapes can be trained against distributions of 2D images.
- The method uses fixed-resolution voxels to represent shape occupancy consistently across instances, while ignoring lighting and material properties.
- PrGANs learn probabilistic distributions for complex categories including chairs, airplanes, and cars, and support unsupervised shape and viewpoint estimation from novel 2D shapes.
2. Related work
Prior approaches recover 3D shape using known viewpoints, photometric cues, motion, deformation models, or aligned 3D data. PrGANs instead target 3D generative modeling directly from 2D images.
- Visual hulls require views from known viewpoints, while photometric stereo assumes fixed viewpoints and known lighting.
- Morphable-model approaches require consistent global correspondences, and non-rigid structure-from-motion can require manually annotated keypoints across instances.
- Single-image methods estimate geometry using priors, recognition networks, or view-generation models with scene or camera information.
- Earlier 3D GAN work learned shape distributions from aligned 3D data, whereas PrGANs learn from 2D image views.
- The PrGAN architecture generates a voxel shape and viewpoint, renders the shape into an image, and uses a 2D discriminator to distinguish generated from real images.
3. Method
PrGANs factorize image generation into 3D shape, viewpoint, and projection components within a GAN. A differentiable renderer converts rotated voxel grids into smooth silhouette images for adversarial training.
- The GAN combines a generator that produces synthetic samples with a discriminator that distinguishes them from real samples.
- PrGANs factorize the 2D image generator into a 3D shape generator, viewpoint generator, and projection module.
- The shape generator maps the first 200 dimensions of z to a 32 × 32 × 32 voxel representation whose values encode occupancy.
- The viewpoint generator maps the final dimension of z to one of eight directions uniformly sampled around the y-axis.
- The projection module rotates the voxel grid according to viewpoint angles using nearest-neighbor sampling before projection.
- The projection module qualitatively matches training views when rendered from sampled viewpoints.
- Projection sums occupancy along orthographic lines of sight and applies exponential falloff, producing a smooth differentiable image whose values approach 1 as occupied voxels increase.
- Training compares rendered images with real images through a 2D convolutional discriminator, using an adaptive strategy to address unequal generator and discriminator parameter counts.
4. Experiments
Experiments evaluate PrGAN as a 2D and 3D shape generator, test sparse-view learning and category breadth, and demonstrate interpolation and unsupervised single-view shape and viewpoint prediction.
- Validation: PrGAN is evaluated against separate 2D-GAN and 3D-GAN baselines using chair images and voxel grids.The 2D-GAN and PrGAN use images, whereas the 3D-GAN is trained on 3D voxel representations.
- Validation: 90.13 versus 88.31 MMD: PrGAN is slightly better than 2D-GAN for generated chair images.MMD is computed between generated samples and training data.
- Validation: 347.55 versus 442.98 MMD: PrGAN performs worse than 3D-GAN on chair voxel-grid generation.The comparison favors 3D-GAN, which receives 3D training data, while PrGAN learns from image views and generates exterior structures more reliably than interior structures.
- View sparsity: PrGAN-generated shapes improve as training views per object increase, while even one view per object yields reasonable shapes.Experiments use 1, 2, 4, and 8 randomly selected views per object.
- Categories: The model generates 3D shapes across airplanes, cars, chairs, vases, and motorbikes, and represents mixed categories without additional supervision.The mixed-category experiment combines airplanes, cars, and motorbikes in one training set.
- Latent structure and prediction: Linear interpolation between latent encodings produces shape transitions for airplane and chair models, while an encoding network predicts shape and viewpoint from one image.The encoding network outputs a viewpoint coordinate and a 200-dimensional shape code that reconstructs a 32^3 voxel grid.
5. Limitations and Future Work
PrGANs have several scope boundaries: view-based inference misses structures hidden from every view, voxel grids constrain resolution, silhouettes limit geometric detail, and real-image use may require learning viewpoint distributions.
- Failure cases: View-based PrGANs cannot recover structures hidden by occlusion from all views, such as concave chair interiors.The model may fill concavities when they do not alter any observed silhouette.
- Higher-resolution models: The current voxel representation generates low-resolution 32^3 shapes because voxel-grid size scales cubically with resolution.Residual architectures and multi-scale reasoning are suggested as possible ways to increase resolution.
- Using multiple cues for shape reasoning: Using only binary silhouettes limits geometric detail, motivating future projection modules that incorporate shading cues and more sophisticated neural rendering.A learned neural renderer could replace the current projection module and support realistically shaded images.
- Learning from real images: Extending PrGANs to real images may fail under non-uniform viewpoint distributions unless viewpoints or camera parameters are learned or provided conditionally.The paper proposes learning viewpoint distributions or using a viewpoint estimator as future directions.
- Conclusion: The framework infers 3D shape distributions from 2D collections and estimates shape and viewpoint from a single image without supervision.The conclusion positions differentiable rendering as applicable to other unobserved scene properties.