PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Jiajun Wu, Chengkai Zhang, Tianfan Xue, William T. Freeman, Joshua B. Tenenbaum

arXiv:1610.07584v2cs.CVcs.LG

TL;DR

The paper addresses generating novel, realistic 3D object shapes and learning useful 3D representations. It introduces 3D-GAN, which maps probabilistic latent vectors to volumetric objects using adversarial and volumetric convolutional modeling; its unsupervised discriminator features achieve strong recognition performance, while the models also support reconstruction and latent-space exploration.

  • Problem

    3D shape generation must produce varied, novel, and realistic objects, but modeling 3D shapes is difficult because their space is higher-dimensional than that of 2D images.

  • Method

    3D-GAN combines volumetric convolutional networks with adversarial modeling to map sampled latent vectors to 3D objects and learn a discriminator-based shape representation without supervision.

  • Results

    The models generate novel, realistic objects, reconstruct 3D objects from images, and yield unsupervised recognition features that outperform other unsupervised methods and approach supervised methods.

  • Takeaways & Limitations

    A probabilistic latent space enables object sampling and exploration, while the discriminator representation supports 3D recognition and related applications.

  • Takeaways & Limitations

    For 3D-VAE-GAN, the variational latent distribution is regularized toward a zero-mean, unit-variance multivariate Gaussian prior.

Abstract

from arXiv · show

We study the problem of 3D object generation. We propose a novel framework, namely 3D Generative Adversarial Network (3D-GAN), which generates 3D objects from a probabilistic space by leveraging recent advances in volumetric convolutional networks and generative adversarial nets. The benefits of our model are three-fold: first, the use of an adversarial criterion, instead of traditional heuristic criteria, enables the generator to capture object structure implicitly and to synthesize high-quality 3D objects; second, the generator establishes a mapping from a low-dimensional probabilistic space to the space of 3D objects, so that we can sample objects without a reference image or CAD models, and explore the 3D object manifold; third, the adversarial discriminator provides a powerful 3D shape descriptor which, learned without supervision, has wide applications in 3D object recognition. Experiments demonstrate that our method generates high-quality 3D objects, and our unsupervisedly learned features achieve impressive performance on 3D object recognition, comparable with those of supervised learning methods.

1 Introduction

3D object generation must produce shapes that are both novel and realistic, while existing approaches often rely on retrieved parts or yield artifacts. The paper proposes adversarial volumetric modeling to address these limitations and supports both generation and recognition.

  • Traditional CAD-library methods synthesize realistic objects by borrowing parts, but their results are not conceptually novel.
  • Deep voxel-based generative approaches learn object representations without explicit part retrieval, but generated objects often contain fragments or holes.
  • 3D-GAN models volumetric objects with an adversarial criterion to generate shapes that are both novel and realistic.
  • The framework samples novel 3D objects from a probabilistic latent space and learns separate generative and discriminative representations without supervision.
  • The generative representation synthesizes realistic objects, while the discriminative representation supports 3D recognition with performance comparable to supervised methods and better than other unsupervised methods.

2 Related Work

3D object understanding and generation is an established graphics and vision problem with extensive prior work. Earlier synthesis methods commonly retrieve and combine shapes or parts from databases, whereas 3D-GAN synthesizes objects without explicitly borrowing repository parts or supervision.

  • 3D object understanding and generation has attracted extensive research in the graphics and vision community.
  • Many traditional shape-synthesis algorithms are nonparametric and create objects by retrieving and combining shapes or parts from databases.
  • 3D-GAN synthesizes objects without explicitly borrowing parts from a repository and requires no supervision during training.

3 Models

The models combine volumetric convolutional networks with adversarial learning to generate 3D voxel objects from latent vectors and to infer latent representations from 2D images. 3D-VAE-GAN adds an image encoder and jointly uses reconstruction, adversarial, and KL-divergence losses.

  • 3 Models: 3D-GAN leverages volumetric convolutional networks and generative adversarial networks for 3D object generation.
  • 3.1 3D-GAN: The generator maps a 200-dimensional probabilistic latent vector to a 64 × 64 × 64 voxelized object, while the discriminator classifies real versus synthesized objects.
  • 3.1 3D-GAN: Training uses binary cross entropy as the adversarial classification loss, with the discriminator distinguishing real objects from generated ones.
  • 3.2 3D-VAE-GAN: 3D-VAE-GAN adds an image encoder that maps a 2D image to a latent representation used by the 3D-GAN generator.
  • 3.2 3D-VAE-GAN: Its objective combines reconstruction, adversarial, and KL-divergence losses, with the KL term pushing the encoder distribution toward the prior used for sampling.
  • 3.2 3D-VAE-GAN: Training 3D-VAE-GAN requires paired 2D images and corresponding 3D models, using rendered shapes across 72 views.

4 Evaluation

The evaluation tests 3D-GAN’s generation quality, learned discriminator features, latent-space behavior, limited-data performance, and single-image reconstruction. Across these settings, the model produces detailed objects, strong unsupervised classification features, and competitive reconstruction results.

  • 3D Object Generation: 3D-GAN generates high-resolution 3D objects with detailed geometries, while low-resolution outputs reveal structure but omit fine details and variations.The comparison uses 64 × 64 × 64 high-resolution voxels and down-sampled 16 × 16 × 16 voxels.
  • 3D Object Generation: Generated objects are similar but not identical to nearest training examples, supporting generalization beyond simple memorization.Nearest neighbors are retrieved from the training set for comparison.
  • 3D Object Classification: 83.3% vs 75.5% on ModelNet40 and 91.0% vs 80.5% on ModelNet10 show that the learned representation outperforms other unsupervised methods by large margins.The comparison is against other unsupervised feature-learning methods.
  • 3D Object Classification: The discriminator representation achieves accuracy higher than some supervised methods and close to state-of-the-art voxel-based supervised approaches.Multi-view CNNs outperform 3D-GAN but require rendered multi-view images and an ImageNet-pretrained model.
  • Limited Training Data: With roughly 25 training samples per class, 3D-GAN achieves comparable ModelNet40 performance to unsupervised methods trained with at least 80 samples per class.This result is reported for limited-training-data evaluation.
  • Single Image 3D Reconstruction: On IKEA single-image reconstruction, the model consistently outperforms previous state-of-the-art voxel-level prediction and other baseline methods across all categories.Evaluation uses average precision at 20 × 20 × 20 resolution with alignment over permutations, flips, and translations up to 10%.

5 Analyzing Learned Representations

The paper analyzes the learned generative and discriminative representations by probing semantic structure in the latent object vector and selectivity in discriminator neurons.

  • The Generative Representation: The 200-dimensional object vector is analyzed through dimension visualization, interpolation, and shape arithmetic.These methods examine individual dimensions, transitions between object vectors, and vector operations for chairs and tables.
  • The Generative Representation: Varying individual latent dimensions reveals semantic properties such as surface thickness and width.Affected voxels are visualized while gradually changing one dimension at a time.
  • The Generative Representation: Interpolating between object vectors produces smooth transitions both within and across object categories.The paper uses these transitions to explore how generated objects change across the latent space.
  • The Generative Representation: Shape arithmetic transfers learned “arm” and “layer” vectors to other chairs and tables, respectively.The figure illustrates vector-based edits across objects within the corresponding categories.
  • The Discriminative Representation: Discriminator neurons are examined using strongly activating objects and guided back-propagation to identify relevant object parts.The visualization covers neurons in the second-to-last convolutional layer and highlights input regions producing activation.
  • The Discriminative Representation: For individual neurons, strongest-activating objects share overall shapes and consistently activating parts reveal semantic knowledge of object parts.These observations indicate selectivity at both the whole-object and part levels.

6 Conclusion

The conclusion presents 3D-GAN for object generation and 3D-VAE-GAN for image-to-3D mapping, alongside latent-space exploration and unsupervised discriminator features for shape classification.

  • 6 Conclusion: 3D-GAN generates novel objects, while 3D-VAE-GAN reconstructs 3D objects from images.The two models address object generation and image-to-3D model mapping, respectively.
  • 6 Conclusion: The unsupervisedly learned GAN discriminator provides an informative 3D feature representation for shape classification.The conclusion reports impressive performance on this task.
  • 6 Conclusion: The paper explores object-vector interpolation, shape arithmetic, and neuron visualization to analyze learned representations.These analyses probe both the latent generative space and discriminator units.

A.1 Network Structure

The appendix specifies convolutional generator, discriminator, and image-encoder architectures for volumetric generation and image-to-latent inference.

  • Generator: The generator uses five fully convolutional layers to map a 200-dimensional input vector to a 64 × 64 × 64 voxel matrix.Its channel counts are {512, 256, 128, 64, 1}, with 4-sized kernels and strides {1, 2, 2, 2, 2}.
  • Generator: The generator places ReLU and batch normalization between convolutional layers and ends with a Sigmoid layer.The output values lie in [0, 1].
  • Discriminator: The discriminator mirrors the generator, accepting a 64 × 64 × 64 matrix and outputting a real number in [0, 1].It contains five volumetric convolution layers with channels {64, 128, 256, 512, 1}.
  • Discriminator: The discriminator uses 4-sized kernels, strides {2, 2, 2, 2, 1}, leaky ReLU layers with parameter 0.2, batch normalization, and a final Sigmoid.These layers form the mirrored volumetric architecture described in the appendix.
  • Image encoder: The image encoder takes a 3 × 256 × 256 image and samples a 200-dimensional vector representing a Gaussian distribution.The 400-dimensional final representation contains 200 dimensions for the mean and 200 for diagonal variance.

A.2 3D Shape Classification

The shape-classification pipeline pools intermediate discriminator responses, concatenates them into a feature vector, and trains linear SVM classifiers on ModelNet40 and ModelNet10.

  • A.2 3D Shape Classification: Responses from the discriminator’s second, third, and fourth convolutional layers are max-pooled and concatenated into a 7,168-dimensional vector.The pooling kernel sizes are {8, 4, 2}, respectively.
  • A.2 3D Shape Classification: A linear SVM uses the concatenated discriminator features for training and classification.The experiments use one-versus-all classifiers, L2 penalty, balanced class weights, and intercept scaling.
  • A.2 3D Shape Classification: For ModelNet40, the linear SVM uses C = 0.07, while ModelNet10 uses C = 0.01.The appendix also reports limited-training-data results on both datasets in Figure A1.

A.3 3D-VAE-GAN Training

Training uses paired 3D shapes and 2D images, samples latent noise, and sequentially updates the discriminator, image encoder, and generator. The encoder objective combines latent-distribution regularization with reconstruction of the corresponding 3D shape.

  • Each training pair contains a 3D shape x_i and its corresponding 2D image y_i, while z_t is sampled from N(0, I) at iteration t.
  • The discriminator D is updated first by minimizing its specified loss function.
  • The image encoder E is updated second by minimizing its specified loss function.
  • The encoder loss combines KL divergence between the predicted latent distribution and N(0, I) with squared reconstruction error ||G(E(y_i)) − x_i||2.
  • The generator G is updated third by minimizing its specified loss function.
Loading 1610.07584v2…