Source-linked AI summary
Learning Descriptor Networks for 3D Shape Synthesis and Analysis
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu
TL;DR
The paper addresses limited energy-based modeling of voxelized 3D shape data and the inference challenge in generator models. It proposes a deep convolutional 3D DescriptorNet trained by analysis by synthesis, uses MCMC sampling and MCMC teaching for generation, and reports realistic synthesis alongside recovery, super-resolution, and shape-analysis utility.
Problem
Energy-based modeling has been less explored for 3D shape data, while generator models involve challenging posterior-sampling inference during maximum likelihood learning.
Method
The 3D DescriptorNet is a deep convolutional energy-based model defined on voxelized shapes, trained by analysis by synthesis and sampled with MCMC; a generator can learn from these samples through MCMC teaching.
Results
Experiments show realistic 3D shape synthesis, while conditional DescriptorNets support object recovery and super-resolution and learned features support classification.
Takeaways & Limitations
The framework provides a probabilistic alternative for 3D shape generation and supports multiple 3D shape-analysis tasks, including recovery, super-resolution, and classification.
Abstract
from arXiv · showhide
This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns. The maximum likelihood training of the model follows an "analysis by synthesis" scheme and can be interpreted as a mode seeking and mode shifting process. The model can synthesize 3D shape patterns by sampling from the probability distribution via MCMC such as Langevin dynamics. The model can be used to train a 3D generator network via MCMC teaching. The conditional version of the 3D shape descriptor net can be used for 3D object recovery and 3D object super-resolution. Experiments demonstrate that the proposed model can generate realistic 3D shape patterns and can be useful for 3D shape analysis.
1. Introduction
The paper develops a volumetric 3D DescriptorNet as an energy-based alternative to existing discriminator and generator models, with applications to synthesis, recovery, super-resolution, and generator training.
- 1.1. Statistical models of 3D shapes: Existing 3D shape models mainly use discriminators for recognition or latent-variable generators for synthesis, whose maximum-likelihood training requires difficult posterior inference.Variational inference and adversarial learning introduce additional mechanisms to address this inference challenge.
- 1.2. 3D shape descriptor network: The paper introduces the 3D DescriptorNet, a volumetric deep convolutional energy-based model that defines a probability density directly on voxelized shapes.Its bottom-up volumetric ConvNet maps 3D shape signals to features that define the energy or feature statistics.
- 1.2. 3D shape descriptor network: Maximum-likelihood training follows analysis by synthesis without an extra inference network or adversarial discriminator, using shared parameters for learning and sampling.The learning process is interpreted as mode seeking and mode shifting, with an adversarial interpretation in the zero-temperature limit.
- 1.2. 3D shape descriptor network: The learned probability distribution supports 3D synthesis through MCMC sampling, including Langevin dynamics, and conditional sampling for object recovery and super-resolution.The conditional model maps corrupted or low-resolution objects to recovered or high-resolution outputs.
- 1.4. Contributions: A cooperative MCMC-teaching scheme trains a 3D generator as an alternative to adversarial learning, while learned features are also reported as useful for 3D object classification.The generator is taught from samples produced by the descriptor model, and the paper proposes metrics for evaluating 3D generative models.
2. 3D DescriptorNet
The 3D DescriptorNet models volumetric shapes with a convolutional energy-based probability density and trains by alternating MCMC-based mode seeking with parameter-based mode shifting.
- Probability density: The 3D DescriptorNet defines a shape probability density through a bottom-up volumetric ConvNet and an energy-based exponential-tilting formulation.The reference distribution may be Gaussian white noise or uniform over a bounded range.
- Analysis by synthesis: Maximum-likelihood training uses analysis by synthesis, approximating the intractable model expectation with Langevin dynamics samples.Langevin dynamics combines energy-gradient descent with Gaussian noise that helps chains escape spurious local minima.
- Mode seeking and mode shifting: Sampling performs mode seeking by moving synthesized examples toward low-energy or high-probability modes.The sampling process seeks to decrease the value function.
- Mode seeking and mode shifting: Learning performs mode shifting by moving synthesized low-energy modes toward observed examples, while also creating and sharpening modes.This step seeks to increase the value function.
- Training algorithm: The resulting algorithm alternates sampling back-propagation, which revises synthesized examples, with learning back-propagation, which updates model parameters.Both updates use derivatives computed efficiently by back-propagation.
- Conditional learning: Conditional DescriptorNet learning supports recovery and super-resolution by sampling while fixing the observed conditional component.The conditional component can be the visible part of an object or its low-resolution version.
3. Teaching 3D generator net
The paper trains a 3D generator from DescriptorNet samples through cooperative MCMC teaching, turning revised synthesized examples into supervision for approximate direct sampling.
- MCMC teaching: The resulting generator can serve as an approximate direct sampler of the 3D DescriptorNet.The generator learns from how Langevin dynamics changes its initially generated examples.
- 3D generator model: The 3D generator is a nonlinear multilayer generalization of factor analysis that maps latent Gaussian factors to volumetric shapes with added white noise.The generator uses a top-down deconvolutional network with parameters α.
- Cooperative training: Cooperative training uses the generator to initialize finite-step Langevin chains, then updates both the DescriptorNet and generator from the revised examples.The procedure alternates generating initial examples, MCMC revision, DescriptorNet updating, and generator updating.
- MCMC teaching: MCMC teaching uses revised examples from finite-step MCMC to teach the generator, with known latent factors making learning a supervised problem.Because each revised example retains its generator latent factor, latent inference is unnecessary.
4. Experiments
Experiments evaluate 3D DescriptorNet on ModelNet10 for synthesis, recovery, super-resolution, generator analysis, and classification. The model produces realistic novel shapes, supports conditional reconstruction, and yields useful learned features.
- Experimental setup: ModelNet10 experiments cover 10 object categories, with 100–700 training examples per category.Categories include chair, sofa, bathtub, toilet, bed, desk, table, nightstand, dresser, and monitor.
- 3D object synthesis: The synthesized 3D objects are realistic and similar, but not identical, to nearest neighbors from the training set.The qualitative synthesis experiments use chair, bed, sofa, table, dresser, and toilet categories.
- 3D object synthesis: Inception scores compare the mixed-category model with 3D-GAN, 3D ShapeNets, and 3D-VAE, while softmax class probabilities evaluate category-specific synthesis.The Inception score uses a reference 3D multi-view convolutional network trained on ModelNet.
- 3D object recovery: Conditional DescriptorNet recovers randomly corrupted voxels by sampling while fixing the uncorrupted voxels, with experiments using 70% corruption and 90 Langevin steps.Recovery errors are measured as average per-voxel differences on corrupted voxels, and Table 3 reports comparisons across 10 categories.
- 3D object classification: DescriptorNet features outperform eight published baseline methods for 3D object classification on the evaluated dataset.The classifier uses pooled first- and second-layer feature maps concatenated into an 8,100-dimensional vector.
5. Conclusion
The paper proposes 3D DescriptorNet models for volumetric synthesis, recovery, super-resolution, and shape analysis, with training interpreted as analysis by synthesis.
- 3D DescriptorNet models volumetric object synthesis, while its conditional version supports 3D object recovery and super-resolution.
- The deep convolutional energy-based model is trained through an analysis by synthesis scheme that can be interpreted as mode seeking and mode shifting.
- A 3D generator can be trained by the DescriptorNet through MCMC teaching.
- Experiments show that the models generate realistic 3D shape patterns and are useful for 3D shape analysis.