PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Descriptor Networks for 3D Shape Synthesis and Analysis

Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, Ying Nian Wu

arXiv:1804.00586v1cs.CV

TL;DR

Energy-based modeling of 3D shape data has received relatively little attention. This paper proposes 3D DescriptorNet, a volumetric convolutional energy-based model, and reports realistic shape synthesis and applications to object recovery and super-resolution.

  • Problem

    Despite progress on discriminator and generator models for 3D shapes, comparatively little work has modeled 3D shape data with energy-based models.

    Source: §1.1 · p. 1

  • Method

    3D DescriptorNet models voxelized shapes with a deep convolutional energy-based probability model trained by an analysis-by-synthesis scheme.

    Source: §1.2 · p. 1, §5 · p. 8

  • Results

    Experiments show realistic 3D shape synthesis, while the conditional model is useful for object recovery and super-resolution.

    Source: §1.2 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    DescriptorNet-trained feature maps are useful for 3D object classification, and its cooperatively trained generator carries semantic information about 3D objects.

    Source: §1.2 · p. 2

Abstract

from arXiv · show

This paper proposes a 3D shape descriptor network, which is a deep convolutional energy-based model, for modeling volumetric shape patterns. The maximum likelihood training of the model follows an "analysis by synthesis" scheme and can be interpreted as a mode seeking and mode shifting process. The model can synthesize 3D shape patterns by sampling from the probability distribution via MCMC such as Langevin dynamics. The model can be used to train a 3D generator network via MCMC teaching. The conditional version of the 3D shape descriptor net can be used for 3D object recovery and 3D object super-resolution. Experiments demonstrate that the proposed model can generate realistic 3D shape patterns and can be useful for 3D shape analysis.

1. Introduction

The paper introduces 3D DescriptorNet, an energy-based model that defines a probability density over voxelized shapes using a volumetric ConvNet. It presents analysis-by-synthesis learning and describes uses including shape synthesis, conditional recovery and super-resolution, generator training, and classification.

1.1. Statistical models of 3D shapes

  • Prior 3D shape models include discriminators trained to predict labels and latent-variable generators, whose maximum-likelihood learning faces difficult posterior inference often addressed with variational or adversarial methods.The paper positions its descriptor network as an alternative to 3D-GAN [28] for shape generation.Source: §1.1 · p. 1

1.2. 3D shape descriptor network

  • 3D DescriptorNet defines a probability density over voxelized shapes using features from a bottom-up volumetric ConvNet and learns through analysis by synthesis without an extra inference network or adversarial discriminator.Its parameters guide both learning and sampling within the same model.Source: §1.2 · p. 1
  • The model can generate shape samples through MCMC such as Langevin dynamics, support conditional object recovery and super-resolution, train generators through cooperative MCMC teaching, and provide features useful for classification.The paper reports that cooperatively trained generators carry semantic information about 3D objects and that unsupervised DescriptorNet features are useful for classification.Source: §1.2 · pp. 1–2

1.3. Related work

  • Unlike nonparametric synthesis methods that retrieve and merge database parts, the proposed model learns a parametric probability distribution and uses a bottom-up 3D ConvNet trained by analysis by synthesis.The introduction situates it among deep-learning approaches to 3D shape synthesis and analysis.Source: §1.3 · p. 2

1.4. Contributions

  • The paper proposes 3D DescriptorNet, interprets its learning as mode seeking and mode shifting and its zero-temperature limit as adversarial, and introduces conditional recovery, generative-model metrics, and cooperative generator training.The cooperative scheme is presented as an alternative to adversarial learning for training a 3D generator.Source: §1.4 · p. 2

2. 3D DescriptorNet

The 3D DescriptorNet models volumetric shapes with a convolutional energy-based probability distribution and learns by alternating synthesis of samples with updates toward observed data. Its conditional form supports recovery tasks including inpainting and super-resolution.

2.1. Probability density

  • The model exponentially tilts a reference distribution using a bottom-up 3D volumetric ConvNet; its partition function is intractable, and a uniform reference gives energy E(Y;θ) = −f(Y;θ).The reference may instead be Gaussian white noise.Source: §2.1 · p. 2

2.2. Analysis by synthesis

  • Maximum-likelihood learning minimizes KL divergence from the data distribution and approximates its intractable model expectation with MCMC, such as Langevin dynamics.Langevin dynamics combines energy-gradient descent with Brownian noise that helps chains escape spurious local minima; parallel chains provide synthesized samples for the likelihood gradient.Source: §2.2 · pp. 2–3

2.3. Mode seeking and mode shifting

  • Sampling seeks low-energy modes and lowers V, while learning shifts, creates, or sharpens modes toward observed examples and raises V; the log-likelihood gradient equals the gradient of V.Source: §2.3 · p. 3

2.4. Alternating back-propagation

  • Alternating back-propagation efficiently computes derivatives for Langevin sample revision and parameter updates from synthesized and observed examples.Source: §2.4 · p. 3, §2.2 · p. 3

2.5. Zero temperature limit

  • At temperature T → 0, Langevin noise diminishes and sampling approaches gradient descent; disabling noise later in training often improves synthesis, and the method relates to herding and [1].The zero-temperature algorithm is interpreted adversarially, with local modes approaching those of V; the aim is many low-energy modes to capture diverse observed and unseen examples.Source: §2.5 · pp. 3–4

2.6. Conditional learning for recovery

  • Conditional DescriptorNet supports recovery, including inpainting from visible voxels and super-resolution from low-resolution input, by fixing the observed condition during conditional Langevin sampling.It can be learned from fully observed training examples by maximizing conditional log-likelihood.Source: §2.6 · p. 4

3. Teaching 3D generator net

The section presents MCMC teaching as a way to train a 3D generator from DescriptorNet sampling, making the generator an approximate direct sampler.

3. Teaching 3D generator net

  • A 3D generator network can learn from MCMC sampling of the 3D DescriptorNet and then be used as an approximate direct sampler of that DescriptorNet.Source: §3 · p. 4

3.1. 3D generator model

  • The 3D generator maps independent standard-normal latent factors through a top-down deconvolutional network and adds white noise to produce an object.The latent factors are sampled from their known prior; the network parameters are denoted α.Source: §3.1 · p. 4

3.2. MCMC teaching of 3D generator net

  • In cooperative training [31], generator samples initialize finite-step Langevin dynamics for the DescriptorNet, whose revised samples train the generator using known latents as supervised targets.The DescriptorNet parameters are updated from the revised samples, while generator parameters are updated by gradient descent; known latents eliminate the need for inference.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate 3D DescriptorNet on ModelNet10 synthesis, conditional recovery and super-resolution, generator learning, and classification. Reported results include realistic synthesized shapes and higher classification accuracy than published methods.

4.1. 3D object synthesis

  • ModelNet10 synthesis produces realistic shapes similar to, but not identical to, training examples; single-category models achieve higher softmax class probabilities than baseline models.The softmax probabilities are assigned by a reference network for each shape’s underlying category.Source: §4.1 · p. 5, Table 2 · p. 5, Figure 1 · p. 6

4.2. 3D object recovery

  • For recovery, Langevin sampling updates corrupted voxels while fixing uncorrupted ones; testing corrupts 70% of each shape and measures average per-voxel error on corrupted voxels.The conditional model is learned directly by randomly corrupting fully observed training shapes; Figure 2 shows original, corrupted, and recovered examples.Source: §4.2 · pp. 5–6, Figure 2 · p. 7

4.3. 3D object super-resolution

  • For super-resolution, training pairs downscaled and full-resolution shapes, then samples high-resolution shapes while projecting updates into the downscaling operator’s null space to preserve the low-resolution input.The projection makes super-resolution analogous to inpainting, with low-resolution voxels serving as the fixed observations.Source: §4.3 · p. 7

4.4. Analyzing the learned 3D generator

  • The MCMC-taught generator produces 3D shapes with smooth latent interpolations, mostly plausible intermediate objects, and latent arithmetic for visual concept manipulation.The reported interpolation suggests a smooth low-dimensional manifold embedding of the object distribution.Source: §4.4 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.5. 3D object classification

  • The learned feature maps outperform eight published methods in ModelNet10 classification accuracy.The unsupervised model supplies pooled first- and second-layer features to a multinomial logistic regression classifier.Source: Table 4 · p. 8, §4.5 · p. 8

5. Conclusion

The paper proposes a deep convolutional energy-based 3D DescriptorNet for volumetric object synthesis and a conditional version for object recovery and super-resolution. It frames training as analysis by synthesis and reports realistic 3D shape generation and usefulness for shape analysis.

5. Conclusion

  • The 3D DescriptorNet is a deep convolutional energy-based model trained through an “analysis by synthesis” scheme.Its training can be interpreted as mode seeking and mode shifting, with a zero-temperature adversarial interpretation.Source: §5 · p. 8
  • The model synthesizes volumetric objects, and MCMC teaching can use the 3D DescriptorNet to teach a 3D generator.Source: §5 · p. 8
  • The conditional 3D DescriptorNet targets 3D object recovery and super-resolution; experiments report realistic shape-pattern generation and usefulness for 3D shape analysis.Source: §5 · p. 8

Related papers

Loading 1804.00586v1…