PaperlayerView exact revision on arXiv

Source-linked AI summary

Point Cloud GAN

Chun-Liang Li, Manzil Zaheer, Yang Zhang, Barnabas Poczos, Ruslan Salakhutdinov

arXiv:1810.05795v1cs.LGstat.ML

TL;DR

Existing GAN frameworks do not straightforwardly handle point clouds because their set structure leaves discriminator constraints and hierarchical object-level dependencies inadequately defined. PC-GAN addresses this with hierarchical sampling, posterior inference for hidden variables, and a sandwiching Wasserstein objective; it reports competitive or better results with smaller networks and generalization to unseen data.

  • Problem

    Existing GAN frameworks are designed for fixed-dimensional samples, whereas point clouds are sets of sets with uninformative marginal point distributions.

  • Method

    PC-GAN combines hierarchical Bayesian and implicit-generative ideas with an inference network that estimates unobserved object representations.

  • Results

    PC-GAN achieves competitive or better results than existing methods using smaller networks and generalizes to delicate details and unseen data.

  • Takeaways & Limitations

    The framework can generate arbitrary numbers of i.i.d. points, learn useful point-cloud interpolations, and support image-to-point-cloud generation.

  • Takeaways & Limitations

    The paper evaluates only 3D applications, while higher-dimensional settings with attributes such as RGB colors and velocity vectors remain future work.

Abstract

from arXiv · show

Generative Adversarial Networks (GAN) can achieve promising performance on learning complex data distributions on different types of data. In this paper, we first show a straightforward extension of existing GAN algorithm is not applicable to point clouds, because the constraint required for discriminators is undefined for set data. We propose a two fold modification to GAN algorithm for learning to generate point clouds (PC-GAN). First, we combine ideas from hierarchical Bayesian modeling and implicit generative models by learning a hierarchical and interpretable sampling process. A key component of our method is that we train a posterior inference network for the hidden variables. Second, instead of using only state-of-the-art Wasserstein GAN objective, we propose a sandwiching objective, which results in a tighter Wasserstein distance estimate than the commonly used dual form. Thereby, PC-GAN defines a generic framework that can incorporate many existing GAN algorithms. We validate our claims on ModelNet40 benchmark dataset. Using the distance between generated point clouds and true meshes as metric, we find that PC-GAN trained by the sandwiching objective achieves better results on test data than the existing methods. Moreover, as a byproduct, PC- GAN learns versatile latent representations of point clouds, which can achieve competitive performance with other unsupervised learning algorithms on object recognition task. Lastly, we also provide studies on generating unseen classes of objects and transforming image to point cloud, which demonstrates the compelling generalization capability and potentials of PC-GAN.

1 Introduction

The paper addresses unsupervised generative modeling for 3D point clouds, whose set structure and permutation invariance make them distinct from fixed-dimensional data. It introduces PC-GAN, combining hierarchical sampling and inference with a sandwiching Wasserstein objective for point-cloud generation.

  • Motivation: 3D point clouds are increasingly used because they preserve more information than 2D images and are easier to represent as permutation-invariant sets.Sensors including automotive Lidar, Kinect, and phone-based face-identification systems have made point-cloud collection more accessible.
  • Motivation: Unsupervised generative modeling for 3D data remains less explored than supervised learning.
  • Contributions: PC-GAN learns point clouds through a hierarchical sampling and inference network without explicitly modeling the underlying density function.The framework can generate new point clouds and draw samples from existing point clouds.
  • Contributions: PC-GAN proposes a sandwiching objective that combines upper and lower Wasserstein-distance bounds to obtain a tighter approximation.The framework is generic and can incorporate existing GAN variants.

2 Problem Definition and Difficulty

Point clouds require modeling distributions over sets of points and their latent object-level structure, rather than treating points as independent fixed-dimensional observations. A straightforward GAN extension can match object-level representations while leaving point-level conditionals arbitrary, so additional constraints are needed.

  • Problem formulation: A point cloud is a set X = {x1, ..., xn} of low-dimensional vectors for an object θ, with d usually 3 and n potentially infinite.A dataset contains collections of such sets across multiple objects.
  • Problem formulation: The generative model must sample entirely new sets and generate additional points for a given set.
  • Hierarchical structure: De Finetti-style factorization represents points as conditionally i.i.d. samples from p(x|θ), where θ is an object-level latent representation.The conditional distribution p(x|θ) describes the latent point distribution associated with an object.
  • Difficulty: The marginal point distribution is uninformative because point-cloud generation requires learning the joint distribution over points and object structure.The image analogy likewise requires modeling all pixels jointly within one image.
  • Difficulty: A naive GAN extension that fixes point number and order and uses a DeepSets discriminator lacks the usual IPM guarantees because 1-Lipschitz functions over sets are not well-defined.
  • Difficulty: A generator with shared noise u and per-point noise zi can produce coherent sets, but the natural GAN formulation still fails to guarantee valid set generation.
  • Difficulty: In the counterexample, matching only p(θ) can reach the lower bound −log(4) while leaving the conditional p(x|θ) arbitrary.This creates a bad solution in which object-level matching does not ensure correct point-cloud structure.

3 Proposed Method

PC-GAN learns point-cloud distributions by combining conditional generation with inferred latent descriptors and hierarchical sampling. Its set-aware inference network handles unordered, variable-sized point sets, while the framework can also be interpreted as an encoder-decoder.

  • Inference: PC-GAN trains a conditional generator together with an inference network that encodes latent object information from each point cloud.The inference network Q(X) is learned concurrently with p(x|θ), because the conditioning variable is unobserved.
  • Inference: The generator Gx(z, ψ) uses per-point noise z and a descriptor ψ encoding information about the latent distribution over objects.The descriptor can represent a posterior p(θ|X), allowing generated points to follow a posterior predictive distribution.
  • Inference: The inference network is designed for sets, accepting variable numbers of points in arbitrary order while producing a consistent descriptor ψ.This addresses the central architectural difficulty caused by point-cloud permutation and size variability.
  • Hierarchical Sampling: After training Gx and Q, PC-GAN trains Gθ(u) to model inferred descriptors for hierarchical sampling of new point clouds.The paper notes that joint rather than layer-wise training could further improve performance.
  • Framework: PC-GAN is a conditional distribution-matching framework with learned inference that can also be interpreted as an encoder-decoder formulation.Its distinction from point-cloud autoencoders is deferred to the paper’s comparison discussion.

4 Different Divergences for Matching Point Clouds

The paper develops divergence-based objectives for matching point-cloud distributions, combining upper and lower Wasserstein estimates to obtain a tighter surrogate. It also identifies dimensionality and discriminator-constraint limits that shape where these estimates are practical.

  • Divergences for Point Clouds: Point-cloud GAN training compares generated and true samples through a discriminator, with the divergence determined by the discriminator’s function constraint.The framework discusses 1-Lipschitz, L2-ball, and Sobolev-ball constraints.
  • Divergences for Point Clouds: Outlier points degrade point-cloud object quality because an object requires many sampled points together, unlike image evaluation that can overlook a few bad samples.The paper contrasts point-cloud coherence with image-level quality evaluation.
  • Sandwiching Objective: Wasserstein GAN’s neural-network dual estimate is a lower bound, whereas approximate primal matching provides an upper bound of the true Wasserstein distance.The lower estimate arises because neural networks recover only a subset of the required 1-Lipschitz functions.
  • Sandwiching Objective: The proposed sandwiching objective combines lower- and upper-bound estimates instead of relying on either estimator alone.The solution is formulated as a minimization problem between the two estimates.
  • Sandwiching Objective: Under stated approximation conditions, some sandwiched estimator Wλ has smaller Wasserstein-distance error than both the upper estimator WU and lower estimator WL.The condition includes ϵ2 > ϵ1 > 0 and ϵ1 > ϵ2/3.
  • Implementation Limits: Primal Wasserstein estimation is favorable for low-dimensional point clouds, while accurate estimation for high-dimensional data remains difficult because empirical error scales as O(1/n^1/d).A low-sample-complexity primal form for high-dimensional data, and its combination with sandwiching, are left for future work.
  • Lower-Bound Implementation: Weight clipping can enforce a valid lower-bound estimate when combined with discriminator regularization, while large clipping ranges can improve numerical stability over original weight-clipping WGAN.The paper notes that small clipping ranges limit network capacity and large ranges can otherwise create numerical issues.

5 Related Works

Related work establishes GAN success across several modalities while emphasizing that point-cloud GANs and learned conditioning remain underexplored. Other approaches use autoencoders or permutation-invariant architectures to address related representation and set-processing challenges.

  • GANs for Point Clouds: GANs have demonstrated results on images, speech, text, video, and 3D voxels, but GAN learning for 3D point clouds remains underexplored.The paper positions PC-GAN within this broader success and gap.
  • Autoencoder-Based Approaches: Adversarial autoencoders constrain encoded data toward a normal distribution using a GAN loss, paralleling VAEs while replacing latent-space KL divergence.The paper relates this approach to the primal form of Wasserstein distance.
  • Conditional GANs: Conditional GANs commonly assume observed labels or images, whereas point-cloud conditioning and inference during training remain underexplored.The related work distinguishes supplied conditioning variables from learned inference.
  • Point-Cloud Architectures: Point-cloud deep learning includes voxel and multi-view transformations as well as permutation-invariant operations that process set data directly.Direct set processing avoids the memory concerns associated with transforming point clouds into voxels.

6 Experiments

Experiments evaluate PC-GAN on synthetic circles and ModelNet40, comparing its objectives and reconstruction behavior with AAE. The results examine distribution recovery, coverage, hierarchical generation, latent interpolation, and image-to-point-cloud extension.

  • Synthetic Datasets: PC-GAN recovers both center and radius distributions on synthetic circles, whereas AAE struggles with radius recovery even with a larger decoder.PC-GAN achieves this with fewer parameters, and its model size is independent of the number of generated points.
  • Quantitative Comparison: The sandwiching objective Ws balances distribution distance and coverage, achieving competitive D2F with WU and empirically better coverage than WL.The authors report that Ws captures object details more effectively, including aeroplane tires and chair legs.
  • Quantitative Comparison: PC-GAN with Ws usually obtains lower D2F than AAE while using fewer parameters, although AAE performs better for Guitar and Sofa classes.AAE has worse coverage, which the authors associate with its fixed output size and Chamfer-distance objective.
  • Hierarchical Sampling: Randomly sampled PC-GAN point clouds are smooth, structured, and nearly symmetric, indicating learned object patterns through hierarchical point-wise generation.The hierarchical process first samples an object representation and then samples points forming its point cloud.
  • Understand the Learned Manifold: Latent interpolation generalizes from eight training rotations to unseen rotations, producing a smooth but nonlinear transformation path.The study also presents interpolation between table and chair point clouds using the learned latent representation.
  • Images to Point Cloud: A proof-of-concept image-to-point-cloud extension regresses the learned latent representation from twelve point-cloud views before conditional generation.The authors identify end-to-end training and single-view transformation as future extensions.

7 Conclusion

PC-GAN modifies GANs for point-cloud generation with hierarchical modeling and a sandwiching objective. It supports flexible generation, detailed and unseen-data generalization, and broader point-cloud applications, while the paper evaluates only 3D applications.

  • 7 Conclusion: PC-GAN combines hierarchical Bayesian modeling with implicit generative models and introduces a sandwiching objective for tighter Wasserstein estimates.The objective performs better empirically, while the architecture learns hierarchical point-cloud generation.
  • 7 Conclusion: PC-GAN can generate arbitrarily many i.i.d. points without pre-specifying the point-cloud size.
  • 7 Conclusion: PC-GAN achieves competitive or better results with smaller networks, captures delicate point-cloud details, and generalizes to unseen data.Its point-wise transformations encourage learning object building components rather than copying whole objects.
  • 7 Conclusion: The framework also demonstrates point-cloud interpolation and image-to-point-cloud transformation.
  • 7 Conclusion: The paper focuses on 3D applications, although the framework can naturally generalize to higher dimensions.Future applications may add attributes such as RGB colors or 3D velocity vectors to each point.

A Technical Proof

The proof establishes that a convexly combined estimator can approximate the Wasserstein distance more tightly than either an upper- or lower-bound estimator alone. Under the stated bounds, a suitable mixing coefficient achieves the result.

  • A Technical Proof: The upper estimator WU and lower estimator WL are assumed to bound the Wasserstein distance with errors controlled by ϵ1 and ϵ2.The assumptions require ϵ2 > ϵ1 > 0 and ϵ1 > ϵ2/3.
  • A Technical Proof: The sandwiched estimator Wλ combines WU and WL to obtain a tighter Wasserstein-distance estimate than either estimator alone.
  • A Technical Proof: ∃λ : |Wλ(P, G) − w(P, G)| < min{|WU(P, G) − w(P, G)|, |WL(P, G) − w(P, G)|}.This inequality states that some mixture has smaller absolute error than both individual approximators.
  • A Technical Proof: The proof bounds the mixture's error by ϵ1, the lower bound for the competing estimator errors.
  • A Technical Proof: Choosing λ in the permitted interval, including ϵ2+ϵ1 < λ < 0.5, yields the claimed strict error bound.

B Permutation Equivariance Layers

Permutation Equivariance Layers provide set-data building blocks that preserve permutation structure while supporting deep neural-network composition. The paper uses them with linear layers and nonlinear activations, and identifies PointNet as a special case.

  • B Permutation Equivariance Layers: A Permutation Equivariance Layer is a stackable deep-learning building block designed for set data.
  • B Permutation Equivariance Layers: The layer applies σ to each point together with a pooled summary of the input set, using max pooling or mean pooling.
  • B Permutation Equivariance Layers: In the experiments, σ is a linear layer followed by a nonlinear activation, and PointNet is a special case of this construction.

C.1 Synthetic Data

The synthetic-data setup uses fixed-size batches and 10,000 samples, with compact multilayer networks for inference, generation, and discrimination.

  • C.1 Synthetic Data: The batch size is fixed at 64, and 10,000 samples are used for training and testing.
  • C.1 Synthetic Data: The inference network uses three mean Permutation Equivariance Layers with hidden size 30 and final output size 15.SoftPlus activations are used.
  • C.1 Synthetic Data: The generator is a five-layer MLP and the discriminator is a four-layer MLP, both with hidden size 30.

C.2 ModelNet40

The ModelNet40 experiments preprocess sampled mesh points with normalization and rotation augmentation, while varying network widths and depths across evaluation settings.

  • Data preprocessing: Each object is represented by 10,000 mesh-sampled points, normalized to zero mean per axis and unit global variance.
  • Data preprocessing: Training augments point clouds with uniform rotations of 0, π/8, …, 7π/8 radians on the x-y plane.
  • Model settings: PC-GAN uses a fixed 10-dimensional random noise vector z2 in all experiments.
  • Model architecture: Single-class models use width-128 MLP components, with a four-layer generator and discriminator and three 128-output permutation-equivariance layers in Q.
  • Model architecture: Whole-ModelNet40 models use width-256 MLPs, five layers for Gx and the discriminator, plus five-layer width-256 hierarchical top networks.
  • Baseline settings: The AAE baseline uses latent code sizes of 128 for single-class models and 256 for whole-ModelNet40 models.
Loading 1810.05795v1…