Learning Representations and Generative Models for 3D Point Clouds

1. Introduction

• Raw 3D encodings are difficult to manipulate directly, and naïve interpolation between two cars in such representations does not yield an intermediate car.Editing objects in those forms can require custom, object-specific parametric models and significant expertise.Source: §1 · p. 1
• The paper introduces deep generative models for point clouds, an area distinct from prior point-cloud architectures focused on recognition or using clouds as intermediate or output representations.The introduction describes those prior architectures as targeting classification and segmentation or using point clouds as intermediate or output representations.Source: §1 · p. 1
• The proposed autoencoder learns compact representations for unseen-shape reconstruction, 3D recognition, semantic operations, interpolation, and shape completion.Figure 1 shows ground-truth test shapes beside their class-specific AE-EMD reconstructions.Source: §1 · p. 1, Figure 1 · p. 2
• Latent GANs trained in a fixed autoencoder space are easier to train than raw GANs and achieve better reconstruction and data-distribution coverage.The paper also reports that multi-class latent GANs perform almost on par with class-specific latent GANs.Source: §1 · pp. 1–2
• The authors find that Chamfer distance fails to identify certain pathological cases when evaluating point-cloud models.The paper also studies metrics for reconstruction objectives and generated-sample evaluation.Source: §1 · p. 2
• The paper introduces fidelity and coverage metrics based on optimal matching between point-cloud collections; coverage can reveal distribution regions missed by a model.Cardinality-based diversity metrics may fail to identify such missed regions.Source: §1 · p. 2

2.1. Point clouds

• A point cloud represents a shape, typically its surface, as N 3D locations; its unordered rows require permutation-invariant encoding and complicate reconstruction comparisons.Raw point clouds lack the grid structure used by standard convolutions.Source: §2.1 · p. 2
• Earth Mover’s distance compares equally sized point sets through a transportation mapping, while Chamfer distance uses nearest-neighbor squared distances and is more efficient to compute.EMD is differentiable almost everywhere; Chamfer distance is differentiable.Source: §2.1 · pp. 2–3

2.2. Fundamental building blocks

• An autoencoder compresses an input into a latent code and decodes that code into a reconstruction; GANs use a generator-discriminator game to synthesize samples.A GMM models multiple Gaussian subpopulations and can generate synthetic samples once fitted.Source: §2.2 · p. 3

• Coverage is the fraction of B's point clouds matched to A, using either CD or EMD to produce COV-CD or COV-EMD.A high score indicates that most of B is roughly represented in A.Source: §3 · p. 3
• MMD matches each cloud in B to its nearest cloud in A and averages distances, measuring fidelity through either MMD-CD or MMD-EMD.Matched examples need not be close for coverage to be high, so MMD measures how faithfully they are represented.Source: §3 · p. 3
• JSD compares point-occupancy distributions across canonical voxels; unlike MMD and coverage, it requires pre-aligned data but is computationally friendlier.The paper reports that JSD correlates well with MMD, making it an efficient alternative for repeated model comparisons.Source: §3 · p. 3
• Small MMD together with large coverage indicates that A captures all modes of B with good fidelity.Coverage and MMD are complementary: coverage alone does not ensure close matches.Source: §3 · p. 3

4. Models for Representation and Generation

• The generative-model study includes a GAN on raw point clouds, a GAN in autoencoder latent space, and Gaussian mixture models in that latent space.Source: §4 · pp. 3–4, §4.2 · p. 4
• Figure 2 illustrates interpolation between point clouds using the learned latent-space representation.Source: Figure 2 · p. 4

4.1. Learning representations of 3D point clouds

• The autoencoder encodes each point independently, pools features into a permutation-invariant latent vector, and decodes it into a 2048 × 3 point cloud.The paper evaluates EMD- and CD-based structural losses and uses a 128-dimensional bottleneck by default.Source: §4.1 · p. 4

4.2. Generative models for Point Clouds

• The raw point-cloud GAN (r-GAN) generates a 2048 × 3 point set from Gaussian noise using five fully connected ReLU layers.Its discriminator follows the autoencoder design, with altered filter sizes and neuron counts, no batch normalization, and leaky ReLUs.Source: §4.2 · p. 4
• The latent-space GAN (l-GAN) trains a generator and discriminator on bottleneck codes from a separately trained autoencoder, then decodes generated codes into point clouds.Its generator has one hidden layer and its discriminator has two, making it simpler than the raw-cloud GAN.Source: §4.2 · p. 4
• The Gaussian mixture models fit distributions to autoencoder latent spaces and generate point clouds by sampling those distributions and decoding the samples.The experiments vary the number of components and use diagonal or full covariance matrices.Source: §4.2 · p. 4

5.1. Representational power of the AE

• AE reconstructions of unseen chairs are nearly as good as training reconstructions, with a small reported generalization gap.The reported chair comparisons use MMD-CD and MMD-EMD for train and test reconstructions.Source: Table 1 · p. 5, §5.1 · p. 5
• Latent interpolation and additive characteristic vectors enable semantic edits, including convertible appearance, chair armrests, and mug-handle removal, as well as smooth morphing.The illustrated armrest edits reflect that chairs with armrests are on average 13% shorter.Source: §5.1 · p. 5, Figure 3 · p. 5
• With minimal adaptation, training on incomplete inputs against complete outputs lets the AE complete severely partial point clouds.Figure 4 presents test-split triplets of partial input, network output, and complete ground truth.Source: §5.1 · p. 5, Figure 4 · p. 6
• The learned AE codes significantly outperform the state of the art on ModelNet10 when classified with a linear SVM.The AE was trained across 55 object categories; the classification experiment used a 512-dimensional bottleneck and gravity-axis random rotations.Source: §5.1 · p. 5

5.2. Evaluating the generative models

• The AE-EMD latent GAN substantially improves coverage and fidelity over the AE-CD latent GAN, while a latent WGAN largely avoids their later mode collapse.Both latent GANs eventually collapse; coverage falls as low as 0.5%.Source: §5.2 · p. 6
• A latent-space GMM achieves quality comparable to a latent WGAN, with fidelity similar to memorized chairs and almost as good coverage.The comparison uses generators selected by validation-set JSD and evaluates them on the chair test split.Source: Table 3 · p. 7, §5.2 · p. 7
• Up to 4× coverage boost and almost 2× improved fidelity for tables: the point-based GMM consistently beats Wu et al. (2016) on four object classes.Wu et al. (2016) trained on all models of each class, while these generators never accessed the test split.Source: Table 4 · p. 7, §5.2 · pp. 7–8
• Multi-class l-WGANs perform similarly in class-specific and shared AE-EMD latent spaces, with little visual-quality sacrifice; generated l-GAN and GMM samples are also reported as high quality.The qualitative samples compare l-GANs with a 32-component GMM.Source: Figure 5 · p. 6, §5.2 · p. 8

6. Related Work

• Earlier deep-learning work used view-based projections, volumetric grids, or graphs; the authors describe their work as the first to study generative and discriminative geometry representations for point clouds.The prior representations were typically voxel grids or view-based image projections.Source: §6 · p. 8
• Training a GMM in an autoencoder’s latent space is related to VAEs, for which prior work documents over-regularization that can harm reconstruction quality.The cited alternatives include gradually increasing the regularizer or using an adversarial autoencoder.Source: §6 · p. 8

7. Conclusion

• The experiments find that a GMM trained in an autoencoder’s fixed latent space is the best generative model for point clouds among those tested.The authors caution that this may not be universal and call for investigation of when latent GMMs match adversarially trained models.Source: §7 · p. 8

A. AE Details

• The autoencoders use independent per-point 1D convolutions with ReLUs for encoding, fully connected ReLU layers for decoding, and Adam for training.Training used an initial learning rate of 0.0005, β1 of 0.9, and batch size 50.Source: A. AE Details · p. 11

A.1. AE used for SVM-based experiments

• For the SVM experiments, the autoencoder used encoder layers with 128, 128, 256, and 512 filters, random gravity-axis rotations, and CD or EMD training.It was trained for 1,000 epochs with CD and 1,100 with EMD.Source: §A.1 · p. 11

A.2. All other AEs

• Other autoencoders used encoder filter sizes 64, 128, 128, 256, and k, with three fully connected decoder layers; training lasted up to 500 epochs for single classes or 1,000 for five classes.The bottleneck size k sets the final encoder filter count.Source: §A.2 · p. 11

A.3. AE regularization

• Among tested bottleneck sizes, k = 128 gave the best test generalization while keeping training reconstruction error minimal; dropout worsened reconstruction, while encoder-only batch normalization slightly improved single-class generalization.The batch-normalization variant also sped up training.Source: §A.3 · p. 11

B. Applications of the Latent Space Representation

• Semantic part editing uses the difference between average latent codes for two subcategories as an offset, then adds that offset to a shape’s code and decodes it.The editing embedding was learned with an AE-EMD trained across all 55 object classes.Source: B. Applications of the Latent Space… · p. 11
• Linear latent interpolation yields morph-like sequences, including cross-category bench-to-sofa transitions, while difference-vector offsets and nearest-neighbor search produce shape analogies.The appendix illustrates analogies and interpolations in Figures 10 and 11.Source: B. Applications of the Latent Space… · pp. 11–12, Figure 10 · p. 11, §A.3 · p. 11

C. Autoencoding Human Forms

• The D-FAUST experiment uses 10,240 point clouds sampled from 10 subjects’ motions, showing unseen-shape reconstructions and decoded linear interpolations between temporal endpoints.Each point cloud contains 4,096 points; the data are split into train, test, and validation sets.Source: C. Autoencoding Human Forms · pp. 12–13, Figure 13 · p. 13

D. Shape Completions

• Completion experiments train class-specific networks on partial and complete ShapeNet pairs, using 2,048-point inputs and 4,096-point complete targets for chairs, tables, and airplanes.The partial scans aggregate views from six limited viewpoints, and the data are split 80%/5%/15%.Source: D. Shape Completions · p. 13

D.1. Architecture

• The completion Abstractor-Predictor uses three independent per-point convolution layers, max pooling, and fully connected layers to output 4,096 points.Its convolution filters are 64, 128, and 1,024; it omits batch normalization.Source: §D.1 · p. 13

D.2. Evaluation

• CD-trained completion outputs are more accurate but less complete than EMD-trained outputs, consistent with CD concentrating predictions near the incomplete input.The comparison reports mean accuracy and coverage across classes at radius ρ = 0.02.Source: §D.2 · p. 13, Table 8 · p. 14
• Figure 14 compares decoded linear interpolations between D-FAUST test-set endpoints with the corresponding ground-truth motion interval.The caption identifies chicken-wings, shoulder shakes, and jumping jacks as captured motion patterns.Source: Figure 14 · p. 14
• Figure 15 shows CD-trained completion triplets as partial input, network output, and complete ground truth, for comparison with EMD-trained examples in Figure 4.The triplets are from the test split.Source: Figure 15 · p. 14

E. SVM Parameters for Auto-encoder Evaluation

• The classification evaluation uses a one-versus-rest linear SVM with an l2 penalty and balanced class weights; dresser/nightstand and flower pot/plant are most confused.The confusion matrix is for ModelNet40 classification using Chamfer loss.Source: E. SVM Parameters for Auto-encoder… · p. 14, Figure 16 · p. 15

F. r-GAN Details

• The r-GAN uses a five-layer fully connected generator and a convolutional discriminator, with 128-dimensional Gaussian noise and Adam training.The discriminator has five 1D-convolutional layers followed by feature-wise max pooling and two fully connected layers.Source: F. r-GAN Details · p. 14

G. l-GAN Details

• The l-GAN uses fully connected generator and discriminator networks; its Wasserstein variant uses a gradient penalty of 10 and five critic steps per generator step.Its training parameters and noise distribution otherwise match those of the r-GAN.Source: G. l-GAN Details · pp. 14–15

H. Model Selection of GANs

• GAN checkpoints are selected by matching generated samples to validation data with JSD or MMD-CD, checking every 100 epochs or every 50 for r-GAN.The appendix also plots generalization estimates and training trends using these metrics.Source: H. Model Selection of GANs · p. 15, Figure 17 · p. 16
• MMD-CD and JSD select models of similar overall quality, with 40 full-covariance Gaussians optimal under MMD-CD and 32 components under JSD.Full covariance performs better than diagonal covariance; 30 or more full-covariance clusters appear sufficient for minimal JSD.Source: H. Model Selection of GANs · p. 15, Table 9 · p. 15, Figure 18 · p. 16

I. Voxel AE Details

• Voxel-based autoencoders are fully convolutional, with 3D Conv-ReLU encoders and Conv-ReLU-transpose decoders; each layer uses stride 2 except the final 32^3 decoder layer, which uses stride 4 and no nonlinearity.Training used Adam for 100 epochs with binary cross-entropy and a learning rate of 0.001.Source: §I · p. 15
• The voxel AEs use convolutional encoders and transpose-convolutional decoders, and their reconstruction quality is comparable to Tatarchenko et al. (2017).They are trained for 100 epochs with Adam and binary cross-entropy.Source: §I · pp. 15–16

J. Memorization Baseline

• The generative models show an approximately 10% coverage drop relative to memorization, while their MMD is almost identical, indicating excellent fidelity.The comparison evaluates memorized training-set samples and generated samples against the chair test split.Source: J. Memorization Baseline · p. 16, Table 11 · p. 17

K. More Comparisons with Wu et al.

• Additional comparisons with Wu et al. report JSD results for two models and test-split Chamfer-based fidelity and coverage across available ShapeNet classes.The comparisons use Wu et al. models publicly available for those classes.Source: K. More Comparisons with Wu et al · p. 16, Table 12 · p. 17, Table 13 · p. 17
• On the chair training split, l-GANs outperform Wu et al. in classification and diversity with fewer epochs, while r-GAN mildly outperforms it in diversity.The comparison also reports synthetic-chair classification scores; the PointNet classifier was trained on ModelNet.Source: K. More Comparisons with Wu et al · p. 17, Table 14 · p. 17

L. Limitations

• Autoencoders may fail to reconstruct uncommon chair geometries faithfully or may omit high-frequency details such as a hole in a chair back.The figure contrasts each input with its reconstruction.Source: L. Limitations · p. 17, Figure 19 · p. 17
• The raw GAN can produce noisy or unrealistic shapes; its chairs remain recognizable, but it struggles more with cars.The paper identifies more robust raw point-cloud GANs as future work.Source: L. Limitations · p. 17, Figure 19 · p. 17
• Shape completion may produce a reasonable shape without preserving some style information from the partial input.The figure illustrates this limitation for the completion network.Source: L. Limitations · p. 17, Figure 21 · p. 18