Source-linked AI summary
Learning Gradient Fields for Shape Generation
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, Bharath Hariharan
TL;DR
Generating point clouds requires modeling distributions of 3D points concentrated near shape surfaces while supporting arbitrary sampling and varying shapes. The paper learns a shape-conditioned log-density gradient and uses stochastic gradient ascent on an unnormalized density to generate points. It reports state-of-the-art auto-encoding and generation, while also extracting implicit surfaces without ground-truth meshes.
Problem
Point-cloud generation needs a generative model that moves sampled points toward high-density shape regions and supports arbitrary-sized point sets and varying topologies.
Method
The method learns a shape-conditioned gradient field of the log-density with denoising score matching, then uses stochastic gradient ascent to sample points and extract surfaces.
Results
The model achieves state-of-the-art performance in point cloud auto-encoding and generation while enabling high-quality implicit-surface extraction without ground-truth surfaces during training.
Takeaways & Limitations
A learned gradient field provides one representation for generating point clouds and extracting the underlying shape surface.
Takeaways & Limitations
Implicit-surface extraction can also produce zero-gradient points at non-unique nearest-surface locations, although this is seldom a practical problem and can be filtered with a second partial-derivative test.
Abstract
from arXiv · showhide
In this work, we propose a novel technique to generate shapes from point cloud data. A point cloud can be viewed as samples from a distribution of 3D points whose density is concentrated near the surface of the shape. Point cloud generation thus amounts to moving randomly sampled points to high-density areas. We generate point clouds by performing stochastic gradient ascent on an unnormalized probability density, thereby moving sampled points toward the high-likelihood regions. Our model directly predicts the gradient of the log density field and can be trained with a simple objective adapted from score-based generative models. We show that our method can reach state-of-the-art performance for point cloud auto-encoding and generation, while also allowing for extraction of a high-quality implicit surface. Code is available at https://github.com/RuojinCai/ShapeGF.
1 Introduction
The paper models point clouds as samples from 3D point distributions and learns their log-density gradient, enabling arbitrary-size generation through stochastic gradient ascent. It reports state-of-the-art point cloud auto-encoding and generation, plus implicit-surface extraction without ground-truth meshes.
- Motivation: Point clouds are useful across recognition, manipulation, synthesis, scanning, and LiDAR applications, motivating a powerful generative prior.Such a prior can densify sparse clouds, create training data, complete scanned objects, or synthesize new shapes.
- Problem: Modeling shapes as distributions of 3D points supports arbitrary-size point clouds and varying topologies, unlike fixed-cardinality representations.The paper identifies parameterizing and learning this distribution from sampled point clouds as a key challenge.
- Approach: The method learns a shape’s log-density gradient with denoising score matching and samples points using stochastic gradient ascent.The sampling procedure is implemented through a variant of Langevin dynamics.
- Advantages: A simple L2 objective, direct gradient modeling, and no partition-function estimation simplify optimization, relax architectural restrictions, and accelerate training.The model can also furnish an implicit surface without ground-truth surfaces during training.
- Contributions: The paper extends score-based generative models to conditional distributions and introduces an algorithm for extracting high-quality implicit surfaces.These contributions target both point-cloud generation and surface reconstruction.
- Results: The model achieves state-of-the-art performance for point cloud auto-encoding and generation.The reported results also indicate that performance can be retained when training uses much sparser point clouds.
2 Related work
Prior point-cloud generative models use fixed-dimensional representations, heuristic set distances, GANs, flows, or autoregressive structures, each with modeling or optimization limitations. This work instead predicts a non-normalized density’s log-gradient, enabling score matching without partition-function computation.
- Point cloud generative modeling: Earlier models represent point clouds as fixed-dimensional N × 3 matrices or rely on Chamfer and Earth Mover distances, creating drawbacks for point-cloud generation.These approaches predefine point count or use heuristic set-distance objectives.
- Energy-based modeling: The proposed approach predicts the gradient of a log-density field for a non-normalized probability density, avoiding partition-function computation and enabling faster training with a simple L2 loss.This connects the method to energy-based modeling and score matching.
- Generating other 3D representations: Implicit representations can model highly detailed shapes and structured local regions, but reconstructing their mesh surfaces requires an additional reconstruction process.The passage situates implicit methods as a separate family of 3D representations.
3 Method
The method models shapes through distributions of surface points, then samples arbitrary-sized point clouds by following a learned log-density gradient across noise levels. It also supports implicit-surface extraction and multiple-shape generation through latent modeling.
- Shapes as distributions: Shapes are modeled as distributions of 3D surface points, enabling point-cloud generation with arbitrary cardinality.The framework separately models the distribution of points conditioned on a shape and the distribution of shapes across the dataset.
- Gradient-field modeling: The method learns a neural network gθ(x, σ) that predicts the gradient of the logarithmic density of a noise-perturbed surface distribution.Sampling then moves points from a prior toward high-probability regions by gradient ascent on the learned field.
- Gradient-field analysis: As σ decreases, the gradient field changes from a coarse average shape toward nearest-surface directions that encode finer geometric details.For small σ, the gradient norm also approximates a scaled distance field near the surface.
- Training objective: The training objective uses denoising score matching to replace the difficult direct gradient target with a computable L2 loss using noisy observations.Multiple noise levels supervise regions at different distances from the surface, mitigating sparse supervision when σ is small.
- Point-cloud sampling: Sampling uses annealed Langevin dynamics: noisy prior points are recursively perturbed and updated by the predicted gradient field toward the surface.The prior may be uniform or Gaussian, provided its points lie in regions reached by perturbed training points.
- Surface extraction and multiple shapes: The learned gradient field defines an implicit surface through a near-zero gradient-norm iso-surface, while a latent GAN enables sampling different shape codes.Ambiguous nearest-surface locations can also produce zero gradients, although the text states these regions are seldom problematic in practice.
4 Experiments
Experiments evaluate auto-encoding, sparse-input upsampling, generation, and ablations on MNIST-CP and ShapeNet. The method performs strongly across reconstruction and generation settings while supporting implicit-surface extraction.
- Experimental setup: Experiments cover point-cloud auto-encoding, upsampling, generation, and ablation studies on MNIST-CP and ShapeNet.ShapeNet experiments include 35,708 training shapes, 5,158 test shapes, and 55 categories.
- Shape auto-encoding: The model consistently outperforms competing methods on EMD across the auto-encoding datasets.It also sometimes outperforms Achilioptas et al. and AtlasNet on CD.
- Point cloud upsampling: Training on significantly sparser shapes yields comparable auto-encoding performance, and performance is not strongly affected by the number of available points N.Airplane models vary N from 1K to 10K and input M from 128 to 2048, while evaluation uses 2048 output points.
- Point cloud upsampling: With only 256 input points, the model can reconstruct shapes and provide a smooth implicit surface.Figure 4 compares reconstructions from 2048-point and 256-point inputs, showing point clouds and implicit surfaces.
- Shape generation: Shape generation achieves comparable performance to PointFlow while requiring less training time.Training takes within one day on a 1080 Ti GPU, compared with at least two days to reproduce PointFlow’s results.
- Ablation study: Training with a single noise level degrades performance, especially at σ = 0.01, where large unsupervised regions cause significant errors.The full procedure trains across multiple noise levels and uses annealed Langevin dynamics during inference.
- Ablation study: Different prior configurations do not affect performance, consistent with the stochastic nature of the sampling procedure.The model remains robust when initialized from Gaussian points, a fixed location, or 3D-letter-shaped priors.
5 Conclusions
The paper proposes a point-cloud generative model based on learning a shape’s log-density gradient field. It produces high-quality point clouds, extracts underlying surfaces, and is demonstrated on auto-encoding, generation, and super-resolution.
- Conclusion: The method learns the gradient field of the logarithmic density function encoding a shape.This gradient-field representation is the paper’s core generative-model formulation.
- Conclusion: The model supports high-quality point-cloud sampling and extraction of the underlying shape surface.The conclusion identifies both point-cloud generation and implicit-surface extraction as capabilities.
- Conclusion: Effectiveness is demonstrated on point-cloud auto-encoding, generation, and super-resolution.The paper identifies extending the method to texture, appearance, and scenes as future work.
B Method Details
The method details describe training and inference for a gradient-field model, annealed Langevin sampling, implicit-surface rendering, and an equivalence between direct and denoising objectives.
- Surface rendering: Ray casting renders the implicit surface by advancing camera rays through the learned gradient field until they converge or miss the surface.Reached surfaces are shaded using the estimated normal, while missed rays receive the background color.
- Training and inference: Training uses a point-cloud encoder and neural network across multiple noise levels, followed by optimization with Adam.Noisy points are formed by adding Gaussian noise before computing the noise-level losses.
- Training and inference: Inference uses annealed Langevin dynamics, repeatedly adding gradient-based updates and Gaussian noise across the trained noise levels.The procedure initializes points, iterates for each noise level, carries the final state forward, and returns the sampled points.
- Objective function: The direct and denoising objectives differ only by a θ-independent constant, so optimizing either yields the same optimal θ.The appendix establishes this equivalence by decomposing both objectives and comparing their terms.
C.1 Network architecture
The architecture encodes each point with a shape latent code and noise level, then predicts a three-dimensional gradient vector. The decoder uses conditional normalization and residual blocks, while generation models latent codes with a separate GAN.
- C.1 Network architecture: The encoder maps each input shape point cloud to a 128-dimensional latent code z.
- C.1 Network architecture: Each decoder input concatenates point x, latent code z, and noise level σ, producing a 131- or 132-dimensional vector for 2D or 3D data.
- C.1 Network architecture: The decoder outputs a three-dimensional vector describing the gradient for the input point.
- C.1 Network architecture: Conditional Batch-Normalization uses the concatenated input [x, z, σ] to generate feature-wise parameters β and γ.
- C.1 Network architecture: Shape generation trains a latent-code GAN with WGAN-GP while keeping the pretrained encoder and decoder fixed.
C.2 Experimental setting
Experiments train across multiple noise levels using Adam, dataset-specific learning rates, and fixed training schedules. Baseline results are obtained from official implementations or retrained under matched preprocessing and iteration settings.
- C.2 Experimental setting: The experiments use ten noise levels σ ranging from 1 to 0.01 and train for 2000 epochs.
- C.2 Experimental setting: ShapeNet uses learning rates of 1 × 10^-4 for the decoder and 1 × 10^-3 for the encoder, with linear decay beginning at epoch 1000.
- C.2 Experimental setting: Inference uses T = 10 and α = 2 × 10^-4.
- C.2 Experimental setting: PointFlow, AtlasNet, l-GAN, and r-GAN results use released implementations with matched or specified preprocessing, latent dimensions, and iteration settings.
- C.2 Experimental setting: GraphCNN-GAN and TreeGAN results use their officially released code.
D.1 Shape generation
The study evaluates shape generation against fixed-dimensional point-cloud baselines, including retrained models under the same data processing. The method is reported as competitive and converges faster to a better auto-encoding result than PointFlow.
- D.1 Shape generation: The evaluation compares shape generation with GraphCNN-GAN and TreeGAN, alongside rGAN and PointFlow comparisons reported in the main paper.
- D.1 Shape generation: Baseline models marked with an asterisk are retrained on full ShapeNet collections using the same splits and preprocessing as the proposed models.
- D.1 Shape generation: The model achieves performance described as on par with the state of the art for car-category generation.
- D.1 Shape generation: On Airplane auto-encoding, the method converges much faster and to a better result than PointFlow.
D.2 Implicit surface
The learned gradient field is used to extract implicit surfaces and is evaluated on Airplane meshes. The extracted surfaces outperform AtlasNet on reported metrics and are competitive with DeepSDF, but surface extraction has identifiable failure modes.
- D.2 Implicit surface: MISE, an octree-based marching cube algorithm, extracts ground-truth meshes from the learned gradient field on Airplane test shapes.
- D.2 Implicit surface: The implicit surfaces outperform AtlasNet in both CD and Mesh accuracy and are competitive with DeepSDF in CD despite DeepSDF using more supervision.
- D.2 Implicit surface: Extracted-mesh failures usually occur near bifurcations, where gradients are close to zero.
- D.2 Implicit surface: Marching cubes can create a double surface around the shape, and improving surface extraction is left for future work.
E Additional Ablation Studies
The ablations examine network components and alternative shape-distribution modeling choices. The full architecture performs better and varies less across inference runs than the tested configurations.
- Network architecture: The study evaluates batch normalization, conditional batch normalization, shortcuts, and alternative placements of latent code and noise-scale inputs.These configurations are compared as architectural design choices.
- Network architecture: The full abcd configuration achieves better performance and significantly lower variance across three inference runs than the other tested configurations.Results are reported as averages and standard deviations over the three runs.
- Modeling the distribution of shapes: The authors also compare their latent-GAN approach with a VAE for modeling the distribution of shapes.The VAE uses the same encoder and decoder setting as the main-paper model, with the encoder outputting both µ and σ.
- Modeling the distribution of shapes: Generation performance is compared using MMD-CD and MMD-EMD, with higher-is-better or lower-is-better directions specified for the metrics.The table caption states that MMD-CD is multiplied by 10^3 and MMD-EMD by 10^2.
F Additional Qualitative Results
Additional experiments show reconstruction on partial scanned data, meaningful latent-space organization, prior-insensitive 2D sampling, and qualitative ShapeNet reconstruction, generation, and interpolation results.
- Scanned data: The method reconstructs denser representations from sparse, partial real-world scans, although it cannot model the complete shape from a single view.Each scanned object is modeled separately using point clouds captured by a handheld 3D scanner.
- Scanned data: The scanned-shape experiment demonstrates that the technique applies beyond synthetic benchmark examples.The real scans are sparse and incomplete, so the reported reconstruction target is a denser partial representation.
- Visualization of latent space: Similar shapes are placed nearby in the learned latent space, suggesting that the latent representation is meaningful.The visualization embeds 128-dimensional latent codes into two dimensions with T-SNE.
- Extended visualizations for 2D and 3D point clouds: Annealed Langevin dynamics is demonstrated for 2D MNIST-CP point clouds, and reconstruction is reported as insensitive to the prior distribution.The demonstrations include uniform, Gaussian, and single-point starting distributions.
- Extended visualizations for 2D and 3D point clouds: Additional ShapeNet figures show auto-encoding of shapes and point clouds, shape generation, and interpolation across Airplane, Car, and Chair categories.The reconstruction examples also include corresponding implicit surfaces and up-sampling outputs.