PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Shape Generation and Completion through Point-Voxel Diffusion

Linqi Zhou, Yilun Du, Jiajun Wu

arXiv:2104.03670v3cs.CV

TL;DR

3D shape generation and completion require faithful outputs while accommodating multiple plausible completions. PVD combines denoising diffusion with a point-voxel representation to generate and complete shapes probabilistically, achieving high-fidelity synthesis and multi-modal completion, including real scans.

  • Problem

    Existing shape-generation models primarily use deterministic encoders and decoders, limiting their ability to represent multi-modal completion from partial observations.

  • Method

    PVD combines denoising diffusion models with a hybrid point-voxel representation and uses a reverse denoising process for generation and conditional completion.

  • Results

    PVD synthesizes high-fidelity shapes, outperforms multiple state-of-the-art methods, and produces multiple completions for partial point clouds and real 3D scans.

  • Takeaways & Limitations

    PVD provides one probabilistic framework for unconditional generation, multi-modal completion, and completion of real 3D scans.

  • Takeaways & Limitations

    Some compared completion methods are omitted because they are sensitive to partial inputs and produce unrealistic completions after Langevin sampling.

Abstract

from arXiv · show

We propose a novel approach for probabilistic generative modeling of 3D shapes. Unlike most existing models that learn to deterministically translate a latent vector to a shape, our model, Point-Voxel Diffusion (PVD), is a unified, probabilistic formulation for unconditional shape generation and conditional, multi-modal shape completion. PVD marries denoising diffusion models with the hybrid, point-voxel representation of 3D shapes. It can be viewed as a series of denoising steps, reversing the diffusion process from observed point cloud data to Gaussian noise, and is trained by optimizing a variational lower bound to the (conditional) likelihood function. Experiments demonstrate that PVD is capable of synthesizing high-fidelity shapes, completing partial point clouds, and generating multiple completion results from single-view depth scans of real objects.

1. Introduction

PVD addresses the need for faithful, probabilistic 3D shape generation and multi-modal completion by combining denoising diffusion with a hybrid point-voxel representation. It generates high-fidelity shapes and multiple completions from partial observations.

  • Existing point-cloud approaches typically condition generation on a shape encoding and cannot capture multi-modal completion.
  • Directly applying diffusion to pure voxels or point clouds produces poor quality because voxels are binary and point clouds require permutation-invariance.
  • PVD combines denoising diffusion models with a hybrid point-voxel representation to generate high-quality shapes probabilistically.
  • PVD unifies unconditional shape generation with conditional, multi-modal completion and samples multiple completions from different diffusion initializations.
  • Experiments show high-fidelity synthesis and multi-modal completion on partial point clouds, depth maps, and real-object scans across ShapeNet, PartNet, and Redwood.

2. Related Works

Related work spans voxel, point-cloud, probabilistic, and diffusion-based generative models. PVD builds on their strengths while addressing memory, fidelity, ordering, and representation constraints.

  • Prior point-cloud generators use autoencoding, reconstruction, adversarial, autoregressive, flow-based, or heuristic-distance approaches.
  • Voxel-based models are straightforward but memory-intensive because voxel grids grow cubically with resolution.
  • Point clouds preserve detailed smooth surfaces and use less memory, but their processing commonly requires permutation-invariant architectures.
  • PVD uses a point-voxel representation to exploit spatial correlation while avoiding direct diffusion on pure voxels or permutation-invariant point clouds.
  • Diffusion models formulate generation as probabilistic iterative denoising from Gaussian noise toward a target output.

3. Point-Voxel Diffusion

PVD learns a reverse diffusion process for point clouds, progressively denoising Gaussian-initialized samples. A conditional variant keeps observed points fixed while diffusing and generating missing points.

  • 3.1. Formulation: Generation starts from Gaussian noise x_T and progressively denoises shape variables until producing the final output x_0.
  • 3.1. Formulation: The forward process adds Gaussian noise, while the learned generative process approximates the reverse process through Gaussian transition probabilities.
  • 3.1. Formulation: Training maximizes a variational lower bound on data likelihood, with fixed forward transitions and a Gaussian prior reducing the objective to reverse-transition likelihood terms.
  • 3.1. Formulation: Because the posterior transitions are analytically tractable Gaussian distributions, the model can learn each timestep independently by matching the corresponding posterior.
  • 3.1. Formulation: The final training loss is an L2 distance between predicted noise and sampled Gaussian noise, teaching the model to predict the noise needed to decorrupt shapes.
  • 3.2. Shape Completion: For completion, the partial shape z_0 remains fixed while only free points are diffused, and the modified objective conditions generation on that observation.

4. Experiments

PVD is evaluated for unconditional generation, single-view shape completion, and multi-modal completion. Across these settings, it produces strong generation quality, realistic completions, and diverse outputs while balancing fidelity to partial observations against ground-truth similarity.

  • 4.1. Shape Generation: Pure point diffusion fails to generate visible shapes, while pure voxel diffusion produces noisy results because binary voxels conflict with the Gaussian diffusion assumption.These comparisons motivate PVD’s hybrid point-voxel representation.
  • 4.2. Shape Completion: PVD obtains the best EMD scores for shape completion, although some baselines achieve better CD scores.The authors argue EMD better reflects completion quality because it enforces matching point density, whereas CD can miss visual inferiority.
  • 4.2. Shape Completion: When the unseen geometry is ambiguous, baselines tend toward mean shapes, whereas PVD samples plausible completions that match the partial input but may differ from the ground truth.A reported example completes a vehicle as a realistic van rather than a sedan, increasing CD despite comparable realism.
  • 4.3. Multi-Modal Completion: PVD supports controlled completion from multiple partial shapes and produces different completions because each noise initialization can yield a different result.This probabilistic behavior contrasts with encoder-decoder baselines that output a single deterministic completion.
  • 4.3. Multi-Modal Completion: PVD generates multiple completion modes from partial observations, with variation decreasing when more of the object is visible.On PartNet, it outperforms cGAN and KNN-latent in average diversity and quality; ShapeNet results also preserve constraints while varying from canonical views.

5. Conclusion

PVD provides a unified framework for shape generation and completion, using diffusion-based denoising and a point-voxel representation. Experiments show strong generation and completion performance, including multiple completions for partial shapes and real 3D scans.

  • PVD unifies unconditional shape generation and conditional shape completion in one diffusion-based framework.The model progressively removes noise from noise-initialized samples, while a minor objective modification supports completion without architectural changes.
  • PVD uses a hybrid point-voxel representation to generate shapes through progressive denoising.
  • PVD produces multiple possible completion results from a partial shape, unlike deterministic encoder-decoder baselines.
  • PVD also completes real 3D scans, supporting practical use in downstream applications.

A. Additional Generation Metrics

Additional experiments evaluate PVD with coverage, minimum matching distance, and 1-NN metrics across generated 3D shapes. The reported comparison finds that PVD significantly outperforms Vox-Diff on Airplane, Chair, and Car.

  • Coverage measures the fraction of reference point clouds matched to at least one generated point cloud.
  • Minimum matching distance measures each reference point cloud’s distance to its nearest neighbor in the generated set.
  • PVD significantly outperforms Vox-Diff on Airplane, Chair, and Car using the 1-NN metric.The passage attributes this result to PVD generating less noisy point clouds.
  • Generation results are also visualized through the generation process and final shapes from all angles.

C. Derivation of the Variational Lower Bound

The diffusion derivation uses a variational lower bound to make likelihood learning tractable. Gaussian forward-process marginals and posteriors provide closed-form quantities needed for the model formulation.

  • The likelihood objective is rewritten as an expectation over the forward process and lower-bounded using Jensen’s inequality.
  • The forward marginal q(x_t|x_0) is Gaussian with mean √tildeα_t x_0 and covariance (1−tildeα_t)I.
  • Because the transition probabilities are Gaussian, the posterior q(x_{t−1}|x_t,x_0) is also Gaussian and has a closed-form expression.

E. Derivation of L2 Loss

The model’s L2 training objective matches learned reverse-process transitions to Gaussian diffusion posteriors and predicts noise instead of transition means. Generation then reverses diffusion iteratively, while latent interpolation enables controlled completion.

  • Derivation of L2 Loss: Maximum-likelihood learning reduces to an L2 loss matching pθ(x_{t−1}|x_t) with q(x_{t−1}|x_t,x_0).Both distributions are Gaussian under a predetermined variance schedule β_1,...,β_T.
  • Derivation of L2 Loss: The model predicts noise εθ(x_t,t), yielding the final objective ||ε−εθ(x_t,t)||^2.The target noise is sampled from N(0,I).
  • Point Cloud Generation Process: Generation samples pθ(x_{t−1}|x_t) successively for t=T,...,1, progressively removing noise from the initial sample.
  • Point Cloud Generation Process: The iterative update resembles Langevin dynamics because both combine a model-dependent shift with a scaled noise term.
  • Controlled Completion: Latent interpolation combines completion features from two partial observations, with λ controlling how much the features are shared.

H.1. Model Architecture

The model uses a point-voxel CNN adapted from PointNet++ within a diffusion process, conditioning computation on time embeddings. For completion, partial observations are repeatedly preserved while noisy points are denoised.

  • The point-voxel CNN replaces PointNet substructures in a PointNet++-modified architecture.The architecture is specified through set abstraction and feature propagation modules.
  • Temporal embeddings are concatenated with point features before set abstraction and feature propagation modules.The embeddings use sinusoidal positional encoding based on the diffusion time step.
  • The same architecture supports both unconditional generation and conditional completion.For completion, the input consists of 200 partial-shape points and 1,848 noise-sampled points, totaling 2,048 points.
  • At each completion step, the first 200 generated points are replaced with the observed partial shape before the next step.This replacement keeps the partial observation fixed throughout the iterative process.
  • The diffusion schedule uses T = 1000 steps, with category-specific β0 and βT settings and linear interpolation between schedule values.The authors report that lower timestep counts, such as 100, are generally insufficient for constructing shapes.

H.3. Training Parameters

Training uses Adam optimization, while the architecture processes point clouds and time steps through sequential abstraction, propagation, and output layers. The supplementary figures document the architecture and category-specific generation processes and views.

  • H.3. Training Parameters: Adam optimization uses a learning rate of 2 × 10^-4 for all experiments.
  • H.3. Training Parameters: Supplementary figures show the model architecture and generation processes or all-angle results for airplanes, cars, and chairs.
  • H.3. Training Parameters: The set abstraction layer applies PVConv modules, an MLP, and sampling and grouping, while feature propagation applies interpolation, PVConv modules, and an MLP.
  • H.3. Training Parameters: The full point-voxel CNN passes input point clouds and time steps through SA 1–4, FP 1–4, and an MLP to produce outputs of the same dimension.Time embeddings and point features are concatenated at the start of each set abstraction and feature propagation module.
  • H.3. Training Parameters: Additional supplementary figures show chair generation processes and chair results from all angles.
Loading 2104.03670v3…