PaperlayerView exact revision on arXiv

Source-linked AI summary

LION: Latent Point Diffusion Models for 3D Shape Generation

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis

arXiv:2210.06978v1cs.CVcs.LGstat.ML

TL;DR

Artist-oriented 3D generation needs high quality, flexible manipulation, and smooth mesh output. LION addresses these needs with hierarchical latent diffusion, achieves state-of-the-art ShapeNet synthesis, and supports denoising, conditioning, interpolation, and mesh reconstruction.

  • Problem

    3D generative models must combine high-quality shapes, flexible manipulation and conditioning, and smooth mesh output for practical digital-art use.

  • Method

    LION uses a hierarchical VAE with global and point-structured latents, training two DDMs in latent space before decoding generated samples into point clouds.

  • Results

    LION achieves state-of-the-art synthesis performance on ShapeNet benchmarks and supports multimodal denoising, voxel-guided synthesis, text- and image-driven generation, interpolation, autoencoding, and smooth meshes.

  • Takeaways & Limitations

    LION provides a flexible 3D shape-generation framework that combines latent point-cloud modeling with artist-oriented conditioning, manipulation, and surface reconstruction.

Abstract

from arXiv · show

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

1 Introduction

LION addresses the quality, flexibility, and smooth-output requirements for artist-oriented 3D shape generation through hierarchical latent diffusion. It combines a hierarchical VAE and latent DDMs, achieving strong synthesis performance while supporting diverse applications and mesh reconstruction.

  • Motivation: Existing 3D DDMs struggle to satisfy high quality, flexible manipulation, and smooth mesh output simultaneously.These requirements include refinement, varied details, noisy or coarse input guidance, and shape interpolation.
  • Method: LION is a hierarchical VAE-based generative model with two latent DDMs operating on a global shape latent and point-structured latent space.Synthesis samples hierarchical latent representations and decodes them back into point clouds.
  • Advantages: LION maps point clouds into regularized latent spaces, making diffusion modeling easier while retaining point structure and adding a global shape latent for expressivity.The latent DDMs model a smoothed distribution rather than potentially complex point clouds directly.
  • Results: LION achieves state-of-the-art synthesis performance on widely used ShapeNet benchmarks, including joint training over many classes without conditioning.The contribution summary reports high-quality and diverse synthesis in this setting.
  • Mesh Reconstruction: Combining LION with SAP-based surface reconstruction produces smooth meshes suitable for practical 3D shape output.Fine-tuning SAP on autoencoder-generated data reduces synthesis noise and improves geometry quality.
  • Applications: Fine-tuned encoders enable multimodal voxel-guided synthesis and shape denoising, while latent spaces support shape interpolation and autoencoding.Conditioning the DDMs on CLIP embeddings additionally enables image- and text-driven 3D generation.

2 Background

Diffusion models progressively add noise to data and learn a reverse process that denoises samples. They can be formulated discretely or continuously, with the latter supporting deterministic ODE-based sampling and encoding.

  • Forward Diffusion: In the discrete formulation, a fixed Markovian forward process gradually perturbs data with Gaussian transitions across T steps.The variance schedule β1, ..., βT is chosen so the terminal distribution approximately approaches a standard Gaussian.
  • Reverse Generation: The reverse process learns Gaussian transition kernels that invert the forward diffusion and generate data from noise.Sampling begins with xT drawn from a standard Gaussian and proceeds iteratively.
  • Training Interpretation: DDMs can be interpreted as latent-variable models whose fixed forward process serves as an approximate posterior for the generative reverse process.Training minimizes a variational upper bound on the negative log-likelihood.
  • Continuous-Time Diffusion: In the continuous-time formulation, diffusion and reverse generation are described by differential equations that enable deterministic sampling and encoding through ODEs.LION uses this framework in its latent modeling procedure.

3 Hierarchical Latent Point Diffusion Models

LION is a hierarchical VAE that models 3D shapes with global and point-structured latent representations, each learned by a latent DDM. Its framework supports conditional synthesis, denoising, interpolation, and smooth mesh generation.

  • 3 Hierarchical Latent Point Diffusion Models: LION models point clouds with a global shape latent and a point-structured latent, then trains latent DDMs after VAE training.The VAE is trained first, followed by two latent DDMs on encoded representations.
  • 3 Hierarchical Latent Point Diffusion Models: LION’s latent DDMs hierarchically generate global and point-structured latents before decoding them into point clouds.The generative model factors into a global latent DDM, a conditional point-latent DDM, and a decoder.
  • 3.1 Applications and Extensions: Partial latent diffuse-denoise produces controlled shape variations while preserving the encoded input’s overall structure.Diffusing for fewer than T steps destroys mainly local details before reverse generation creates variants.
  • 3.1 Applications and Extensions: Fine-tuned encoders enable voxel-conditioned synthesis and noisy-shape denoising without retraining the latent DDMs.Voxelized or noisy inputs are mapped to latent encodings that reconstruct detailed or denoised point clouds.
  • 3.1 Applications and Extensions: LION supports interpolation in its Gaussian latent priors and smooth mesh output through SAP-based surface reconstruction.SAP is fine-tuned on LION autoencoder data to better accommodate generated point-cloud noise.
  • 3.2 LION’s Advantages: The hierarchical latent design improves expressivity, while the VAE framework provides flexibility beyond direct point-cloud DDMs.The additional global latent captures overall shape and latent points capture local details; encoders can be adapted for multiple tasks.

4 Related Work

LION builds on latent and hierarchical diffusion approaches while targeting 3D point-cloud generation. It differs from closely related PVD and DPM methods through its hierarchical latent design and broader application flexibility.

  • 4 Related Work: LION extends latent-space diffusion and hierarchical modeling ideas previously explored for image, music, upsampling, text-to-image, and semantic image generation.The related approaches generally do not train separate conditional DDMs in the same framework.
  • 4 Related Work: PVD trains a DDM directly on point clouds, whereas DPM combines a shape latent variable with a weaker point-wise conditional DDM.LION’s design is motivated by retaining point-cloud structure while modeling regularized hierarchical latents.
  • 4 Related Work: Image- and text-driven 3D generation are described as complementary directions, while LION can be augmented with image-based training or text guidance.The paper also relies on SAP for mesh generation despite the availability of alternative surface-reconstruction methods.

5 Experiments

Experiments evaluate LION across standard and challenging multi-class settings, low-data training, voxel-guided synthesis, mesh reconstruction, interpolation, and accelerated sampling. LION achieves strong generation quality while supporting diverse conditioning and practical outputs.

  • Single-Class 3D Shape Generation: LION outperforms all baselines and achieves state-of-the-art performance across ShapeNet classes and dataset versions.It also surpasses the DDM-based baselines PVD and DPM by large margins, with diverse and visually pleasing samples.
  • Mesh Reconstruction and Shape Interpolation: Combining LION with mesh reconstruction produces smooth, high-quality meshes and supports semantically plausible interpolation paths.DDIM-based sampling also generates high-quality shapes in under one second.
  • Many-class Unconditional 3D Shape Generation: LION synthesizes high-quality, diverse shapes when jointly trained without conditioning on 13 ShapeNet categories.Its global latent captures overall shape while latent points model details.
  • Low-data Generation: LION also generates plausible shapes in low-data settings, including mugs, bottles, and 553 TurboSquid animal assets.The Mug and Bottle classes contain 149 and 340 training samples, respectively.
  • Voxel-guided Shape Synthesis and Denoising: LION performs best for almost all voxelized and noisy inputs in reconstruction comparisons against DPM and PVD.Unlike the baselines, LION handles voxel inputs and outlier noise while balancing output quality and input correspondence across denoising steps.
  • Voxel-guided Shape Synthesis and Denoising: More diffuse-denoise steps lower 1-NNA while also lowering voxel IOU, and LION provides the best balance between quality and voxel agreement.The trade-off is visualized using generated point clouds and voxelizations.

6 Conclusions

LION is a hierarchical latent diffusion model for 3D point-cloud generation that can be extended to meshes and interactive manipulation. Its main scope boundaries are point-cloud-only training, single-object generation, and untextured outputs.

  • Conclusions: LION uses a hierarchical VAE with latent DDMs and can be combined with SAP for mesh generation.The model represents global shape and point-structured information in latent spaces.
  • Conclusions: The framework supports voxel-conditioned synthesis, multimodal shape denoising, shape interpolation, autoencoding, and text- or image-driven generation.These applications reuse LION’s latent spaces and can adapt encoders or condition DDMs as needed.
  • Limitations and Future Work: LION is currently trained only on 3D point clouds, cannot directly generate textured shapes, and focuses on single-object generation.The paper identifies image-based training, texture synthesis, scene generation, and accelerated sampling as future extensions.
  • Applications: The diffuse-denoise process injects controlled diversity and cleans imperfect encodings from noisy or voxelized inputs.Small diffusion depths preserve the original shape while larger depths enable more variation.

C.2 Encoder Fine-Tuning for Voxel-Conditioned Synthesis and Denoising

LION adapts its encoders to voxelized or noisy inputs, enabling guided synthesis and denoising without retraining latent DDMs. Its latent spaces also support interpolation, while SAP converts generated point clouds into smooth meshes.

  • Encoder fine-tuning: LION fine-tunes its encoders to ingest voxelized or perturbed point clouds instead of clean inputs.The perturbations include uniform, Gaussian, outlier noise, and voxelization.
  • Guided generation: Voxelized or noisy inputs can guide LION’s generative model for multimodal synthesis and shape denoising.The latent DDMs remain unchanged while the encoders are adapted to the guidance inputs.
  • Shape interpolation: LION interpolates shapes in the prior spaces of its latent DDMs rather than directly interpolating point-cloud coordinates.The method uses square-root-based spherical interpolation because linear interpolation can leave the typical Gaussian shell and produce poorer results.
  • Surface reconstruction: Decoded interpolated latents produce point clouds that can optionally be converted into meshes with SAP.SAP-based reconstruction is also used to augment LION with smooth surface output.
  • Surface reconstruction: SAP reconstructs a smooth surface by predicting a point-cloud indicator function and extracting its zero level set.Its pipeline densifies points, predicts normals, solves a Poisson PDE, and can produce polygonal meshes.

D.4 Two-stage Training

LION training proceeds in two stages: first learning a hierarchical VAE, then fitting diffusion priors to its latent encodings while freezing the VAE.

  • First stage: LION first trains its two encoders and decoder as a VAE with annealed KL regularization.The KL weights start at 10^-7, are annealed over the first 50% of epochs, and finish at 0.5.
  • Second stage: The second stage freezes the encoders and decoder and trains only the two latent DDM prior networks.The shape-latent and latent-point DDMs are optimized with their respective diffusion objectives.
  • Second stage: During second-stage training, clean point clouds are encoded into z0 and h0, diffused at sampled time steps, and denoised by the latent priors.The shape prior receives zt and t, while the latent-point prior receives z0, t, and ht.
  • Regularization: Spectral normalization, group-normalization regularization, and exponential moving averages stabilize latent DDM training and inference.The parameter EMAs are used during inference when calling the DDM priors.

E.1 Different Datasets

Experiments evaluate LION on ShapeNet categories using dataset conventions inherited from prior work, with separate preprocessing considerations for baseline comparisons and SAP.

  • ShapeNet datasets: Unconditional generation experiments use ShapeNet airplane, chair, and car categories with PointFlow preprocessing.Each shape contains 15,000 points, with 2,048 randomly sampled during training.
  • Model configuration: The reported architecture configuration includes separate hyperparameter tables for the decoder, shape-latent DDM, and latent-point DDM.These tables document the model components used in the experiments.
  • Normalization: LION is evaluated under both global and per-shape normalization because different baselines require different coordinate conventions.These normalization choices define different generative modeling problems and therefore affect comparisons.
  • SAP data: SAP training uses separate data splits and preprocessing that provide watertight meshes for volumetric supervision.The PointFlow data does not provide the signed distance fields required for SAP training.

E.2 Evaluation Metrics

LION’s evaluation prioritizes 1-NNA because it measures both quality and diversity, while COV and MMD have known limitations. Metrics are computed on point clouds for fair comparisons.

  • Metric limitations: COV measures diversity and mode coverage but does not quantify generated-shape quality.Low-quality yet diverse samples can still obtain high coverage scores.
  • Metric limitations: MMD measures average distance to the nearest generated shape but does not reliably capture overall quality, diversity, or mode coverage.Low-quality generated clouds may simply remain unmatched to reference shapes.
  • Primary metric: 1-NNA is the primary evaluation metric because it measures distribution similarity, quality, and diversity between generated and reference sets.A value near 50% indicates that generated and reference samples are difficult for the nearest-neighbor classifier to distinguish.
  • Results: LION generally outperforms all baselines on the more meaningful 1-NNA metric across experiments.The evaluation uses both Chamfer distance and earth mover distance variants.
  • Evaluation protocol: For fair comparison, metrics are computed directly on generated point clouds rather than SAP-generated meshes.A separate ablation evaluates performance after mesh reconstruction by sampling points from the reconstructed surfaces.
  • Experimental caveat: The many-class LION model receives no hyperparameter tuning and may therefore be further improved.The authors identify dropout probability and the shape-latent DDM hidden dimension as influential settings.

F Additional Experimental Results

The supplementary experiments extend LION’s evaluation across architectures, latent dimensions, datasets, classes, autoencoding, accelerated sampling, text/image conditioning, and interpolation.

  • Additional generation results: LION’s supplementary studies cover single-class, 13-class, 55-class, Mug, Bottle, and animal-shape generation experiments.
  • Other applications: Supplementary experiments provide additional voxel-guided synthesis, denoising, shape interpolation, and latent-space visualization results.
  • Autoencoding: LION’s autoencoding performance is quantified and compared with baselines, which LION outperforms.
  • Accelerated synthesis: Additional experiments evaluate accelerated DDIM-based synthesis in LION.
  • Text and image conditioning: Text2Mesh generates textures from text prompts for synthesized LION samples, while CLIP conditioning supports text-driven generation and single-view 3D reconstruction.

F.1.1 Ablation Study on LION’s Hierarchical Architecture

The hierarchical-architecture ablation compares LION with models removing shape latents, latent points, or all latent variables, while controlling approximately for parameter count.

  • Experimental settings: The ablation evaluates LION without shape latents, without latent points, and without any latent variables.
  • Experimental control: Model sizes are increased in ablated variants so they have approximately as many parameters as the full LION model.

F.1.2 Ablation Study on the Backbone Point Cloud Processing Network Architecture

The backbone ablations compare PVCNN, DGCNN, and PointTransformer for LION’s point-cloud networks and support PVCNN as the selected architecture.

  • Compared architectures: PVCNN, DGCNN, and PointTransformer are evaluated as backbones for LION’s encoder, decoder, and latent-points prior.
  • Encoder and decoder: PVCNN achieves the strongest performance for the encoder and decoder backbones.
  • Latent-points prior: PVCNN also performs best as the network implementing LION’s latent-points diffusion prior.
  • Experimental control: The ablations use similar hyperparameters and generally configure networks to consume the same GPU memory.
  • Latent dimensions: The latent-points extra dimension D_h is ablated, with D_h = 1 providing the overall best performance.

F.1.4 Ablation Study on SAP Fine-Tuning

SAP fine-tuning improves mesh reconstruction from LION outputs, while supplementary evaluations characterize generation, diversity, conditioning, scalability, and autoencoding across ShapeNet settings.

  • SAP fine-tuning: Without fine-tuning, reconstructed meshes have slightly lower 1-NNA quality, presumably because their noise differs from SAP’s training noise.
  • SAP fine-tuning: Training SAP with a mixed number of diffuse-denoise steps tends to provide good overall 1-NNA sample quality and is used in all experiments.
  • SAP fine-tuning: Fine-tuning SAP on LION-generated data produces smoother surfaces and improves robustness to LION’s noise distribution.
  • Generation results: LION achieves state-of-the-art 1-NNA performance across the reported single-class settings and remains competitive on MMD and COV.
  • Multi-class generation: In the 13-class setting, LION performs better than all evaluated baselines and generates diverse, high-quality shapes.
  • Scalability: For 55 classes, LION generates diverse shapes including the cap class with only 39 training samples, but no baselines are trained because of limited compute.
  • Conditioned synthesis: Voxel-guided synthesis and denoising generate multiple plausible shapes and reasonable diverse details across voxelized, normal-noise, uniform-noise, and outlier-noise inputs.
  • Autoencoding: LION’s VAE achieves much better reconstruction performance than all compared baselines while retaining state-of-the-art generation quality.

F.10 Per-sample Text-driven Texture Synthesis

LION-generated meshes support text-driven texture synthesis and broader conditional 3D generation. The framework also demonstrates interpolation, single-view reconstruction, and accelerated sampling across varied settings.

  • Per-sample Text-driven Texture Synthesis: Text2Mesh applies text-driven textures to LION-generated meshes, with optional local mesh refinement and vertex displacement.The mesh-based pipeline produces objects such as snow- or potato-chip-covered shapes and extends to other categories.
  • Sampling Efficiency: DDIM sampling reduces per-object generation time from 27.09 seconds at 1,000 steps to less than 1 second at 25 steps, whereas 5 steps fails.Sampling with more than 10 steps can generate high-quality shapes.
  • Per-sample Text-driven Texture Synthesis: LION’s SAP-based surface reconstruction enables Text2Mesh applications that require smooth meshes rather than point clouds.The authors explicitly attribute the downstream texturing results to the mesh reconstruction stage.
  • Conditional Shape Generation: RGB-only conditioning enables multimodal single-view reconstruction, while CLIP text embeddings support text-guided generation of chairs and cars.The authors report qualitative results comparable to PVD and at least as good as AutoSDF, while noting that the demonstration was not hyperparameter-tuned.
  • Shape Interpolation: Compared with LION, PVD transitions appear less smooth, while DPM interpolations are noisy and both baselines break down for very different shapes in 13-class models.The comparison uses deterministic interpolation paths for the competing methods.
  • Shape Interpolation: LION generally produces coherent interpolations across different shapes, including models trained jointly on 13 ShapeNet classes.Examples include gradual transitions between chairs and tables or airplanes and chairs.
Loading 2210.06978v1…