PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models

Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, Paul Guerrero

arXiv:2212.00842v2cs.CV

TL;DR

Neural implicit functions can represent accurate, efficient surfaces, but it is not straightforward to apply diffusion to them. 3D-LDM applies diffusion in an auto-decoder’s latent space, reports improved unconditional generation over direct latent sampling and prior state of the art, and supports text/image conditioning and shape variation.

  • Problem

    Voxels are computationally and memory intensive at high resolution, point clouds need lossy post-processing to obtain surfaces, and how to generate neural implicit representations with diffusion is unclear.

    Source: §1 · p. 2

  • Method

    3D-LDM trains an auto-decoder for SDF-based shapes, freezes it, and trains a DDPM on the shapes’ latent representations.

    Source: §3 · p. 3

  • Results

    3D-LDM is significantly better than direct auto-decoder latent sampling and outperforms the previous state of the art in unconditional 3D shape generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach supports high-quality surface generation, text- and image-conditioned 3D generation, and noise-based variations of existing shapes.

    Source: §1 · p. 2

    The reported training scope is a single category; training jointly across multiple categories is proposed as future work.

    Source: §5 · p. 7

Abstract

from arXiv · show

Diffusion models have shown great promise for image generation, beating GANs in terms of generation diversity, with comparable image quality. However, their application to 3D shapes has been limited to point or voxel representations that can in practice not accurately represent a 3D surface. We propose a diffusion model for neural implicit representations of 3D shapes that operates in the latent space of an auto-decoder. This allows us to generate diverse and high quality 3D surfaces. We additionally show that we can condition our model on images or text to enable image-to-3D generation and text-to-3D generation using CLIP embeddings. Furthermore, adding noise to the latent codes of existing shapes allows us to explore shape variations.

1. Introduction

Diffusion has only recently been applied to 3D, where representations vary and point clouds or voxels have practical drawbacks. 3D-LDM uses diffusion in an auto-decoder’s latent space to generate neural implicit shapes, with conditional generation and shape exploration as additional capabilities.

1. Introduction

  • Neural implicit functions offer accurate, efficient surfaces, but applying diffusion to them is not straightforward; voxels are costly at high resolution and point clouds require lossy surface post-processing.3D DDPMs also face a less standardized choice of representation than 2D image models.Source: §1 · pp. 1–2
  • 3D-LDM applies diffusion to the latent space of an auto-decoder for signed distance functions, bypassing direct diffusion on continuous surfaces.The paper reports better results than direct sampling from the auto-decoder’s latent space and prior state of the art in unconditional shape generation.Source: §1 · p. 2
  • 3D-LDM supports text- and image-conditioned shape generation using context such as CLIP embeddings.This is presented as an additional capability of the latent diffusion approach.Source: §1 · p. 2
  • Adding noise to generated or existing shapes provides guided shape exploration.The paper describes this as a way to create shape variations.Source: §1 · p. 2

2. Related Work

Related work spans 3D generative models, diffusion representations, implicit-function generation, and conditional generation. The paper positions 3D-LDM among work on 3D diffusion and implicit shapes, while noting representation and computational trade-offs.

2. Related Work

  • 3D generative models include GANs, flows, score-matching, autoregressive, diffusion, and energy-based models; likelihood-based metrics help assess overfitting and novelty when generated shapes lack ground truth.These models aim to approximate a dataset distribution from which novel shapes can be sampled.Source: §2 · p. 2
  • Most prior 3D diffusion work uses point clouds or voxels, whereas 3D-LDM models latent distributions over implicit functions for continuous surfaces.The paper notes that point and voxel representations can limit graphics applications such as animation.Source: §2 · p. 2
  • Prior generative implicit-surface work includes multi-scale Neural Wavelet and part-aware Spaghetti; LION uses a differentiable Shape as Point module but does not leverage its differentiability within the network, adding computational burden.The passage frames implicit functions as continuous representations that preserve fine-grained surface details.Source: §2 · p. 2
  • Conditional 3D generation includes depth-map-conditioned PVD and sparse-image-conditioned GAUDI, while 3D-LDM uses CLIP embeddings obtainable from text or images.The paper says novel 3D scene generation by GAUDI is not demonstrated conclusively.Source: §2 · pp. 2–3

3. Method

3D-LDM separates shape representation learning from shape generation: a pretrained auto-decoder represents 3D shapes, and a diffusion model learns to generate its latent codes. The method extends this pipeline to CLIP-conditioned generation from images or text.

3. Method

  • A pretrained auto-decoder learns shape representations, then a diffusion model is trained on its latent space to generate codes that the decoder turns into neural implicit shapes.This two-stage design decouples the shape representation from the output representation used by the DDPM.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Auto-decoder for Neural Implicit 3D Shapes

  • The auto-decoder represents each shape with a learned latent vector and neural SDF, optimizing shape codes and network parameters with clamped SDF reconstruction and latent regularization.Training points are sampled uniformly in the shape bounding box or near the surface; distance values are clamped because accurate predictions far from the surface are unnecessary.Source: §3.1 · p. 3

3.2. Latent Diffusion Model

  • At generation time, the reverse process starts from a standard-normal latent and iteratively produces a final shape code, which the pretrained auto-decoder decodes into an implicit shape.The forward process is designed to make the latent distribution indistinguishable from standard normal after T steps.Source: §3.2 · pp. 3–4, §3 · p. 3, Figure 2 · p. 4
  • The diffusion model learns from shape latents by predicting total noise; its forward process adds scheduled Gaussian noise, and its reverse process iteratively produces a latent for decoding.The reverse-process mean is predicted by a neural network, while the variance is computed from the variance schedule.Source: §3.2 · pp. 3–4

3.3. Conditional Generation

  • Conditional generation concatenates a shared CLIP image or text embedding with the shape latent as input to the denoising network; training uses embeddings of randomly rendered dataset shapes.The shared text-image embedding space allows the image-trained model to be conditioned on text prompts as well.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

4. Results

The experiments assess unconditional generation, CLIP-conditioned image and text generation, and guided variation of existing shapes.

4.1. Unconditional Generation

  • 3D-LDM performs close to PointDiffusion while providing accurate neural implicit surfaces, and improves upon PVD and SPAGHETTI.The comparison covers unconditional generation; the supplied caption identifies three baselines and three ablations.Source: Table 1 · p. 6, §4.1 · p. 6

4.2. Conditional Generation

  • CLIP-conditioned generation produces shapes matching descriptive text prompts and similar to input chair images, with multiple samples per prompt or image.Text prompts include descriptions beyond object categories; image conditioning uses chair renderings.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7

4.3. Guided Shape Exploration

  • Moderately high noise yields local styling variations; with 2000 noise steps, examples preserve the input's coarse shape while changing details.The amount of noise controls similarity to the initial shape: near-maximum noise produces unrelated shapes, while very little leaves the shape identical.Source: Figure 7 · p. 7, §4.3 · p. 7

5. Conclusions

3D-LDM generates neural implicit 3D shapes by applying latent diffusion after training a DeepSDF auto-decoder, with performance on par with point-cloud diffusion models. The authors identify multi-category training, textured scenes, and NeRF generation as future directions.

5. Conclusions

  • 3D-LDM trains a DeepSDF auto-decoder followed by latent diffusion, achieving performance on par with point-cloud diffusion models.The two-stage training captures the distribution of the implicit SDF representation.Source: §5 · p. 7
  • Text and image conditioning support conditional generation, while adding and removing noise enables guided shape exploration.Source: §5 · p. 7
  • Future directions include multi-category training, textured-scene generation, and NeRF generation using differentiable volume rendering and multi-view supervision.Source: §5 · pp. 7–8

A. Overview

The appendix outlines implementation details, qualitative comparisons, novelty checks, and diffusion hyperparameter ablations, and states that code and training data will be published.

A. Overview

  • Appendix materials cover implementation, qualitative comparisons, nearest-neighbor novelty checks, hyperparameter ablations, and planned code and data release.Source: A. Overview · p. 10

B. Implementation Details

The implementation uses a DeepSDF-style auto-decoder, an MLP denoiser conditioned on diffusion time, and sampled point data to train the auto-decoder.

B. Implementation Details

  • The auto-decoder follows DeepSDF with eight fully connected layers, 512-dimensional hidden features, and a skip connection, but omits dropout and weight normalization.Its final layer uses a linear output instead of tanh; the authors report improved generated-shape quality and diversity.Source: B. Implementation Details · p. 10
  • The denoiser is an eight-layer, 512-feature MLP with softplus activations and three skip connections that concatenate the first hidden-layer output.Each layer is conditioned on a sinusoidal time-step embedding; later layers independently transform it before adding it to the layer input.Source: B. Implementation Details · pp. 10–11
  • Auto-decoder training uses Adam and 500k precomputed points per shape (5% uniform, the rest near exterior surfaces), sampled in 16k-point batches.Source: B. Implementation Details · p. 11

C. Qualitative Comparison

Qualitative comparisons show shape variety on par with point-cloud diffusion methods, alongside direct surface visualization and point distributions with minimal outliers.

C. Qualitative Comparison

  • Across ten uncurated chair and airplane examples, shape quality is comparable with artifacts in each method; chair samples suggest 3D-LDM includes rarer types.The examples include swivel and deck chairs, while SPAGHETTI examples mainly show more common chair types.Source: C. Qualitative Comparison · p. 11
  • 3D-LDM is on par with point-cloud diffusion methods in shape variety while offering direct surface visualization and point distributions with minimal outliers.Source: C. Qualitative Comparison · p. 11

D. Shape Novelty

The authors assess novelty by comparing generated shapes with their nearest training-set neighbors using cosine similarity. They report clear differences between the generated shapes and those neighbors.

D. Shape Novelty

  • Novelty is assessed by comparing five randomly chosen generated shapes with their five nearest neighbors in the training set.Source: D. Shape Novelty · p. 11
  • Nearest neighbors are identified by maximum cosine similarity between generated and training-set latent codes.Source: D. Shape Novelty · p. 11
  • The sampled generated shapes differ clearly from their nearest training-set neighbors, which the authors interpret as evidence of novelty.The comparison uses five randomly chosen generated shapes and their five nearest neighbors.Source: D. Shape Novelty · p. 11

E. Hyperparameter Ablations

The ablations examine denoising steps, batch size, and learning rate. The authors report improved shape quality with 30k rather than 10k steps and select settings using generated-chair metrics.

E. Hyperparameter Ablations

Loading 2212.00842v2…