Source-linked AI summary
Shap-E: Generating Conditional 3D Implicit Functions
Heewoo Jun, Alex Nichol
TL;DR
3D generation needs representations that are efficient to generate and useful across downstream applications. Shap·E encodes 3D assets into implicit-function parameters and models them with conditional diffusion, generating diverse assets quickly; compared with Point·E, it converges faster with comparable or better quality.
Problem
3D assets lack a representation that is both efficient to generate and easy to use in downstream applications.
Method
Shap·E trains an encoder to map 3D assets to implicit-function parameters, then trains a conditional diffusion model over those parameters for NeRF and textured-mesh representations.
Results
Shap·E generates diverse, recognizable text-conditioned 3D samples and converges faster than Point·E while achieving comparable or superior results.
Takeaways & Limitations
Generating implicit representations can provide more flexibility than explicit representations, including rendering as both NeRFs and textured meshes.
Takeaways & Limitations
The model has limited ability to compose multiple concepts, often produces rough or low-detail samples, and depends on a separate encoder model.
Abstract
from arXiv · showhide
We present Shap-E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap-E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap-E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point-E, an explicit generative model over point clouds, Shap-E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at https://github.com/openai/shap-e.
1 Introduction
Shap·E addresses the challenge of representing 3D assets for generation by modeling implicit functions that support multiple renderable representations. It produces diverse text-conditioned assets quickly and performs comparably or better than Point·E.
- Motivation: 3D assets lack a representation that is both efficient to generate and easy to use downstream.Fixed-size tensor representations are natural for images and audio, but their suitability for 3D assets is less clear.
- Representation: Implicit neural representations encode location-specific 3D information, can be queried at arbitrary points, and support differentiable downstream applications.Examples include NeRFs for densities and colors and mesh-oriented functions for colors, signed distances, and vertex offsets.
- Method: Shap·E trains an encoder to produce implicit-function parameters, then trains a diffusion model on those encoded outputs.Its implicit functions represent NeRFs and textured meshes simultaneously, enabling multiple rendering paths and downstream 3D use.
- Results: Several million 3D assets enable Shap·E to generate diverse, recognizable samples conditioned on text prompts.Figure 1 reports roughly 13 seconds per sample on a single NVIDIA V100 GPU without a separate text-to-image model.
- Results: Compared with Point·E, Shap·E converges faster and achieves comparable or superior results using the same architecture, datasets, and conditioning mechanisms.Its sample quality remains below optimization-based text-conditional methods, but inference is orders of magnitude faster.
2 Background
This section introduces implicit representations for 3D assets and the diffusion-modeling framework used to generate them. It covers NeRFs, textured implicit meshes, volumetric rendering, and conditional latent diffusion.
- 2.1 Neural Radiance Fields (NeRF): Neural radiance fields map coordinates and viewing directions to densities and RGB colors, enabling rendering from arbitrary views.NeRFs query these values along camera rays and can be trained against ground-truth scene renderings.
- 2.1 Neural Radiance Fields (NeRF): NeRF rendering approximates ray integrals by sampling increasing timesteps and accumulating contributions over intervals between samples.Coarse weights can define a piecewise-constant density PDF for finer sampling near high-density regions.
- 2.2 Signed Distance Functions and Texture Fields (STF): Signed distance functions define shape boundaries as the zero level set, allowing meshes to be constructed with marching cubes or marching tetrahedra.The sign also determines normal orientation along the boundary.
- 2.2 Signed Distance Functions and Texture Fields (STF): DMTet and GET3D represent textured meshes with implicit functions that predict signed distances, colors, and vertex offsets for differentiable mesh rendering.The resulting meshes can be rendered efficiently with differentiable rasterization.
- 2.3 Diffusion Models: Diffusion models learn to generate continuous vectors by progressively adding Gaussian noise and then denoising samples back toward the data distribution.This work predicts the denoised sample directly and uses the Heun sampler for quality with reasonable latency.
- 2.3 Diffusion Models: Classifier-free guidance improves conditional sample coherence at the cost of diversity, with guidance scales above one typically producing more coherent samples.The authors use guidance for all models and report that it is necessary for their best results.
3 Related Work
Related 3D-generation methods model explicit representations, implicit decoders, per-example latent codes, or individually optimized samples. These approaches differ from Shap-E in representation and generation procedure, with optimization-based methods requiring repeated expensive work per sample.
- 3 Related Work: Earlier 3D generative models use auto-encoders over point clouds or voxel occupancy grids, followed by latent-space GANs, GMMs, diffusion models, or autoregressive models.These explicit representations are often fixed-resolution or otherwise limited in flexibility.
- 3 Related Work: Other methods train auto-encoders with implicit decoders conditioned on latents derived from SDF grids, voxel grids, rendered views, or related representations.These approaches retain implicit decoding while using explicit or view-based encoders.
- 3 Related Work: Auto-decoder and meta-learning approaches produce latent-conditional implicit representations without a learned encoder by using per-example embeddings or shared parameters.Examples include implicit SDF and NeRF decoders and meta-learned implicit functions.
- 3 Related Work: Gradient-based 3D-generation methods optimize NeRFs, meshes, voxel grids, or related parameters to match text prompts or image-diffusion objectives.Their common shortcoming is expensive optimization that must be repeated for every generated sample.
4 Method
Shap·E trains an encoder to map dense 3D asset representations into MLP parameters, then trains conditional diffusion models over the resulting latent sequences. The implicit outputs support both NeRF and textured-mesh rendering through staged objectives and fine-tuning.
- Overall method: The method first encodes dense explicit 3D assets into latent representations that are linearly projected into MLP weights, then trains a conditional diffusion prior over those latents.The diffusion model supports image or text conditioning.
- Data: The models are trained on 3D assets paired with renderings, point clouds, and text captions, using an expanded text dataset with roughly 1 million additional assets and 120K human captions.The expanded text dataset randomly selects between human-provided and original captions when both are available.
- Encoder: The encoder processes point clouds and rendered views with cross-attention and a transformer backbone, producing latent vectors that map to rows of the MLP weight matrices.The MLP represents the asset as an implicit function.
- NeRF rendering: NeRF pre-training uses sampled rays and an L1 color loss plus a transmittance loss, with shared parameters between coarse and fine rendering stages.The encoder is first trained only with the NeRF objective because it was more stable than mesh-based objectives.
- STF rendering: STF heads predict signed distances and texture colors; meshes are extracted from SDF values on a regular 128^3 grid, textured, and rendered differentiably.The heads are initialized by distillation before end-to-end fine-tuning with both NeRF and STF rendering.
- Latent diffusion: The diffusion model uses 1024 tokens for 1024×1024 latent sequences and directly predicts x0, while retaining Point·E conditioning strategies and roughly equivalent base-model compute.The output space is higher-dimensional because each token generates more input and output channels.
5 Results
Shap·E’s encoder and diffusion model are evaluated against Point·E across reconstruction, text-conditional, and image-conditional settings. The results show strong representation quality, improved text-conditional metrics, and similar image-conditional performance, with distinct qualitative behaviors.
- Encoder Evaluation: The encoder is evaluated using PSNR on reconstructions and CLIP R-Precision on reconstructed NeRF and STF renders, including Point·E 1B samples.Table 1 tracks these metrics after each training stage; Point·E 1B peak CLIP R-Precision is roughly 46.8%.
- Encoder Evaluation: Distillation hurts NeRF reconstruction quality, while fine-tuning recovers and slightly improves NeRF quality and substantially improves STF render quality.
- Comparison to Point·E: In text-conditional evaluation, Shap·E improves on both CLIP R-Precision and CLIP score over the comparable Point·E model.The comparison uses the same base diffusion-model architecture, dataset, and conditioning modes, although the models have different auxiliary components.
- Comparison to Point·E: Text-conditional Shap·E and Point·E often produce qualitatively different samples for the same prompts, despite using the same dataset and base model size.Shap·E’s text-conditional evaluations begin worsening before training ends, which the authors attribute to likely overfitting to captions.
- Comparison to Point·E: Image-conditional Shap·E and Point·E reach roughly the same final performance, with slight metric advantages differing by CLIP R-Precision and CLIP score.Their samples also share similar failure cases, while some differences appear in modeling thin features such as bench slits.
- Comparison to Other Methods: Shap·E has faster inference than Point·E because it does not require an additional upsampling diffusion model.
6 Limitations and Future Work
Shap·E remains limited in compositional text understanding and fine visual detail, motivating improved data, encoders, and combinations with optimization-based methods.
- Limitations: Text-conditional Shap·E struggles to bind multiple attributes to different objects and reliably count more than two objects.These failures are likely related to limited paired training data and could potentially be alleviated with larger annotated 3D datasets.
- Evaluation Context: Table 2 compares 3D generation techniques using CLIP R-Precision on COCO evaluation prompts while also reporting compute and sampling latency.The caption notes that the best text-conditional results use an expanded 3D-asset dataset.
- Limitations: Generated assets are often recognizable but rough or lacking fine details.The encoder can lose detailed textures, such as cactus stripes, suggesting improved encoders could recover some generation quality.
- Future Work: Shap·E could potentially be combined with optimization-based 3D generative techniques for better results.A generated NeRF or mesh could initialize methods such as DreamFusion, potentially leading to faster convergence.
7 Conclusion
Shap·E generates latent 3D implicit functions renderable as NeRFs and textured meshes, matching or outperforming a comparable explicit model under matched conditions. Its text-conditional models produce diverse objects without image intermediates, highlighting the flexibility of implicit representations for 3D.
- Shap·E is a latent diffusion model over 3D implicit functions renderable as both NeRFs and textured meshes.
- Shap·E matches or outperforms a similar explicit generative model given the same dataset, architecture, and training compute.
- Pure text-conditional Shap·E models generate diverse, interesting objects without relying on images as intermediate representations.
- The results highlight the potential of generating implicit representations where 3D applications benefit from greater flexibility than explicit representations.
A.1 Encoder Architecture
The encoder combines point-cloud and multiview inputs through cross-attention and a Transformer, producing latent vectors that parameterize shared-architecture implicit functions. Its design includes diffusion noise, sinusoidal coordinate embeddings, and view-consistency handling.
- Input Representations: The encoder receives an RGB point cloud and a multiview point cloud built from 20 rendered 256 × 256 views with surface coordinates.The point cloud contains 16,384 points, while patch embedding produces 20,480 multiview vectors.
- Encoder Architecture: Point-cloud embeddings are cross-attended with learned queries, followed by cross-attention to multiview patches and a Transformer that outputs 1K latent vectors.
- Latent Processing: Gaussian diffusion noise is applied to latents with probability 0.1, after which projected latents are stacked into four 256 × 256 MLP weight matrices.The encoder uses the schedule ¯αt = 1 −t5 for this noise injection.
- Implicit Function: The implicit representation uses a 6-layer bias-free MLP whose first four layers are encoder-determined and whose final two layers are shared across examples.Input coordinates receive sinusoidal embeddings.
- View Consistency: Ray-direction embeddings stabilize encoder training, but setting them to zero at test time is used to enforce view consistency.Because density can depend on ray direction during training, the test-time setting is out-of-distribution but effective.
A.5 Evaluation
For CLIP-based evaluation, samples are rendered as NeRFs at 128 × 128 from randomly sampled camera positions around the z-axis with fixed 30° elevation.
- Rendering Protocol: CLIP-based metrics use 128 × 128 NeRF renders from random cameras around the z-axis at constant 30 degree elevation.This setup is chosen because training objects are usually oriented with the z-axis as their logical vertical direction.
B Overfitting in Text-Conditional Models
The text-conditional model begins to worsen on evaluations after roughly 600K iterations, with overfitting particularly pronounced at the noisiest diffusion timesteps. Guidance-scale sweeps also show different saturation behavior for Shap·E and Point·E.
- Roughly 600K iterations marks the point after which the text-conditional model begins to worsen on evaluations.
- Shap·E benefits from increasing guidance scale up to 20.0, whereas Point·E saturates at lower scales and then becomes worse.
- The training loss decreases faster than the validation loss, although the overall validation loss never starts increasing.
- Overfitting is stronger for the noisiest diffusion timesteps, where validation loss begins increasing at about 600K iterations.
C Bias and Misuse
The text-conditional model can reproduce dataset biases and generate objects whose real-world use may be harmful without validation. Its limited photorealism and prompt-following ability reduces some risks but does not eliminate misuse concerns.
- Ambiguous captions produce samples exhibiting common gender-role stereotypes, indicating that dataset biases affect the text-conditional model’s behavior.
- The models are not typically adept at producing photo-realistic samples or accurately following long and complex prompts.
- Limited photorealism alleviates concerns about creating convincing DeepFakes, but fabrication methods such as 3D printing introduce other risks.
- Generated 3D objects could be introduced into the real world without adequate validation or safety testing and could potentially be harmful when they do not meet the desired prompt.
D Guidance in Image Space
Shap·E’s latent diffusion process can be guided using image-space gradients computed from rendered NeRF images. DreamFusion guidance makes samples more closely match text prompts despite limited standalone DDPM performance.
- A noised latent is decoded into a model prediction, rendered with NeRF, and differentiated through an image-based objective to obtain an image-space gradient.
- The diffusion update follows classifier guidance by moving each step in the direction of a scaled gradient s · ∂L.
- DreamFusion supplies image-space gradients that incentivize rendered views of Shap·E samples to match a text prompt.
- With s = 0, text-conditional Shap·E often fails to capture prompts, while increasing s makes samples more closely match them.