Source-linked AI summary
3D Neural Field Generation using Triplane Diffusion
J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, Gordon Wetzstein
TL;DR
Existing 3D diffusion methods have limited quality and resolution, motivating a representation that can use powerful 2D diffusion models while retaining 3D structure. The paper factors occupancy fields into triplane features, trains diffusion on those 2D feature maps, and reports high-fidelity, diverse generation that outperforms existing 3D-aware models. Its main practical limitation is the computational cost and slow evaluation of diffusion models.
Problem
Existing 3D diffusion methods operate on discrete point clouds and exhibit limited quality and resolution.
Method
The method factors 3D scenes into triplane features with a shared decoder, then trains a 2D diffusion model to generate those features as 3D neural fields.
Results
The authors report high-fidelity and diverse 3D scene generation that outperforms state-of-the-art 3D GANs and improves object quality and diversity over existing 3D-aware generative models.
Takeaways & Limitations
Triplane feature maps connect 2D diffusion backbones with 3D-aware neural-field generation while preserving a built-in 3D inductive bias.
Takeaways & Limitations
Training and evaluating the diffusion model are computationally demanding and slower than real-time GAN evaluation after training.
Abstract
from arXiv · showhide
Diffusion models have emerged as the state-of-the-art for image generation, among other tasks. Here, we present an efficient diffusion-based model for 3D-aware generation of neural fields. Our approach pre-processes training data, such as ShapeNet meshes, by converting them to continuous occupancy fields and factoring them into a set of axis-aligned triplane feature representations. Thus, our 3D training scenes are all represented by 2D feature planes, and we can directly train existing 2D diffusion models on these representations to generate 3D neural fields with high quality and diversity, outperforming alternative approaches to 3D-aware generation. Our approach requires essential modifications to existing triplane factorization pipelines to make the resulting features easy to learn for the diffusion model. We demonstrate state-of-the-art results on 3D generation on several object classes from ShapeNet.
1. Introduction
The paper addresses limited quality and resolution in existing 3D diffusion by representing scenes as triplanes and training a 2D diffusion model on those features. This framework generates high-fidelity, diverse 3D scenes and supports interpolation between generated shapes.
- Existing 3D diffusion methods operate on discrete point clouds and exhibit limited quality and resolution.
- The framework also supports interpolation between generated objects, including objects with different topologies.
- Triplanes represent 3D scenes as axis-aligned 2D feature planes, making them compatible with conventional 2D generator architectures.
- The framework factors each training scene into per-scene triplane features and a shared decoder, then trains a 2D diffusion model on those triplanes.
- The method provides a 3D diffusion framework using 2D diffusion backbones while retaining a built-in 3D inductive bias.
- The authors report high-fidelity, diverse 3D scene generation that outperforms state-of-the-art 3D GANs.
2. Related Work
Prior work established efficient triplane-based neural fields and strong 2D generator backbones, while existing 3D diffusion methods remained limited by discrete representations or weak 3D operations. The paper builds on this gap by applying diffusion directly to triplane features.
- Neural fields represent scenes as continuous functions and can scale better with scene complexity than discrete representations.
- Hybrid triplane representations use efficient 2D feature planes with a small MLP decoder and scale with surface area rather than volume.
- Diffusion models offer stable training and better coverage of the training distribution, whereas GANs can suffer from mode collapse.
- Existing 3D GANs often synthesize triplanes with expressive 2D generator backbones and decode them using small, efficient MLPs.
- Current 3D diffusion methods use single latents or discrete point clouds, and some do not perform diffusion directly in 3D.
3. Triplane Diffusion Framework
The framework represents 3D occupancy fields with triplane features and a shared decoder, then trains a 2D diffusion model to generate novel triplanes. Regularization makes these features more suitable for diffusion, whose outputs are decoded into neural fields and meshes.
- 3.1. Representing a 3D Scene using a Triplane: Neural fields map 3D coordinates to continuous signals; this work focuses on binary occupancy fields indicating whether points lie inside or outside objects.
- 3.1. Representing a 3D Scene using a Triplane: A triplane uses three axis-aligned 2D feature planes, whose sampled features are summed and passed through an MLP decoder to query the neural field.The planes are fxy, fxz, and fyz, each with resolution N × N and C feature channels.
- 3.2. Representing a Class of Objects with Triplanes: Feature planes for many objects are jointly optimized with one shared decoder so that generated triplanes can be decoded at inference.Training triplanes and decoders independently would produce object-specific MLPs that fail to generalize to generated triplanes.
- 3.3. Regularizing Triplanes for Effective Generalization: The naive triplane-training procedure produces generated shapes with significant artifacts, motivating regularization during feature optimization.
- 3.3. Regularizing Triplanes for Effective Generalization: Total variation regularization suppresses spurious high-frequency feature information, while L2 regularization discourages outlying values before diffusion training.The L2 term addresses sensitivity to outliers introduced by normalization to the DDPM input range [-1,1].
- 3.4. Training a Diffusion Model for Triplane Features: The diffusion model denoises stacked triplane channels from Gaussian noise, then denormalizes and splits the result into three planes decoded by the pretrained MLP.At inference, marching cubes extracts meshes from the resulting neural fields.
4. Experiments
Experiments show that NFD generates detailed and diverse ShapeNet objects, outperforming baselines across render-quality metrics. Ablations further show that density and triplane regularization are necessary for artifact-free generated shapes.
- Qualitative evaluation: The model generates sharp, finely detailed, and diverse objects across the three ShapeNet categories.Generated samples include delicate structures such as car suspensions, chair slats, and airplane armaments, including rare varieties.
- Qualitative evaluation: Compared with PVD, NFD preserves fine elements that a fixed-size 2048-point representation cannot synthesize.PVD produces varied but coarse shapes because its point-cloud representation is limited to 2048 points.
- Qualitative evaluation: Compared with SDF-StyleGAN, NFD better defines fine structures and apparently produces greater object diversity.Examples include the car tire–wheel-well gap, airplane tails and engines, and chair legs and planks.
- Quantitative evaluation: NFD achieves state-of-the-art FID, precision, and recall across Cars, Chairs, and Planes.FID measures overall quality, while precision and recall measure fidelity and diversity independently.
- Latent-space interpolation: Smooth interpolation between noise triplanes produces semantically meaningful interpolation between generated neural fields.The model learns a continuous latent space of triplanes and uses deterministic DDIM sampling with spherical interpolation.
- Ablation studies: Without explicit density regularization, optimizing even a single triplane can overfit sparse coordinate–occupancy data and produce floater artifacts.The density regularization ablation directly examines this failure in a single-shape occupancy field.
- Ablation studies: TV, L2, and density regularization smooth triplane features and enable diffusion models to generate convincing, artifact-free shapes.Unregularized features contain high-frequency artifacts that form a difficult manifold for diffusion learning, whereas regularized features are easier to model.
5. Discussion
The paper presents a 3D-aware diffusion model that generates triplane feature maps assembled into 3D neural fields, improving generated-object quality and diversity. It also identifies computational cost and broader extensions as important considerations.
- The model uses a 2D diffusion backbone to generate triplane feature maps that are assembled into 3D neural fields.
- The approach improves the quality and diversity of generated objects over existing 3D-aware generative models by a large margin.
- Limitations: Diffusion training is slow and computationally demanding, and inference is slower than real-time evaluation of trained GANs.
- Future Work: The method could extend beyond occupancy fields to other triplane-representable neural fields, including radiance fields.
- Ethical Considerations: The authors warn that generative models could be extended to create DeepFakes posing societal threats.
- Conclusion: The work connects powerful 2D diffusion models with 3D object synthesis as a step toward diffusion-based 3D generation.
1. Implementation Details
The implementation converts ShapeNet objects into occupancy-based triplane representations, learns a shared decoder, and trains diffusion models on normalized triplane features. Separate models are trained for cars, chairs, and planes.
- Data: The dataset contains 7496 cars, 4971 chairs, and 4045 planes from ShapeNet V1, with a separate model trained for each class.
- Watertighting: Meshes are watertighted, rendered from 20 evenly spaced views, and processed with marching cubes before occupancy computation.
- Triplane Features: Triplane features have dimension 128×128×32×3, balancing decoded-mesh degradation against time and memory constraints.
- Shared MLP: The shared decoder is a lightweight MLP with Fourier feature mapping followed by three 128-dimensional ReLU layers.
- Training: Triplanes and the MLP are trained jointly on a subset, then per object with the MLP frozen; resulting triplanes serve as pseudo-ground-truth images.
- Normalization: Feature channels are zero-centered, clipped to 16 standard deviations, and scaled to [−1, 1] before diffusion training.
2. Triplane Regularization
The paper regularizes triplane features to suppress high-frequency artifacts and make their gradient statistics resemble those of natural images, improving their suitability for 2D diffusion.
- Supplementary Figure 1 compares gradient distributions for natural images and triplane features with and without regularization.
- Without regularization, autodecoder-trained triplanes contain high-frequency artifacts; total-variation regularization produces smoother features.
- Regularized triplane gradients closely resemble gradients in natural images modeled by a hyper-Laplacian with α = 0.5.
3. Generated Samples
The paper reports uncurated samples generated for three ShapeNet object categories: chairs, planes, and cars. Each category is illustrated with a set of 96 generated samples.
- Chairs: The chairs model generates a set of 96 uncurated samples.
- Planes: The planes model generates a set of 96 uncurated samples.
- Cars: The cars model generates a set of 96 uncurated samples.
4. Comparison to Implicit-Grid [2] Baseline
The comparison evaluates the proposed method against the Implicit-Grid baseline using generated shapes and standard quality and diversity metrics. The proposed method outperforms all baselines in FID, precision, and recall.
- Supplementary Figure 5 shows shapes generated by the Implicit-Grid baseline method.
- The proposed method is compared with baseline methods using FID, precision, and recall.
- The proposed method outperforms all baselines in FID, precision, and recall.The reported results indicate high-quality and diverse 3D shape generation.