Source-linked AI summary
3DGen: Triplane Latent Diffusion for Textured Mesh Generation
Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, Barlas Oğuz
TL;DR
3DGen addresses the challenge of learning scalable latent representations that jointly model 3D geometry and color. It combines a triplane VAE with a conditional diffusion model, achieving substantial improvements across generation settings while scaling to large datasets for improved quality and diversity.
Problem
Existing methods had not established how to learn scalable 3D latent representations that jointly support geometry and color generation efficiently.
Method
3DGen uses a two-stage pipeline combining a triplane VAE for textured-mesh latents with a diffusion model that generates triplane features.
Results
3DGen substantially improves prior methods across unconditional, text-conditioned, image-conditioned, geometry-only, and colored mesh generation settings, with FiD gains up to 70%.
Takeaways & Limitations
The model generates high-quality textured or untextured meshes across diverse categories in a few seconds on a single GPU, with quality and diversity improving through data scaling.
Takeaways & Limitations
The model remains substantially less general than state-of-the-art image-generation models trained on billions of images.
Abstract
from arXiv · showhide
Latent diffusion models for image generation have crossed a quality threshold which enabled them to achieve mass adoption. Recently, a series of works have made advancements towards replicating this success in the 3D domain, introducing techniques such as point cloud VAE, triplane representation, neural implicit surfaces and differentiable rendering based training. We take another step along this direction, combining these developments in a two-step pipeline consisting of 1) a triplane VAE which can learn latent representations of textured meshes and 2) a conditional diffusion model which generates the triplane features. For the first time this architecture allows conditional and unconditional generation of high quality textured or untextured 3D meshes across multiple diverse categories in a few seconds on a single GPU. It outperforms previous work substantially on image-conditioned and unconditional generation on mesh quality as well as texture generation. Furthermore, we demonstrate the scalability of our model to large datasets for increased quality and diversity. We will release our code and trained models.
1. Introduction
3DGen addresses the challenge of combining scalable 3D representations with joint geometry-and-texture generation through a two-stage triplane VAE and conditional diffusion pipeline. It reports substantial improvements across unconditional, text-conditioned, and textured mesh generation settings.
- Motivation: Existing methods do not clearly establish how to combine scalable latent representations, joint geometry and color modeling, and practical computation for high-quality 3D generation.Prior approaches include multi-category discrete VAEs, single-category models, and methods that omit color.
- Approach: 3DGen encodes optionally colored point clouds into triplane Gaussian latents, reconstructs textured meshes, and diffuses triplane features for image-, text-, or unconditional generation.Both stages decode triplane features into differentiable colored meshes for rendering-based training.
- Results: 23%: 3DGen improves unconditional geometry-generation FiD over NFD, while text-conditioned geometry improvements range from 15-20% over 3DILG.The comparisons cover geometry-only and textured mesh generation in unconditional and image-conditioned settings.
- Scaling: Pre-training on almost half a million 3D objects produces significant improvements in quality and diversity.The model is scaled to a large dataset to evaluate the effect of pre-training.
2. Method
The method represents 3D meshes with triplane latent features and decodes them into differentiable meshes, then trains diffusion models to generate those features. Rendering-based losses support reconstruction and texture learning, while 3D-aware diffusion and conditioning enable generation across settings.
- Representation: Triplane neural fields represent scenes with three axis-aligned feature planes, balancing efficiency and quality for 3D object representation.Volumetric rendering combines trilinear feature interpolation with an MLP decoder in related triplane approaches.
- Triplane VAE: The VAE encoder maps point clouds x ∈ R3×N to triplane latent features z = h_xz, h_xy, h_yz with resolution H×W and C channels.The encoder uses a PointNet followed by a UNet with position embeddings and 3D-aware convolutions.
- Triplane VAE: The decoder predicts signed-distance values and vertex deformations from tetrahedral-grid coordinates and triplane features, then forms a triangulated mesh with differentiable marching tetrahedra.This mesh-decoding procedure directly produces a mesh suitable for downstream applications.
- Training: Rendering-based reconstruction, KL-divergence, and Laplacian smoothing losses train the VAE while preserving details that preprocessing-based SDF or occupancy losses can lose.Differentiable rendering and DMTet make the rendering-based loss possible.
- Texture Prediction: Colored point clouds and a texture field extend the triplane VAE to predict semantically consistent surface colors for arbitrary-topology meshes.The textured model adds a surface-color loss against ground-truth colors.
- Diffusion Model: A UNet diffusion model generates rolled-out triplane features, using 3D-aware convolutions and adaptive group normalization for image-text conditioning.Rolled-out triplanes are treated as 2D images, enabling reuse of image-diffusion techniques.
3. Experiments
Experiments evaluate 3DGen on image- and text-conditioned generation, unconditional geometry and textured mesh generation, and large-scale pre-training. Across these settings, the model improves mesh quality, texture fidelity, and category-specific generation results.
- Image-conditioned generation: 3DGen improves image-conditioned meshes in both ground-truth alignment and overall mesh quality.The evaluation uses Chamfer distance for alignment and shading FiD for overall quality; qualitative comparisons also show greater image faithfulness and geometry detail.
- Pre-training and scalability: Large-scale pre-training on Objaverse and 3D Warehouse produces significant improvements, especially for low-resource categories.The scaling experiment uses nearly half a million filtered Objaverse objects and reports that pre-training improves FiD most substantially in low-resource categories.
- Text-conditioned generation: Text conditioning generates well-formed objects matching descriptions by replacing image embeddings with text embeddings.The paper notes that this simple prompting approach is not always robust, but demonstrates generation flexibility.
- Textured mesh generation: 3DGen outperforms GET3D on all three textured-mesh categories, improving chair FiD from 27.4 to 11.1 and table FiD from 42.2 to 12.3.The generated meshes capture both geometry and texture details; comparisons include GET3Dflat and GET3Dlit variants.
4. Analysis
The analysis examines triplane compatibility, VAE training choices, decoder refinement, and conditioning-encoder scale. These studies identify design choices that improve reconstruction, mesh detail, smoothness, or image alignment.
- The analysis includes dedicated ablations of image-encoder choice and VAE training.
- Triplane features are continuous and 2D in nature, making them more compatible with state-of-the-art image diffusion models.
- Decoder-only finetuning improves fine details and mesh smoothness while avoiding the computational cost of directly training at higher tetrahedral resolution.Increasing inference resolution substantially improves shading FiD, and finetuning the decoder at that resolution provides further gains.
- The conditioning encoder’s size is evaluated across bi-encoder models ranging from 300M to 1.8B parameters for improving generated-mesh alignment.The analysis measures alignment using Chamfer distance on the validation set.
5. Related Work
Related work develops latent representations, diffusion models, and text-image guidance for 3D generation. These approaches span point clouds, shape latents, and optimization-based 3D modeling.
- Latent shape-generation methods commonly train an autoencoder before generating low-dimensional shape representations instead of original 3D data.CLIP-forge is described as learning single-vector shape representations and using a normalizing-flow network to generate them.
- Diffusion models have been extended from image generation to 3D generation, including applications to point clouds.
- Diffusion-based 3D methods generate uncolored point clouds or colored point clouds from CLIP embeddings.Point-E uses a transformer-based diffusion model and shows generalization to diverse point clouds from complex text prompts.
- Another line of work uses large-scale image-text models to supervise 3D generation through rendered-image and text similarity.Examples include DreamField, CLIP-Mesh, and PureCLIPNeRF, while later methods address limitations of CLIP supervision.
6. Conclusion
The paper presents a textured mesh generation model that produces high-quality outputs in a few seconds on current GPU hardware and improves quality and diversity with data scaling. It also identifies a remaining generality gap relative to state-of-the-art image-generation models.
- 3DGen produces high-quality textured meshes in a few seconds on current GPU hardware.
- Data scaling improves the model’s quality and diversity.
- The model remains less general than state-of-the-art image-generation models trained on billions of images.The paper points to 2D image datasets and 2D generative models as directions for closing this gap.