Source-linked AI summary
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models
Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, Xinggang Wang
TL;DR
3D diffusion models provide consistency but face limited data and generalization, while 2D diffusion models offer detailed generation but do not guarantee 3D consistency. GaussianDreamer bridges them through 3D Gaussian splatting, generating detailed, consistent 3D assets within 15 minutes on one GPU with direct real-time rendering.
Problem
3D diffusion models have strong 3D consistency but limited quality and generalization because 3D data is scarce and expensive, whereas 2D diffusion models do not guarantee 3D consistency.
Method
GaussianDreamer bridges 3D and 2D diffusion models through Gaussian splatting, using 3D diffusion priors for initialization and noisy point growing and color perturbation to enrich the Gaussians.
Results
GaussianDreamer generates detailed and realistic 3D geometry and appearance while maintaining 3D consistency, with each sample generated within 15 minutes on one GPU and directly rendered in real time.
Takeaways & Limitations
Bridging 3D and 2D diffusion models through Gaussian splatting is presented as a promising direction for efficient 3D asset generation.
Takeaways & Limitations
The method has limited effectiveness for large-scale scenes and can still produce multi-face problems when front and back appearances differ despite minimal geometric differences.
Abstract
from arXiv · showhide
In recent times, the generation of 3D assets from text prompts has shown impressive results. Both 2D and 3D diffusion models can help generate decent 3D objects based on prompts. 3D diffusion models have good 3D consistency, but their quality and generalization are limited as trainable 3D data is expensive and hard to obtain. 2D diffusion models enjoy strong abilities of generalization and fine generation, but 3D consistency is hard to guarantee. This paper attempts to bridge the power from the two types of diffusion models via the recent explicit and efficient 3D Gaussian splatting representation. A fast 3D object generation framework, named as GaussianDreamer, is proposed, where the 3D diffusion model provides priors for initialization and the 2D diffusion model enriches the geometry and appearance. Operations of noisy point growing and color perturbation are introduced to enhance the initialized Gaussians. Our GaussianDreamer can generate a high-quality 3D instance or 3D avatar within 15 minutes on one GPU, much faster than previous methods, while the generated instances can be directly rendered in real time. Demos and code are available at https://taoranyi.com/gaussiandreamer/.
1. Introduction
GaussianDreamer bridges 3D and 2D diffusion models through 3D Gaussian Splatting to combine 3D consistency with detailed generation. It uses 3D priors for initialization and 2D diffusion for refinement.
- Motivation: 3D diffusion models provide strong consistency but are limited by scarce, expensive 3D data and weaker coverage of complex prompts, geometry, and appearance.Current 3D datasets are much smaller than 2D datasets.
- Motivation: 2D diffusion models handle diverse prompts and detailed geometry and appearance, but camera-view unawareness makes consistent 3D geometry difficult.The issue is especially pronounced for structurally complicated instances.
- Approach: GaussianDreamer bridges both approaches with an explicit, efficient 3D Gaussian representation intended to combine geometry consistency and rich visual details.Its Gaussian representation has geometry priors from its point-cloud-like structure.
- Approach: The framework uses text-to-3D or text-to-motion diffusion models to generate a coarse instance, then enriches initialized Gaussians with noisy point growing and color perturbation.The method subsequently optimizes the Gaussians with a 2D diffusion model.
- Outcome: 15 minutes on one GPU generates a 3D instance that can be rendered in real time, making GaussianDreamer substantially faster than previous methods.The method is described as simple and effective.
2. Related Works
Related work spans 3D pretrained diffusion, 2D-to-3D lifting, and 3D representations. GaussianDreamer builds on these directions by using Gaussian Splatting for fast text-to-3D generation.
- 3D Diffusion Models: 3D pretrained diffusion models generate 3D assets or motion from text-3D or text-motion pairs, but motion outputs lack texture information before further processing.Examples include Point-E, Shape-E, and text-to-motion models converted to SMPL.
- Lifting 2D Diffusion Models to 3D: Lifting 2D diffusion models to 3D is training-free and benefits from abundant image data, but generated assets may suffer from multi-face problems.SDS updates a 3D representation using a 2D diffusion model.
- 3D Representation Methods: Prior 3D generation methods use NeRF, explicit meshes, or point clouds, with mesh representations reducing rendering costs and improving resolution.These approaches form the broader representation context for text-to-3D generation.
- 3D Representation Methods: 3D Gaussian Splatting offers NeRF-comparable rendering with real-time performance and supports faster optimization than optimizable mesh representations.Concurrent work applies Gaussian Splatting to image-conditioned and text-conditioned 3D generation.
- GaussianDreamer: GaussianDreamer’s pipeline initializes point clouds with a 3D diffusion model, enriches them, optimizes Gaussians with SDS and 2D diffusion, and renders them with Gaussian Splatting.Text-to-3D and text-to-motion diffusion models serve as example initialization sources.
3. Method
GaussianDreamer initializes explicit 3D Gaussians from 3D diffusion priors, enriches them through noisy point growing and color perturbation, and optimizes them with a 2D diffusion model. The resulting representation combines 3D-consistent initialization with detailed geometry and appearance and supports rapid real-time rendering.
- 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes with anisotropic Gaussians and renders them through splatting for real-time novel-view synthesis.Each Gaussian stores position, covariance, color, and opacity; ray colors are computed by compositing intersected Gaussians.
- Framework overview: The framework uses 3D diffusion models to generate coarse assets, transforms them into point clouds, and initializes 3D Gaussians from the processed points.The two-stage framework pairs 3D-diffusion initialization with subsequent optimization using a 2D diffusion model.
- Text-to-3D diffusion initialization: For text-to-3D initialization, mesh vertices and colors are converted into point clouds, whose sparse positions and simple colors motivate further enrichment.The mesh is constructed by querying SDF values and texture colors on a regular grid of size 128^3.
- Noisy point growing and color perturbation: Noisy point growing samples points within the surface bounding box, retains points within normalized distance 0.01 of mesh points, and assigns perturbed nearby colors.A KDTree supports nearest-point search, and perturbations are sampled between 0 and 0.2 before merging the original and grown point clouds.
- Gaussian initialization: The final point-cloud positions and colors initialize Gaussian centers and colors, while opacity is set to 0.1 and covariance uses the nearest-two-point distance.The initialization is written as θb(µb, cb, Σb, αb), with centers and colors taken from the enriched point cloud.
- Text-to-motion diffusion initialization: For text-to-motion inputs, a text-conditioned motion sequence supplies a pose that is converted to an SMPL mesh and then to point clouds for Gaussian initialization.The motion prompt may be simplified to retain motion-relevant content before generating the sequence; SMPL colors are randomly initialized because the model lacks textures.
4. Experiments
GaussianDreamer is evaluated quantitatively, qualitatively, and through ablations, showing fast generation, detailed and consistent assets, and benefits from diffusion-based initialization and Gaussian enrichment.
- Implementation Details: 15 minutes on one RTX 3090 is sufficient for all experiments, with 1024 × 1024 rendering optimized at 512 × 512 and real-time rendering at 512 × 512.The implementation uses 1200 training iterations and batch size 4.
- Quantitative Comparison: GaussianDreamer outperforms compared methods on T3Bench while maintaining a short generation time across single-object, surroundings, and multi-object categories.T3Bench evaluates quality and consistency across these three increasing-complexity text categories.
- Text-to-3D Initialization: 21−24 times less generation time than DreamFusion, Magic3D, Fantasia3D, and ProlificDreamer is reported with comparable quality for text-to-3D initialization.For the prompt involving a plate and chocolate-chip cookies, GaussianDreamer includes both objects while several compared methods omit the plate.
- Text-to-Motion Initialization: 4−24 times speedup over DreamFusion, DreamAvatar, DreamWaltz, and AvatarVerse is reported for avatars while maintaining comparable quality.Text-to-motion initialization also enables generation with specified body poses, including poses selected from generated motion sequences.
- Initialization Ablation: 3D diffusion initialization improves consistency and geometry by avoiding unreasonable multi-head geometry and helping outputs remain closer to prompts than random initialization.Compared with Shap-E alone, GaussianDreamer produces more complex geometries and realistic appearances in the shown examples.
- Point Growing and Color Perturbation: Noisy point growing and color perturbation improve sniper-rifle details and make an amigurumi motorcycle better match the prompt’s style.The ablation compares results with and without the combined operations, denoted “Grow&Pertb.”
5. Conclusion
GaussianDreamer bridges 3D and 2D diffusion models through Gaussian splatting to generate detailed, realistic, and 3D-consistent assets efficiently. Each sample can be generated within 15 minutes on one GPU.
- GaussianDreamer bridges 3D and 2D diffusion models through the Gaussian splatting representation.The method combines 3D diffusion priors with 2D diffusion capabilities.
- GaussianDreamer generates detailed and realistic geometry and appearance while maintaining 3D consistency.
- 15 minutes is sufficient to generate each sample on one GPU.The paper attributes convergence-speed improvements to 3D diffusion model priors and geometry priors from 3D Gaussians.
A.1. More Results
The additional results examine quantitative similarity, diverse generations, prompt sensitivity, human-model selection, and generation with ground or fine-grained prompts. They show that GaussianDreamer can match refined prompts and produce varied assets, while supporting real-time Gaussian-splatting outputs.
- Table 2 evaluates GaussianDreamer and other methods using CLIP similarity.The comparisons use results from Instant3D, official Shap-E results, and threestudio implementations of DreamFusion and ProlificDreamer.
- GaussianDreamer produces more details than Instant3D, while its CLIP similarity increases only marginally.
- Diversity: Different random seeds generate diverse 3D assets for the same prompt.
- Generation with More Fine-grained Prompts: GaussianDreamer better matches fine-grained prompts than Shap-E, which produces similar axe results across different descriptions.
- Automatically Select A Human Model: CLIP-guided selection of initialized human body models produces good rendering effects across various human body models.The paper identifies extending these assets to dynamic ones as a possible direction.
A.2. More Ablation Studies
The ablations assess the choice of 2D diffusion model and the point-growth box size. Stable Diffusion 2.1 produces richer details, whereas larger growth boxes yield fatter, blurrier assets.
- 2D Diffusion Model: Stable Diffusion 2.1 produces 3D assets with richer details than DeepFloyd IF in the 2D diffusion-model ablation.The comparison evaluates stabilityai/stable-diffusion-2-1-base and DeepFloyd/IF-I-XL-v1.0.
- Box Size in Point Growth: A larger point-growth box leads to a fatter asset with a blurrier appearance.
A.3. More Discussions
The discussions show how domain-limited 3D diffusion models handle complex prompts and how GaussianDreamer relates to other 3D-generation approaches. Bridging domain-abundant 2D diffusion helps complete missing object parts and add finer details.
- Limitations Introduced by the 3D Datasets: For complex prompts, the domain-limited 3D diffusion model generates only parts of the desired object with rough appearances.
- Limitations Introduced by the 3D Datasets: GaussianDreamer completes remaining object parts and provides finer details by bridging the domain-abundant 2D diffusion model.
- Recent Works: GaussianDreamer directly generates 3DGS-based assets from text by connecting 3D and 2D diffusion models.The discussion contrasts this focus with DreamGaussian, which generates mesh-based assets from an image or text-generated image.
- Recent Works: DreamGaussian is presented as orthogonal to GaussianDreamer, with a possible future combination.