Source-linked AI summary
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, Gang Zeng
TL;DR
Existing SDS-based 3D generation methods can be slow because of per-sample optimization. DreamGaussian uses generative Gaussian Splatting with mesh extraction and UV-space texture refinement, producing high-quality textured meshes within minutes while balancing speed and fidelity.
Problem
SDS-based 3D generation methods often require slow per-sample optimization, limiting their practical usage.
Method
DreamGaussian adapts 3D Gaussian Splatting for generative 3D content creation and adds mesh extraction with UV-space texture refinement.
Results
DreamGaussian produces ready-to-use high-quality polygonal meshes from a single image or text description within a few minutes while balancing generation quality and speed.
Takeaways & Limitations
The framework offers efficient 3D content creation with explicit textured meshes for downstream use.
Takeaways & Limitations
The method shares multi-face Janus, over-saturated texture, and baked-lighting problems with previous works, and back-view textures may be blurry in image-to-3D results.
Abstract
from arXiv · showhide
Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods.
1 INTRODUCTION
DreamGaussian targets the slow optimization of SDS-based 3D generation with generative Gaussian Splatting, mesh extraction, and UV-space texture refinement. It produces textured assets quickly while balancing generation speed and fidelity.
- Motivation: SDS-based NeRF methods improve 3D generation quality but can require hours-long optimization because NeRF rendering is costly.This limits their deployment in real-world applications.
- Results: 2 minutes is sufficient to generate photorealistic 3D assets with explicit meshes and texture maps from a single-view image.The image-to-3D pipeline produces a coarse shape within seconds and converges in around 500 steps on one GPU.
- DreamGaussian framework: DreamGaussian adapts 3D Gaussian Splatting to generative 3D content creation and reduces optimization-based generation time.The framework refines design choices in an optimization-based pipeline.
- DreamGaussian framework: The pipeline combines generative Gaussian Splatting with mesh extraction and UV-space texture refinement to improve asset usability and quality.The mesh extraction algorithm converts generated Gaussians into polygonal meshes, while UV refinement enhances texture details.
- Results: Extensive Image-to-3D and Text-to-3D experiments show that DreamGaussian balances optimization time and generation fidelity.The reported balance is intended to support real-world deployment of 3D content generation.
2 RELATED WORK
Related work uses NeRF and diffusion-based approaches for 3D reconstruction and generation, but optimization cost, dataset scarcity, and limited diversity remain important challenges.
- Neural Radiance Fields: NeRF uses volumetric rendering and enables 3D optimization from 2D supervision, but optimizing it can be time-consuming.NeRF has been used in both 3D reconstruction and generation.
- Text-to-3D generation: Text-to-3D generation transfers successful 2D diffusion models to 3D, but large-scale 3D dataset curation remains challenging.Existing 3D-native diffusion models often focus on a single object category and have limited diversity.
- Image-to-3D generation: Image-to-3D generation creates assets from reference images, while conventional single-view reconstruction can produce blurry results because it lacks uncertainty modeling.Text-to-3D methods can also be adapted to image-to-3D generation using image captioning models.
3 OUR APPROACH
DreamGaussian uses generative 3D Gaussian Splatting for efficient initialization, then extracts a textured mesh and refines its UV-space texture. Its two-stage pipeline addresses blurry Gaussian results through efficient mesh extraction and multi-step texture refinement.
- Framework: The framework adapts 3D Gaussian Splatting for efficient initialization through SDS, followed by textured-mesh extraction and UV-space texture refinement.The pipeline uses single-step SDS for Gaussian initialization and multi-step MSE-based refinement for the texture.
- Generative Gaussian Splatting: The method initializes randomly positioned Gaussians and periodically densifies them more frequently than in reconstruction to follow generation progress.The Gaussians are initialized inside a sphere with unit scaling and no rotation, then optimized with SDS from sampled camera poses.
- Generative Gaussian Splatting: SDS supervision can yield blurry Gaussian results because inconsistent guidance makes spatial densification difficult.The method therefore separates initial Gaussian generation from later geometry and texture refinement.
- Efficient Mesh Extraction: DreamGaussian extracts polygonal meshes from Gaussian densities using block-wise local queries, Marching Cubes, and post-processing.The space is divided into overlapping blocks, reducing the number of Gaussians queried locally before constructing a dense grid and extracting the surface.
- Efficient Mesh Extraction: The extracted mesh receives texture through UV unwrapping and back-projection of rendered RGB images from multiple viewpoints.The back-projected texture initializes the subsequent refinement stage, while unstable boundary pixels are excluded.
- UV-space Texture Refinement: Direct UV-space SDS fine-tuning produces artifacts, whereas the proposed multi-step denoising and pixel-wise MSE refinement avoids this problem.The refinement starts from a coarse rendered texture, limits noise strength, and optimizes the texture against the refined image.
4 EXPERIMENTS
Experiments show that DreamGaussian balances generation speed and mesh quality across image-to-3D and text-to-3D comparisons. Its exported meshes support downstream use, while ablations indicate that each tested stage-1 design choice contributes to quality.
- The reported average generation time for DreamGaussian is approximately 2 minutes, compared with ~20 minutes for Zero-1-to-3, ~45 seconds for One-2-3-45, and ~27 seconds for Shap-E.
- DreamGaussian achieves comparable quality to optimization-based methods while being only marginally slower than inference-only methods on image-to-3D tasks.
- DreamGaussian achieves better quality than inference-based methods and faster speed than other optimization-based methods in text-to-3D comparisons.
- The exported meshes have uniform triangulation, smooth surface normals, and clear texture images, supporting rigging and animation in downstream applications.
- Over 10 times acceleration is achieved for image-to-3D generation, although mesh quality falls slightly behind other optimization-based methods.
- Omitting periodical densification, linear timestep annealing, or the reference view loss degrades generated model quality.
5 LIMITATIONS AND CONCLUSION
DreamGaussian produces ready-to-use polygonal meshes from single images or text descriptions within a few minutes, but retains several known limitations of 3D generative models.
- DreamGaussian produces ready-to-use 3D assets with high-quality polygonal meshes from a single image or text description within a few minutes.
- The method shares common problems including the multi-face Janus problem, over-saturated texture, and baked lighting.
- Back-view textures in image-to-3D results may look blurry, a limitation that can be alleviated with longer stage 2 training.
- The optimization-based 2D lifting approach may inherit unintended biases from its 2D diffusion prior models and could affect 3D creative professionals.
A.1 PRELIMINARY
SDS uses a pretrained 2D diffusion model to provide guidance for optimizing a differentiable 3D representation from rendered views. The formulation samples camera poses, diffusion timesteps, and Gaussian noise, then back-propagates denoising guidance through rendering.
- SDS leverages pretrained 2D diffusion models as priors to optimize a differentiable 3D representation such as NeRF.
- The rendered image x is produced from camera pose p by a differentiable rendering function g_Θ with optimizable representation parameters Θ.
- SDS samples a diffusion timestep t, camera pose p, and Gaussian noise ϵ to form its denoising guidance.
- The denoising gradient is back-propagated to the rendered image and then through differentiable rendering to optimize the underlying 3D representation.
- UV mapping projects a 2D texture image onto a polygonal mesh by assigning UV coordinates to mesh vertices and interpolating surface values.
A.2 MORE IMPLEMENTATION DETAILS
The implementation uses staged optimization, Gaussian densification and pruning, mesh simplification, and task-specific evaluation protocols. Additional experiments examine text-to-3D guidance, timestep annealing, and human judgments.
- Optimization: Gaussian splatting uses Adam with parameter-specific learning rates, while mesh texture fine-tuning uses a texture-image learning rate of 0.2.Position learning rate decays from 1 × 10^-3 to 2 × 10^-5 over 500 steps.
- Gaussian processing: Gaussian densification and pruning are controlled by gradient, scaling, and opacity thresholds that differ between image-to-3D and text-to-3D.Text-to-3D uses a gradient threshold of 0.01, while Gaussians below opacity 0.01 or above scaling 0.05 are pruned.
- Mesh processing: Mesh extraction applies isotropic remeshing followed by quadric edge-collapse decimation to control mesh complexity.The mesh is remeshed to average edge length 0.015 and decimated to 10^5 faces.
- Evaluation: Image-to-3D quality is evaluated with CLIP similarity on 30 images rendered from eight uniformly sampled azimuth views at zero elevation.
- Text-to-3D experiments: Text-to-3D experiments use MVDream as a guidance model and evaluate timestep annealing.
- Evaluation: Human evaluation compares four methods using reference-view consistency and overall model quality ratings from 60 volunteers.The study collects 900 valid scores from 60 volunteers.
A.3 MORE RESULTS
DreamGaussian produces comparable image-to-3D quality while reducing time, supports elevated inputs, and enables text-to-image-to-3D and mesh animation workflows. Remaining limitations include blurry back views and text-to-3D artifacts.
- Image-to-3D: Both DreamGaussian stages are faster than previous two-stage image-to-3D methods while reaching comparable generation quality.
- Image-to-3D: DreamGaussian supports image-to-3D inputs with non-zero elevation angles when the estimated elevation is provided.
- Text-to-image-to-3D: Text-to-image-to-3D first synthesizes an image from text and then performs image-to-3D, usually improving results over direct text-to-3D while taking less time.
- Downstream applications: Exported meshes are demonstrated for downstream mesh animation applications.
- Text-to-3D: With MVDream guidance, camera information mitigates the multi-face Janus problem, while timestep annealing yields a more reasonable shape at the same training iterations.
- Limitations: The image-to-3D pipeline may produce blurry back views and fine-detail mismatches, while text-to-3D retains Janus artifacts and baked lighting.Longer stage-2 training can alleviate the blurry back-view problem.