PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, Gang Zeng

arXiv:2309.16653v2cs.CV

TL;DR

SDS-based 3D-generation approaches commonly use NeRF and can require hours-long optimization, restricting real-world deployment. DreamGaussian adapts generative Gaussian Splatting with mesh extraction and UV-space texture refinement, producing photorealistic assets with explicit meshes and texture maps from a single-view image in 2 minutes.

  • Problem

    SDS-based 3D-generation approaches commonly use NeRF and can require hours-long optimization, restricting real-world deployment.

    Source: §1 · p. 2

  • Method

    DreamGaussian adapts 3D Gaussian Splatting for generative 3D creation, then extracts textured meshes and refines textures in UV space.

    Source: §3 · pp. 3–4

  • Results

    2 minutes: DreamGaussian generates photorealistic assets with explicit meshes and texture maps from a single-view image, balancing optimization time and fidelity across Image-to-3D and Text-to-3D experiments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework produces ready-to-use polygonal-mesh assets from either a single image or a text description.

    Source: §5 · p. 9

    The method shares common problems with prior work, including multi-face Janus effects, oversaturated textures, and baked lighting.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10 times acceleration compared to existing methods.

1 INTRODUCTION

DreamGaussian adapts 3D Gaussian Splatting for efficient 3D content generation, pairing it with mesh extraction and UV-space texture refinement; it produces textured meshes from single-view images in about 2 minutes.

Introduction

  • SDS-supervised NeRF methods can take hours to optimize, and occupancy pruning is ineffective under ambiguous generative supervision (Poole et al., 2022; Mildenhall et al., 2020; M¨uller et al., 2022; Sara Fridovich-Keil and Alex Yu et al., 2022).The paper identifies costly NeRF rendering as a barrier to deployment at scale.Source: §1 · p. 2
  • Photorealistic textured 3D assets can be generated from a single-view image in about 2 minutes.The pipeline produces a coarse shape within seconds and converges in around 500 steps on a single GPU.Source: §1 · p. 2
  • DreamGaussian adapts 3D Gaussian Splatting to generative tasks, using progressive densification to simplify optimization and improve generation efficiency.The authors argue densification better matches optimization progress in generative settings than occupancy pruning.Source: §1 · p. 2
  • To address blurry Gaussian outputs, DreamGaussian extracts meshes by local density querying and refines UV textures with image-space supervision to avoid blocky artifacts from latent-space SDS.The refinement stage is presented as improving fidelity while retaining high efficiency.Source: §1 · p. 2
  • Experiments on Image-to-3D and Text-to-3D demonstrate a balance between optimization time and generation fidelity.The authors state that this balance supports real-world deployment of 3D content generation.Source: §1 · p. 2

2 RELATED WORK

The related work frames 3D Gaussian splatting as a faster alternative representation for optimization-based generation, addressing the long optimization times of 2D-lifting methods. DreamGaussian applies it to both text-to-3D and image-to-3D generation.

2.1 3D REPRESENTATIONS

  • 3D Gaussian splatting offers differentiable rendering and fast training without spatial pruning, which fails to accelerate NeRF generation; DreamGaussian adapts it to generation.NeRF is popular for 2D-supervised reconstruction and generation but can be time-consuming to optimize; prior NeRF acceleration efforts mainly address reconstruction (Mildenhall et al., 2020; M¨uller et al., 2022; Sara Fridovich-Keil and Alex Yu et al., 2022).Source: §2.1 · p. 3

2.2 TEXT-TO-3D GENERATION

  • Text-to-3D methods lift pretrained 2D diffusion models to optimize 3D representations across viewpoints, but usually incur long per-case optimization; DreamGaussian uses 3D Gaussians and reports a simpler optimization landscape.Native 3D diffusion models are described as category-limited and lacking diversity, while 2D lifting aims to support open-vocabulary generation (Jain et al., 2022; Poole et al., 2022).Source: §2.2 · p. 3

2.3 IMAGE-TO-3D GENERATION

  • An efficiency-optimized framework shortens image-to-3D optimization time to 2 minutes with little sacrifice on quality; Zero-1-to-3 (Liu et al., 2023b) has long optimization, while One-2-3-45 (Liu et al., 2023a) accelerates at the cost of generation quality.Source: §2.3 · p. 3

3 OUR APPROACH

DreamGaussian uses Gaussian splatting for efficient initialization, then extracts a textured mesh and refines its texture in UV space. The framework uses SDS for Gaussian optimization and multi-step denoising with MSE for texture refinement.

3.1 GENERATIVE GAUSSIAN SPLATTING

  • Gaussian splatting can support generation, and the method optimizes randomly initialized Gaussians with SDS while densifying fewer Gaussians more frequently than reconstruction.The Gaussians start at random positions inside a sphere with unit scaling and no rotation; camera poses orbit the object during optimization.Source: §3.1 · p. 4

3.2 EFFICIENT MESH EXTRACTION

  • Block-wise local density queries make Gaussian-to-mesh extraction more efficient than brute-force dense-grid querying, followed by Marching Cubes and mesh post-processing.The method queries overlapping blocks and combines them into a 128^3 grid; decimation and remeshing make the extracted mesh smoother and more compact.Source: §3.2 · p. 5
  • The extracted mesh is UV-unwrapped and textured by back-projecting rendered RGB from multiple views, excluding pixels with small camera-space z-direction normals.Views include 8 azimuths, 3 elevations, and top and bottom views; the resulting texture initializes the refinement stage.Source: §3.2 · pp. 5–6

3.3 UV-SPACE TEXTURE REFINEMENT

  • UV texture refinement avoids SDS-induced artifacts by denoising coarse-texture renders and optimizing the texture with pixel-wise MSE.The starting noise timestep limits content changes, and about 50 steps usually produce good detail; more iterations can further enhance texture details.Source: §3.3 · p. 6, Figure 3 · p. 6

4 EXPERIMENTS

Experiments report improved speed with competitive generation quality, describe the implementation and exported meshes, and find that removing any of three stage-1 design choices degrades quality.

4.1 IMPLEMENTATION DETAILS

  • Image-to-3D takes about 1 minute per stage and text-to-3D about 2 minutes per stage; the method uses less than 8 GB of GPU memory.The stages use 500 and 50 training steps, respectively; experiments run on an NVIDIA V100 (16 GB) GPU.Source: §4.1 · pp. 6–7

4.2 QUALITATIVE COMPARISONS

  • Across image- and text-to-3D, DreamGaussian balances quality and speed, outperforming inference-only methods in quality and optimization-based methods in speed; exported meshes have uniform triangulation, smooth normals, and clear textures.The meshes are described as suitable for downstream applications such as rigging and animation.Source: Figure 4 · p. 7, §4.2 · p. 8, Figure 5 · p. 8

4.3 QUANTITATIVE COMPARISONS

  • Over 10 times faster than other optimization-based methods, DreamGaussian's mesh quality falls slightly behind theirs, while its two-stage results improve view consistency and generation quality over inference-only methods.The quantitative comparisons report CLIP similarity and average generation time; the user study assesses reference-view consistency and overall generation quality.Source: §4.3 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 9

4.4 ABLATION STUDY

  • Omitting periodic Gaussian densification, linear timestep annealing, or the reference-view loss degrades model quality, increasing blurriness and inaccuracies that affect fine-tuning.The ablations focus on generative Gaussian-splatting training because mesh fine-tuning has been explored in prior methods.Source: §4.4 · p. 9, Figure 7 · p. 9

5 LIMITATIONS AND CONCLUSION

DreamGaussian combines generative Gaussian splatting, mesh extraction, and texture fine-tuning to produce ready-to-use assets from images or text within minutes, while retaining known generation and ethical limitations.

5 LIMITATIONS AND CONCLUSION

  • DreamGaussian’s Gaussian-splatting pipeline, mesh extraction, and texture fine-tuning produce high-quality polygonal 3D assets from a single image or text description within a few minutes.The paper presents these assets as ready to use.Source: §5 · p. 9
  • DreamGaussian shares the multi-face Janus problem, over-saturated texture, baked lighting, and potentially blurry back-view textures in image-to-3D results.The authors suggest score debiasing (Armandpour et al., 2023; Hong et al., 2023), camera-conditioned 2D diffusion models (Shi et al., 2023; Liu et al., 2023c; Zhao et al., 2023; Li et al., 2023b), and a BRDF auto-encoder (Xu et al., 2023b) as promising avenues; longer stage 2 training may alleviate blurry back-view textures.Source: §5 · p. 9

ETHICS STATEMENT

A APPENDIX · A.1 PRELIMINARY

The preliminary section explains how SDS uses pretrained 2D diffusion priors to optimize differentiable 3D representations, then outlines UV mapping for mesh texture optimization.

A.1 PRELIMINARY

  • SDS, introduced by Dreamfusion (Poole et al., 2022), uses pretrained 2D diffusion models as priors to optimize a parametric image generator.Source: §A.1 · p. 15
  • A representative SDS setup uses a differentiable 3D representation such as NeRF (Mildenhall et al., 2020) as the image generator.Source: §A.1 · p. 15
  • The setup renders an image x from camera pose p using a differentiable rendering function with optimizable NeRF parameters Θ.Source: §A.1 · p. 15
  • SDS samples a timestep and orbiting camera pose, adds Gaussian noise, and uses a pretrained noise predictor conditioned on text embedding e.The weighting function is w(t) = σ²t, as specified for DDPM (Ho et al., 2020).Source: §A.1 · p. 15
  • The denoising guidance gradient is back-propagated through the rendered image and differentiable rendering to optimize NeRF parameters into a shape matching the text description.The rendering function is differentiable (Mildenhall et al., 2020).Source: §A.1 · p. 15
  • UV mapping assigns mesh vertices image-plane coordinates so a 2D texture can be projected onto a polygonal mesh surface.UV unwrapping automatically computes the coordinates, and barycentric interpolation retrieves texture values at surface points.Source: §A.1 · p. 15
  • NVdiffrast (Laine et al., 2020) supports texture mapping and differentiable rendering, enabling optimization of the texture image from rendered images.Source: §A.1 · p. 15

A.2 MORE IMPLEMENTATION DETAILS

This appendix specifies optimization and geometry-processing settings, along with protocols for evaluating generated 3D models.

Learning Rate

  • Gaussian position learning rate decays from 1 × 10−3 to 2 × 10−5 over 500 steps; feature, opacity, scaling/rotation, and texture rates are 0.01, 0.05, 5 × 10−3, and 0.2, respectively.Adam is used for both Gaussian splatting and mesh texture fine-tuning.Source: §A.2 · p. 15

Densification and Pruning

  • Following Kerbl et al. (2023), densification thresholds are 0.5 accumulated gradient for image-to-3D and 0.01 for text-to-3D; pruning uses opacity <0.01 or max scaling >0.05.Image-to-3D densification also requires max scaling <0.05.Source: §A.2 · p. 15

Mesh Extraction

  • Marching Cubes extraction is followed by remeshing to average edge length 0.015 and decimation to 105 faces.Isotropic remeshing and quadric edge collapse decimation control mesh complexity.Source: §A.2 · p. 15

Evaluation Settings

  • Image-to-3D quality is measured by mean CLIP similarity to the reference across 8 zero-elevation azimuth views, using a 30-image dataset and the specified bigG checkpoint.The sampled azimuths are 0, 45, 90, 135, 180, 225, 270, and 315 degrees; the checkpoint is laion/CLIP-ViT-bigG-14-laion2B-39B-b160k1.Source: §A.2 · p. 15
  • The user study collected 900 valid scores from 60 volunteers rating 15 samples each for reference-view consistency and overall model quality.It used 60 rotating videos generated from 15 images across Zero-1-to-3 (Liu et al., 2023b), One-2-3-45 (Liu et al., 2023a), Shap-E (Jun & Nichol, 2023), and DreamGaussian.Source: §A.2 · pp. 15–16

A.3 MORE RESULTS

The additional results show DreamGaussian across image-to-3D and text-conditioned workflows, with exported meshes suited to animation. They also document remaining limitations in back-view detail and text-to-3D geometry and appearance.

Image-to-3D

  • Both DreamGaussian stages run faster than prior two-stage image-to-3D methods while achieving comparable quality, and the pipeline supports non-zero input elevations.Figure 10 compares the training stages with Zero-1-to-3 (Liu et al., 2023b), noted as Zero-1-to-3∗. Elevation sampling covers the input angle and at least [−30, 30] degrees.Source: §A.3 · p. 16, Figure 10 · p. 17, Figure 11 · p. 17

Text-to-image-to-3D

  • Text-to-image-to-3D first synthesizes an image from a prompt, then generates 3D from it, usually outperforming direct text-to-3D while taking less time.The exported meshes are ready for downstream applications such as rigged animation.Source: §A.3 · p. 16, Figure 12 · p. 17, Figure 13 · p. 18

Text-to-3D with MVDream

  • With MVDream guidance, camera information significantly mitigates multi-face Janus artifacts, while timestep annealing yields a more reasonable shape at the same training iterations.The results still show over-saturation and unsmooth geometry.Source: §A.3 · p. 16

Limitations

  • Image-to-3D can produce blurry, low-detail back views that mismatch the front reference, while text-to-3D retains multi-face Janus artifacts and baked lighting.Longer stage-2 training can alleviate blurry back views.Source: §A.3 · p. 16, Figure 14 · p. 18

Related papers

Loading 2309.16653v2…