PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-3D using Gaussian Splatting

Zilong Chen, Feng Wang, Yikai Wang, Huaping Liu

arXiv:2309.16585v4cs.CV

TL;DR

Existing SDS-based text-to-3D methods often produce collapsed or Janus geometry and have difficulty incorporating explicit 3D priors. GSGEN uses Gaussian Splatting with joint 2D and 3D guidance plus progressive geometry and appearance optimization, producing detailed, geometrically accurate assets with strong high-frequency detail capture.

  • Problem

    Existing text-to-3D methods often suffer from collapsed geometry and limited fidelity because their implicit representations make explicit 3D priors difficult to incorporate.

  • Method

    GSGEN represents scenes with 3D Gaussians and combines 2D score distillation sampling with a 3D point cloud diffusion prior in a two-stage geometry and appearance optimization process.

  • Results

    GSGEN generates 3D assets with accurate geometry, intricate details, and enhanced high-frequency components, while producing assets in about 40 minutes.

  • Takeaways & Limitations

    Gaussian Splatting enables GSGEN to incorporate direct geometric priors while refining detailed appearance through compactness-based densification.

  • Takeaways & Limitations

    GSGEN can produce unsatisfying results for complex prompts and may still experience degeneration when guidance diffusion models are strongly biased.

Abstract

from arXiv · show

Automatic text-to-3D generation that combines Score Distillation Sampling (SDS) with the optimization of volume rendering has achieved remarkable progress in synthesizing realistic 3D objects. Yet most existing text-to-3D methods by SDS and volume rendering suffer from inaccurate geometry, e.g., the Janus issue, since it is hard to explicitly integrate 3D priors into implicit 3D representations. Besides, it is usually time-consuming for them to generate elaborate 3D models with rich colors. In response, this paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting that enables the incorporation of 3D prior. Specifically, our method adopts a progressive optimization strategy, which includes a geometry optimization stage and an appearance refinement stage. In geometry optimization, a coarse representation is established under 3D point cloud diffusion prior along with the ordinary 2D SDS optimization, ensuring a sensible and 3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an iterative appearance refinement to enrich texture details. In this stage, we increase the number of Gaussians by compactness-based densification to enhance continuity and improve fidelity. With these designs, our approach can generate 3D assets with delicate details and accurate geometry. Extensive evaluations demonstrate the effectiveness of our method, especially for capturing high-frequency components. Our code is available at https://github.com/gsgen3d/gsgen

1. Introduction

GSGEN addresses inaccurate geometry and limited fidelity in text-to-3D generation by combining explicit Gaussian Splatting with direct 3D priors. Its progressive optimization produces geometry-consistent assets with detailed appearances, especially high-frequency components.

  • Existing SDS-based text-to-3D methods often suffer collapsed geometry, limited fidelity, and difficulty incorporating 3D priors into implicit representations.
  • GSGEN uses explicit 3D Gaussians and an additional 3D point cloud diffusion prior to enhance geometrical coherence.
  • The method progressively optimizes Gaussian representations through geometry optimization followed by appearance refinement.Geometry optimization uses joint 2D and 3D guidance, while appearance refinement enriches details through compactness-based densification.
  • GSGEN combines direct geometric priors with Gaussian Splatting to generate assets with accurate geometry and enhanced fidelity.
  • GSGEN demonstrates superior performance in capturing high-frequency components such as feathers, intricate textures, and animal fur.

2. Related Work

Related work spans differentiable 3D scene representations, diffusion models, and text-to-3D generation. GSGEN builds on these directions by using Gaussian Splatting with point-cloud diffusion guidance.

  • NeRF-based methods represent scenes with coordinate-based neural networks and have advanced reconstruction, scalability, dynamics, training, rendering, and downstream tasks.
  • Diffusion models learn and sample complex data distributions through forward noising and learned inverse denoising processes.
  • Early text-to-3D methods used CLIP guidance, while DreamFusion introduced SDS optimization with a pretrained image diffusion prior.
  • Text-to-3D Generation: GSGEN organizes optimization into geometry and appearance stages to balance coherent geometry with detailed texture.
  • Text-to-3D Generation: A 3D diffusion model trained on millions of 3D models provides valuable 3D guidance and coarse initialization.

3. Preliminary

SDS optimizes differentiable 3D representations using image diffusion guidance, while Gaussian Splatting provides an explicit, renderable scene representation. GSGEN combines SDS with 3D Gaussian Splatting at both 2D and 3D levels.

  • Score Distillation Sampling: SDS optimizes a differentiable image parameterization by rendering it under camera parameters and iteratively refining its parameters against a diffusion prior.
  • Score Distillation Sampling: GSGEN combines SDS with 3D Gaussian Splatting at 2D and 3D levels using different diffusion models.
  • 3D Gaussian Splatting: Gaussian Splatting represents scenes with anisotropic 3D Gaussians parameterized by positions, covariances, colors, and opacities, which are projected onto the camera plane.
  • 3D Gaussian Splatting: The explicit Gaussian representation enables direct integration of 3D diffusion priors into text-to-3D generation.

4. Approach

GSGEN uses explicit 3D Gaussians and a point-cloud diffusion prior to shape coherent geometry, then refines appearance with compactness-based densification for greater detail and fidelity.

  • Geometry Optimization: GSGEN represents 3D content with Gaussians and combines 3D SDS with a point-cloud diffusion model to promote 3D-consistent geometry.The explicit representation supports direct geometric-prior integration and aims to mitigate the Janus problem.
  • Geometry Optimization: The geometry stage optimizes Gaussian positions using Point-E guidance alongside the ordinary 2D image diffusion prior.Applying 3D SDS avoids direct point-cloud alignment issues such as registration, scaling, and potential degeneration.
  • Appearance Refinement: Appearance refinement uses only the 2D image prior to iteratively enrich details after the 3D prior has shaped the rough geometry.The separate stage addresses the observed disturbance of appearance learning caused by the 3D prior.
  • Appearance Refinement: Compactness-based densification adds Gaussians between nearby neighbors whose distance is smaller than the sum of their radii, filling geometric holes.The method supplements positional-gradient splitting because SDS makes its threshold difficult to choose reliably.
  • Initialization with Geometry Prior: Gaussian positions can be initialized from Point-E-generated point clouds or user-provided meshes and point clouds instead of a simple pattern.For general text-to-3D generation, the method uses random colors rather than Point-E's generated colors based on empirical observations.

5. Experiments

GSGEN is evaluated against prior text-to-3D methods through qualitative comparisons and ablations of initialization, 3D guidance, and densification. It produces more accurate geometry, richer high-frequency details, and comparable generation time, while compactness-based densification supplements standard SDS densification.

  • 5.2. Text-to-3D Generation: GSGEN produces delicate assets with more accurate geometry and intricate details than prior methods under comparable guidance.Previous SDS-based methods often generate collapsed geometry, while ProlificDreamer remains susceptible to the Janus problem.
  • 5.2. Text-to-3D Generation: GSGEN preserves high-frequency details such as sushi patterns, peacock feathers, and thatched roofs better than several competing methods.Magic3D and Fantasia3D produce over-smoothed geometry, whereas ProlificDreamer is prone to the multi-face problem.
  • 5.2. Text-to-3D Generation: About 40 minutes are required to generate each 3D asset, on par with Magic3D and Fantasia3D but with improved fidelity and richer details.
  • 5.3. Ablation Study: Point-E initialization breaks symmetry through an anisotropic geometry prior, producing more 3D-consistent objects than DreamFusion-like initialization.Origin-centered Gaussian initialization causes severe degeneration, especially for asymmetric prompts.
  • 5.3. Ablation Study: Adding 3D point-cloud guidance mitigates the Janus problem beyond image diffusion guidance alone, particularly for asymmetric geometries.Image-only guidance improves consistency over random initialization but still exhibits Janus artifacts in examples such as the dog and panda.
  • 5.3. Ablation Study: Compactness-based densification supplements standard view-space-gradient densification under SDS guidance.Large thresholds yield over-smoothed appearance, whereas small thresholds are vulnerable to unstable stochastic gradients.

6. Limitations and Conclusion

GSGEN generates detailed, 3D-consistent assets through two-stage Gaussian optimization, but its results remain limited by language understanding and biased diffusion guidance. The method mitigates rather than eliminates geometric degeneration.

  • Limitations: Complex scene descriptions or complicated logic can produce unsatisfying results because Point-E and StableDiffusion’s CLIP text encoder have limited language understanding.
  • Limitations: 3D priors mitigate the Janus problem but do not eliminate potential degenerations, especially when guidance diffusion models are extremely biased.
  • Conclusion: GSGEN uses geometry optimization with joint point-cloud and image SDS guidance, followed by appearance refinement and compactness-based densification.
  • Conclusion: The method is designed to generate highly detailed and 3D-consistent assets with improved continuity and fidelity.

A. Implementation Details

The implementation reimplements Gaussian Splatting with a learnable background, specifies rendering and guidance components, and uses fixed training settings for reproducible experiments. Resources and comparison implementations are documented separately.

  • 3D Gaussian Splatting Details: The authors reimplement 3D Gaussian Splatting to support a learnable MLP background and use plenoxel-style gradient backpropagation.
  • 3D Gaussian Splatting Details: Depth maps use Gaussian-center view-space depth, while a z-variance renderer supports z-var loss, which has limited observed impact on generated assets.
  • Guidance Details: StableDiffusion v1-5 provides the main 2D guidance, while tested StableDiffusion 2 checkpoints produce no observed improvements.
  • Guidance Details: The official Point-E implementation and checkpoints are used for the point-cloud diffusion model.
  • Training Details: Training uses a single NVIDIA 3090 GPU, batch size 4, and about 40 minutes per prompt, producing roughly 1e5 to 1e6 Gaussians after densification.
  • Open-sourced Resources and Corresponding Licenses: The paper lists open-source code and resources together with their corresponding licenses.
  • Baseline Implementations: Baseline results are obtained using Stable DreamFusion, threestudio, or official Fantasia3D implementations depending on the method.

B. Additional Results

Additional experiments examine user-guided generation and the effects of color and point-cloud representations. These results document the evaluated prompts and comparison settings without reporting a stated outcome for every figure.

  • User-guided generation: User-guided generation initializes points by uniformly sampling the user-provided mesh surface and uses a small position learning rate to preserve the desired shape.
  • User-guided generation: The user-guided prompts include a German Shepherd, robot hand, teddy bear in a tuxedo, Lego man, and Lego house.
  • Color: Figure 9 evaluates the impact of adopting Point-E-generated color.
  • Point-cloud comparison: Figure 10 compares Point-E-generated point clouds with GSGEN-generated 3D assets.

B.2. More Text-To-3D Results

GSGEN produces additional 3D assets with accurate geometry and improved fidelity, alongside qualitative comparisons with prior text-to-3D methods.

  • GSGEN generates additional 3D assets with accurate geometry and improved fidelity.These results are presented in Figures 15 and 16.
  • The supplementary evaluation includes qualitative comparisons with DreamFusion, Magic3D, Fantasia3D, and LatentNeRF.The compared images are copied from the respective papers or project pages for fairness.
  • Point clouds optimized under Point-E and ULIP are shown for the prompt “A corgi.”

B.4.1 Color Initialization

The supplementary studies examine 3D-prior guidance and image-guidance choices, showing benefits from Point-E and MVDream while exposing trade-offs across guidance models.

  • 3D Point Cloud Guidance: Point-E provides rough 3D guidance that helps GSGEN maintain geometrical consistency despite sub-optimal point-cloud generation.It corrects major shape deviations early without needing to guide fine-grained geometric details.
  • 3D Point Cloud Guidance: ULIP performs poorly for generation because its inner-product similarity produces messy point clouds rather than a consistent rough shape.Point-E, in contrast, guides the point cloud to a consistent rough shape with SDS loss.
  • Diffusion Guidance: DeepFloyd IF improves text-to-3D alignment, whereas StableDiffusion produces sharper appearance because DeepFloyd IF requires 64×64 inputs.The stronger T-5 text encoder supports alignment, while downsampling may cause blur.
  • MVDream Guidance: MVDream contributes to multi-view consistency, yielding more accurate geometry and more complete 3D assets.Examples include a more complete panda and a Janus-free ostrich.

C. Failure Cases

GSGEN still exhibits Janus artifacts and failures tied to limited language understanding, although its figures document a broad range of generated assets and comparisons.

  • GSGEN does not completely eliminate the Janus problem despite using a direct 3D prior.The authors attribute this to the ill-posed nature of 2D-prior text-to-3D generation and limited 3D-prior capability.
  • Some failures preserve correct geometry but introduce appearance-level Janus artifacts, such as an extra ostrich head.
  • Other failures arise from limited language understanding in the guidance model, which also prevents StableDiffusion from producing reasonable images for those prompts.
  • The section includes additional generated assets, DreamFusion comparisons, and StableDiffusion-guided qualitative results.
Loading 2309.16585v4…