PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splatting

Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, Ziwei Liu

arXiv:2311.17061v2cs.CV

TL;DR

Text-driven 3D human generation remains challenging because existing approaches struggle to combine fine detail with efficient rendering. HumanGaussian uses structure-aware Gaussian Splatting with RGB and depth guidance, negative-prompt guidance, and size-based pruning; it is reported to render vivid humans with competitive quality and superior efficiency.

  • Problem

    Text-driven 3D human generation must capture complex articulations and fine detail, but mesh and NeRF approaches trade detail against efficient high-resolution rendering.

    Source: §1 · p. 2

  • Method

    HumanGaussian uses SMPL-X-anchored Gaussian Splatting with joint RGB-depth SDS, annealed negative-prompt guidance, and size-conditioned pruning.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    HumanGaussian is reported to render more realistic, coherent, view-consistent, fine-detailed humans and to perform best in user ratings for texture, geometry, and text alignment.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    The framework renders vivid 3D humans across diverse scenarios with reported superior efficiency and competitive quality.

    Source: §1 · p. 3

Abstract

from arXiv · show

Realistic 3D human generation from text prompts is a desirable yet challenging task. Existing methods optimize 3D representations like mesh or neural fields via score distillation sampling (SDS), which suffers from inadequate fine details or excessive training time. In this paper, we propose an efficient yet effective framework, HumanGaussian, that generates high-quality 3D humans with fine-grained geometry and realistic appearance. Our key insight is that 3D Gaussian Splatting is an efficient renderer with periodic Gaussian shrinkage or growing, where such adaptive density control can be naturally guided by intrinsic human structures. Specifically, 1) we first propose a Structure-Aware SDS that simultaneously optimizes human appearance and geometry. The multi-modal score function from both RGB and depth space is leveraged to distill the Gaussian densification and pruning process. 2) Moreover, we devise an Annealed Negative Prompt Guidance by decomposing SDS into a noisier generative score and a cleaner classifier score, which well addresses the over-saturation issue. The floating artifacts are further eliminated based on Gaussian size in a prune-only phase to enhance generation smoothness. Extensive experiments demonstrate the superior efficiency and competitive quality of our framework, rendering vivid 3D humans under diverse scenarios. Project Page: https://alvinliu0.github.io/projects/HumanGaussian

1. Introduction

Text-driven 3D human generation seeks detailed, realistic bodies, but existing approaches face quality or efficiency trade-offs. HumanGaussian uses human-structure guidance with Gaussian Splatting to jointly optimize appearance and geometry, while addressing oversaturation and floating artifacts.

1. Introduction

  • Mesh methods struggle with fine details, while NeRF methods are costly for high-resolution rendering, leaving efficient fine-grained generation unresolved.Earlier SDS approaches also have difficulty modeling detailed bodies with complex articulations.Source: §1 · p. 2
  • 3D Gaussian Splatting’s sparse gradients can hinder geometry and appearance optimization, and naive SDS can cause oversaturation and unstable density control.These challenges motivate structural guidance and a more stable approach to density control.Source: §1 · p. 2
  • Structure-Aware SDS anchors Gaussian positions on the SMPL-X mesh and uses aligned RGB and depth denoising to guide densification and pruning.The approach focuses these processes near the body surface to capture deformations such as accessories and wrinkles.Source: §1 · p. 2
  • Annealed Negative Prompt Guidance combines a noisier generative score with a cleaner classifier score, while size-based pruning removes floating artifacts.The decreasing negative guidance supports realistic generation at nominal classifier-free guidance scales.Source: §1 · p. 3
  • HumanGaussian combines structural guidance with Gaussian Splatting to generate 3D humans with fine-grained geometry and realistic appearance.Its Structure-Aware SDS jointly learns appearance and geometry, while Annealed Negative Prompt Guidance and size-based pruning address oversaturation and floating artifacts.Source: §1 · pp. 2–3

2. Related Work

Related work spans multiple 3D representations and two broad text-to-3D pipelines. HumanGaussian focuses on applying efficient Gaussian Splatting to text-driven 3D humans, where fine detail and realistic texture are both required.

3D Neural Representations

  • Voxel, point-cloud, mesh, and neural-field representations each trade off rendering, detail, topology, resolution, or optimization speed.3D Gaussian Splatting has shown quality and convergence advantages in reconstruction, motivating its use for text-driven human generation.Source: §2 · p. 3

Text-to-3D Generation

  • Text-to-3D methods include 3D-native pipelines and per-prompt optimization that distills 2D generative priors into 3D representations.Gaussian splatting has been adapted to reduce the training burden associated with NeRF, while HumanGaussian targets the 3D human domain.Source: §2 · p. 3

Text-Driven 3D Human Generation

  • Text-driven human methods adapt general techniques with body priors, while differing in representations and guidance strategies.Examples include SMPL- or imGHUM-based methods and approaches using CLIP, SDS, or text-to-depth/normal guidance.Source: §2 · p. 3

3. Our Approach

HumanGaussian adapts 3D Gaussian Splatting to text-driven human generation by using structural guidance for density control. It combines joint RGB/depth optimization with annealed negative-prompt guidance and size-based pruning.

3. Our Approach

  • HumanGaussian uses 3D Gaussian Splatting to generate text-driven 3D humans, with adaptive density control guided by intrinsic human structures.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Preliminaries

  • SMPL-X supplies body, hand, and face topology, while 3D Gaussian Splatting represents scenes with anisotropic Gaussians and supports GPU-friendly rasterization.Source: §3.1 · p. 4

3.2. Structure-Aware SDS

  • Structure-Aware SDS initializes dense Gaussians on SMPL-X, trains a pose-conditioned joint RGB-depth denoiser, and optimizes appearance and geometry with aligned texture and structural guidance.Source: §3.2 · p. 5

3.3. Annealed Negative Prompt Guidance

  • Annealed Negative Prompt Guidance uses a cleaner classifier score and a negative score whose weight decreases at smaller timesteps to regularize stochastic SDS gradients.Source: §3.3 · p. 6
  • A prune-only phase removes oversized Gaussians associated with floating artifacts; dense SMPL-X initialization can compensate for detail loss during pruning.Source: §3.3 · p. 6

4. Experiments

Experiments report that HumanGaussian compares favorably with recent text-to-3D and human-generation methods, and users rate it best across three quality aspects. Ablations associate the framework’s components with improvements to body structure, appearance, geometric errors, and floating artifacts.

4.1. Implementation Details

  • Training runs for 3,600 iterations and takes one hour on a single NVIDIA A100 (40GB) GPU.Densification and pruning run from iterations 300 to 2,100; a prune-only phase follows from 2,400 to 3,300.Source: §4.1 · p. 6

4.2. Text-Driven 3D Human Generation

  • The comparison covers general text-to-3D methods DreamGaussian [77] and GaussianDreamer [89], both concurrent models using 3DGS as neural representation, alongside specialized text-to-3D human models TADA [41] and DreamHuman [36].Source: §4.2 · p. 6
  • HumanGaussian renders more realistic appearance, coherent body structure, better view consistency, and finer detail than recent comparison methods.The comparisons cover five prompts with two camera views each; the 3DGS baselines were strengthened with additional priors.Source: Figure 3 · p. 7, §4.2 · p. 7

4.3. Ablation Study

  • Ablations associate body priors and pose guidance with coherent structure, negative guidance with realistic appearance, dual-branch SDS with fewer limb and hair errors, and size-based pruning with fewer floating artifacts.The full model applies size-conditioned Gaussian removal in a prune-only phase.Source: §4.3 · p. 8, Figure 4 · p. 8

4.2. Text-Driven 3D Human Generation

  • HumanGaussian performs best on texture quality, geometry quality, and text alignment in a study of 30 prompts rated by 17 participants.Ratings use a 1-to-5 Mean Opinion Score scale.Source: Table 1 · p. 8, §4.2 · p. 8

5. Discussion

HumanGaussian combines Structure-Aware SDS with Annealed Negative Prompt Guidance, and experiments show vivid 3D humans with competitive quality and efficiency. A limitation is that existing text-to-image models can make hand and foot generation unfaithful.

Limitations and Future Work

  • The method sometimes fails to render hands and feet faithfully because existing text-to-image models have limited performance on those parts.The authors identify this as future work.Source: §5 · p. 8

Conclusion

  • The authors characterize HumanGaussian as an early attempt to adapt 3D Gaussian Splatting to text-driven 3D human generation.They report realistic results across diverse scenarios.Source: §5 · p. 8

Related papers

Loading 2311.17061v2…