PaperlayerView exact revision on arXiv

Source-linked AI summary

AvatarVerse: High-quality & Stable 3D Avatar Creation from Text and Pose

Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang, Li Chen, Chao Long, Feida Zhu, Kang Du, Min Zheng

arXiv:2308.03610v1cs.CV

TL;DR

Text-guided 3D avatar creation remains challenging when customized descriptions must yield expressive, stable, detailed results. AvatarVerse combines DensePose-conditioned control with progressive high-resolution synthesis, and the authors report superior avatar quality and stability in evaluations and user studies.

  • Problem

    Generating stable, detailed 3D avatars from customized text and pose guidance is challenging, with text-only view conditioning inadequate for arbitrary views and uncontrolled SDS prone to the Janus problem.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    AvatarVerse conditions score distillation on a DensePose signal for pose control and view consistency, then uses progressive high-resolution generation and surface smoothing.

    Source: §1 · p. 1, §3.4 · p. 6

  • Results

    AvatarVerse significantly outperforms previous and contemporary approaches in experiments and user studies, and 81% of volunteers favored it over DreamHuman for realistic humans.

    Source: §4.3 · p. 7, Conclusion · p. 8

  • Takeaways & Limitations

    The authors conclude that AvatarVerse produces high-quality avatars that align with SMPL joints and are easily animatable through skeletal binding.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Creating expressive, diverse and high-quality 3D avatars from highly customized text descriptions and pose guidance is a challenging task, due to the intricacy of modeling and texturing in 3D that ensure details and various styles (realistic, fictional, etc). We present AvatarVerse, a stable pipeline for generating expressive high-quality 3D avatars from nothing but text descriptions and pose guidance. In specific, we introduce a 2D diffusion model conditioned on DensePose signal to establish 3D pose control of avatars through 2D images, which enhances view consistency from partially observed scenarios. It addresses the infamous Janus Problem and significantly stablizes the generation process. Moreover, we propose a progressive high-resolution 3D synthesis strategy, which obtains substantial improvement over the quality of the created 3D avatars. To this end, the proposed AvatarVerse pipeline achieves zero-shot 3D modeling of 3D avatars that are not only more expressive, but also in higher quality and fidelity than previous works. Rigorous qualitative evaluations and user studies showcase AvatarVerse's superiority in synthesizing high-fidelity 3D avatars, leading to a new standard in high-quality and stable 3D avatar creation. Our project page is: https://avatarverse3d.github.io

1. Introduction

Automating text-guided 3D avatar creation could reduce the substantial labor of manual modeling, but stable, detailed avatars remain difficult to synthesize. AvatarVerse uses DensePose-based pose control and progressive high-resolution generation to address these challenges.

1. Introduction

  • Manual construction of intricate 3D avatars can require thousands of hours from skilled artists, motivating automation from natural-language descriptions.The paper notes applications in games, social media, AR/VR, and human-computer interaction.Source: §1 · p. 1
  • Existing text-guided 3D avatar generation remains difficult to stabilize across poses, appearances, and shapes; uncontrolled score distillation can produce Janus artifacts, while outputs may lack fine texture and accessories.Earlier reconstruction methods rely on limited visual priors from videos or reference images, constraining their ability to create avatars from complex text prompts.Source: §1 · p. 1
  • AvatarVerse combines text descriptions and reference poses with DensePose-conditioned score distillation to support pose-aware, stable, view-consistent avatar creation and mitigate the Janus problem.The introduction also links DensePose supervision to alignment with SMPL joints for skeletal binding and control.Source: §1 · p. 1
  • A progressive high-resolution strategy improves local geometry fidelity, while a smoothness loss regularizes density-grid gradients to alleviate coarse avatar surfaces.The introduction describes the smoothness loss as encouraging smoother gradients in the explicit NeRF density voxel grid.Source: §1 · pp. 1, 3
  • The authors report that qualitative evaluations and user studies show AvatarVerse excels in avatar quality and stability, supporting zero-shot creation of high-fidelity avatars.The contribution summary identifies text and reference pose as the inputs.Source: §1 · p. 3

2. Related work

Prior work applies text-image models to 3D content and avatar generation, while separate efforts pursue higher-fidelity avatars. The paper positions AvatarVerse as addressing avatar-specific quality, control, and detail challenges across these lines of work.

2.1. Text-guided 3D content generation

  • Text-guided 3D content methods use CLIP or diffusion-based supervision to optimize meshes and NeRFs, but can yield low-quality avatars with Janus artifacts or incorrect body parts.The cited approaches include CLIP-forge, Dream-Fields, CLIP-Mesh, DreamFusion, Latent-NeRF, TEXTure, and ProlificDreamer.Source: §2.1 · p. 3

2.2. Text-guided 3D Avatar generation

  • Avatar-generation approaches use shape priors, pretrained diffusion, or landmark control, while DreamWaltz's skeleton-conditioned diffusion may struggle with view consistency, partial poses, and avatar contours.The authors characterize DensePose-conditioned ControlNet as supporting view-consistent generation across viewpoints and body parts.Source: §2.2 · p. 3

2.3. High-quality 3D Avatar Generation

  • High-fidelity avatar methods include multi-view reconstruction and coarse-to-fine refinement; the paper contrasts DreamHuman's 64 × 64 body-region renders and skintight outputs with AvatarVerse's progressive zooming and body-part focus.The authors say their strategy supports a wider range of accessories and clothing, and progressive grid supports fine-grained generation.Source: §2.3 · p. 3

3. Methodology

AvatarVerse is a fully automatic pipeline for creating realistic 3D avatars from text and body pose. It combines DensePose-conditioned score distillation with progressive high-resolution generation and surface smoothing.

3. Methodology

  • AvatarVerse's method includes a DensePose-conditioned SDS loss for pose-aware synthesis and Janus-problem mitigation, alongside progressive high-resolution generation and avatar surface smoothing.The section overview describes the pipeline as using text and body pose to create a realistic 3D avatar.Source: §3 · p. 3
  • The pipeline takes a text prompt and DensePose signal to optimize an explicit NeRF through a DensePose-COCO pretrained ControlNet.The overview also identifies progressive grid, progressive radius, and focus mode as strategies for high-resolution avatar generation.Source: Figure 2 · p. 4

3.1. Preliminaries

The preliminaries describe score distillation sampling as diffusion-based supervision for a differentiable 3D representation, alongside SMPL and DensePose concepts used to represent and condition human bodies.

3.1. Preliminaries

  • Score distillation sampling transfers a pretrained diffusion model's prior to a differentiable 3D representation by comparing predicted noise with added noise for a rendered image.The weighting function depends on noise level and text prompt.Source: §3.1 · p. 4
  • SMPL represents a parametric human body using pose and shape parameters, with articulated deformation mapping the template through joint positions, pose, and blend weights.The paper specifies 6,890 body vertices and 24 keypoints; each observed-pose vertex is formed using weighted joint transforms.Source: §3.1 · p. 4
  • DensePose establishes dense correspondences between a 2D image and a 3D surface-based human model, assigning SMPL mesh faces to 24 predefined body parts for viewpoint-specific part-labeled images.Source: §3.1 · p. 4

3.2. DensePose SDS Loss

Text prompts alone inadequately condition diffusion models on arbitrary views, contributing to instability and the Janus problem. AvatarVerse uses DensePose-conditioned SDS to align pose and avatar views and support detailed pose control.

3.2. DensePose SDS Loss

  • OpenPose-controlled images often miss the avatar’s backside and struggle with part generation, while DensePose results include non-skin-tight realistic and fictional avatars.Figure 3 compares 10 images per signal at matching viewpoints and body parts.Source: Figure 3 · p. 5, §3.2 · p. 5
  • DensePose provides precise 2D localization of 3D body parts, including when parts are partially concealed, unlike skeletal conditions that may miss details and boundaries.The authors use DensePose as a more robust control signal for arbitrary views.Source: §3.2 · pp. 4–5
  • Rendering SMPL DensePose conditions and NeRF views from the same camera viewpoint supports coherent representations between pose and scene.The method trains a DensePose-conditioned ControlNet on DeepFashion part-labeled annotations and uses the rendered condition in SDS.Source: §3.2 · p. 5

3.3. Progressive High-Resolution Generation

Global SDS guidance can leave hands and faces without high-quality detail. AvatarVerse addresses this with progressive grid refinement, changing camera radius, focus mode, and mesh refinement.

3.3. Progressive High-Resolution Generation

  • Global SDS guidance can fail to produce high-quality details, especially for hands and faces, motivating progressive grid, focus mode, and progressive radius strategies.These strategies target accurate, detailed geometry and textures.Source: §3.3 · p. 5

Progressive grid

  • The progressive grid starts with fewer voxels and doubles their number during optimization, reducing the early learning space and floating artifacts while gradually refining the avatar.The voxel count doubles after predetermined optimization steps, and voxel size updates accordingly.Source: Progressive grid · p. 5
  • Shrinking the bounding box around density-filtered avatar regions decreases voxel size, allowing more voxels around the avatar to capture contours, facial features, and clothing folds.The method uses a density threshold to identify the region to enclose.Source: Progressive grid · p. 5

Progressive Radius

  • At selected checkpoints, the camera radius decreases by 20%, applying conditioned SDS to smaller avatar regions to refine details stage by stage.The authors report qualitative comparisons against four state-of-the-art methods, with higher-resolution details and fine-grained geometry in AvatarVerse outputs.Source: Progressive Radius · pp. 5–6, Figure 4 · p. 6

Focus Mode

  • Focus mode places the camera near SMPL-derived body-part positions and computes loss over a small region at 512 × 512 resolution, enabling detailed partial-body generation without extra computational resources.It is used in both coarse and fine stages.Source: Focus Mode · p. 6

Mesh Refinement

  • Mesh refinement initializes geometry from the trained explicit NeRF and optimizes deformable tetrahedral grids using DensePose-conditioned SDS gradients.The mesh strategy targets textured avatars within memory and computation budgets.Source: Mesh Refinement · p. 6

3.4. Avatar Surface Smoothing

Explicit voxel grids can produce less smooth surfaces because individual voxel optimization limits information sharing and may reach local minima. AvatarVerse adds a smoothness regularizer to the density-grid gradient.

3.4. Avatar Surface Smoothing

  • Independent voxel optimization limits information sharing across the grid, making globally coherent avatar shapes challenging and producing less smooth surfaces and local minima.The challenge arises from the explicit grid’s high degrees of freedom and lack of spatial coherence.Source: §3.4 · p. 6
  • A Gaussian-convolution-based smoothness loss regularizes the density-voxel gradient, encouraging smoother surfaces and mitigating noisy points in free space.The overall loss combines the P-SDS loss with the smoothness term weighted by λ.Source: §3.4 · p. 6

4. Experiments

The experiments compare AvatarVerse with recent methods and demonstrate partial-avatar and pose-controlled generation. The implementation combines DensePose training with progressive high-resolution generation and mesh refinement.

4.1. Implementation Details

  • AvatarVerse uses 5,000 coarse-stage and 4,000 mesh-refinement iterations; the whole generation process takes around 2 hours on one single NVIDIA A100 GPU.Source: §4.1 · p. 7

Initialization

  • Initialization uses a small ellipsoidal density blob around the origin, with dimensions based on SMPL pose coordinates, plus SMPL-derived density bias.Source: Initialization · p. 7

DensePose Training

  • DensePose ControlNet training uses more than 800K DeepFashion image pairs annotated with pretrained DensePose and paired with BLIP2-generated text prompts.The diffusion model used is SD1.5.Source: DensePose Training · p. 7

Progressive High-Resolution Generation

  • The progressive grid doubles voxel count at 500, 1,500, and 2,000 coarse-stage iterations, then shrinks the bounding box after 3,000 steps to regions with density above 0.1.The camera radius is reduced at 1,000 and 2,000 iterations, and focus mode starts at coarse-stage step 1,000 and continues through mesh refinement.Source: Progressive High-Resolution Generation · p. 7

Comparison with SOTA methods

  • AvatarVerse qualitatively outperforms DreamFusion, DreamAvatar, DreamWaltz, and DreamHuman in geometry and texture quality, with richer details than DreamHuman across skin, faces, and clothing.The authors attribute clear surfaces to progressive high-resolution generation.Source: Comparison with SOTA methods · p. 7

Flexible Avatar Generation

  • AvatarVerse generates partial avatars by modifying the input DensePose signal, without extra descriptive prompts, and supports varied poses with stable view consistency.Shown partial forms include full-body, half-body, head-only, and hand-only avatars.Source: Flexible Avatar Generation · p. 7
  • Figure 5 presents partial generation of Stormtrooper and Batman prompts and arbitrary-pose generation.Source: Figure 5 · p. 7

4.3. User Study

The user study reports a strong preference for AvatarVerse over DreamHuman in realistic-human comparisons and significant preference over three other methods. Volunteers judged rendered rotating videos for geometry and texture quality.

4.3. User Study

  • Volunteers significantly preferred AvatarVerse to DreamFusion, DreamAvatar, and DreamWaltz in comparisons using the same text prompts.The study used 30 generated outcomes shown as rendered rotating videos, with 16 volunteers voting based on geometry and texture quality.Source: §4.3 · p. 7
  • The user study compared generated avatar results based on geometry and texture quality.Sixteen volunteers assessed 30 outcomes presented as rendered rotating videos.Source: §4.3 · p. 7
  • 81% of volunteers voted in favor of AvatarVerse over DreamHuman for realistic humans.The comparison was based on realistic-human results.Source: §4.3 · p. 7

4.4. Ablation Study

The ablations examine progressive high-resolution strategies, DensePose control, and surface smoothing. The reported comparisons describe added detail and reduced artifacts with progressive stages, more precise and stable control with DensePose, and compact, smooth surfaces with smoothing.

Effectiveness of Progressive Strategies

  • Progressive grid, radius narrowing, focus mode, and mesh refinement successively add avatar-region detail, reduce floating artifacts, sharpen local detail, and refine texture.Focus mode optimizes selected body parts at 512 × 512 resolution; mesh refinement further optimizes the coarse avatar mesh.Source: Effectiveness of Progressive Strategies · p. 8, Figure 7 · p. 8

Effectiveness of DensePose Control

  • DensePose control produces more precise boundaries, intricate details, and stable avatar control than skeleton control, which leaves blurry edges and a severe Janus problem.Skeleton control makes avatars more human-like than using no additional control.Source: Effectiveness of DensePose Control · p. 8, Figure 8 · p. 8

Effectiveness of Surface Smoothing

  • Surface smoothing is reported to yield compact geometry and smooth surfaces, with visual quality and realism depending on balancing smoothness and conditioned SDS losses.Figure 9 compares results with and without surface smoothing under the same text prompt.Source: Effectiveness of Surface Smoothing · p. 8, Figure 9 · p. 8

Conclusion

AvatarVerse generates high-quality, stable 3D avatars from text prompts and poses, with full- or partial-body control during explicit NeRF optimization. The conclusion reports superior texture and geometry quality, easy skeletal animation, and better performance than previous and contemporary approaches.

Conclusion

  • AvatarVerse generates high-quality, stable 3D avatars from textual prompts and poses, with partial- or full-body control during explicit NeRF optimization.Source: Conclusion · p. 8
  • The generated avatars exhibit superior texture and geometry quality, attributed to progressive high-resolution generation.Source: Conclusion · p. 8
  • The avatars are easily animatable through skeletal binding and show high alignment with SMPL joints.Source: Conclusion · p. 8
  • Experiments and user studies show AvatarVerse significantly outperforms previous and contemporary approaches.Source: Conclusion · p. 8

Related papers

Loading 2308.03610v1…