Source-linked AI summary
DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models
Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, Kwan-Yee K. Wong
TL;DR
DreamAvatar addresses the challenge of generating high-quality 3D human avatars by combining text and shape guidance with controllable poses. It uses a dual-observation-space NeRF framework with diffusion-based self-supervision and achieves state-of-the-art generation quality.
Problem
Generating high-quality 3D human avatars remains challenging because human shape, pose, and appearance are complex.
Method
DreamAvatar combines a trainable NeRF, pretrained text-to-image diffusion models, SMPL shape and pose guidance, jointly optimized canonical and posed spaces, and full-body and zoomed-in head losses.
Results
DreamAvatar significantly outperforms existing methods and achieves state-of-the-art text-and-shape guided 3D human avatar generation.
Takeaways & Limitations
The framework generates high-quality avatars with text-consistent geometry, geometry-consistent texture, controllable poses, and improved facial details.
Takeaways & Limitations
The method adopts A-pose rather than T-pose as its canonical model, based on its suitability for diffusion-model interpretation.
Abstract
from arXiv · showhide
We present DreamAvatar, a text-and-shape guided framework for generating high-quality 3D human avatars with controllable poses. While encouraging results have been reported by recent methods on text-guided 3D common object generation, generating high-quality human avatars remains an open challenge due to the complexity of the human body's shape, pose, and appearance. We propose DreamAvatar to tackle this challenge, which utilizes a trainable NeRF for predicting density and color for 3D points and pretrained text-to-image diffusion models for providing 2D self-supervision. Specifically, we leverage the SMPL model to provide shape and pose guidance for the generation. We introduce a dual-observation-space design that involves the joint optimization of a canonical space and a posed space that are related by a learnable deformation field. This facilitates the generation of more complete textures and geometry faithful to the target pose. We also jointly optimize the losses computed from the full body and from the zoomed-in 3D head to alleviate the common multi-face ''Janus'' problem and improve facial details in the generated avatars. Extensive evaluations demonstrate that DreamAvatar significantly outperforms existing methods, establishing a new state-of-the-art for text-and-shape guided 3D human avatar generation.
1. Introduction
DreamAvatar addresses the difficulty of generating detailed 3D human avatars by combining text guidance with SMPL shape and pose priors. Its dual-observation design and head-focused optimization improve pose-faithful geometry, complete textures, facial details, and overall generation quality.
- Motivation: Traditional 3D human modeling is time-consuming and highly dependent on trained artists and engineers.Deep learning reduces manual effort but still struggles to recover complete image details and depends heavily on training data.
- Motivation: Human avatar generation remains difficult because bodies have complex articulated structures and varied poses across body parts.
- DreamAvatar: DreamAvatar combines a trainable NeRF, pretrained text-to-image diffusion models, and SMPL shape priors for 2D-supervised 3D avatar generation.SMPL provides shape and pose guidance during generation.
- DreamAvatar: Its dual-observation-space design jointly optimizes canonical and posed spaces linked by a learnable deformation field.The design supports more complete textures and geometry faithful to the target pose.
- DreamAvatar: Full-body and zoomed-in head losses jointly reduce the multi-face “Janus” problem and improve facial details.
- Results: DreamAvatar significantly outperforms existing methods in evaluations spanning movie, anime, video-game characters, and general people.The generated avatars exhibit text-consistent geometry and geometry-consistent texture.
2. Related Work
Related work progresses from text-guided 2D generation and image-text-guided 3D optimization toward diffusion-based 3D content generation. Human-specific generative models use multiple 3D representations and 2D datasets, but generally cannot generate unseen subjects or support the broader text-and-shape setting addressed by DreamAvatar.
- Text-guided generation: Diffusion models extend text-guided image generation toward more complex semantic concepts, while text-to-3D remains limited by scarce paired data and high training cost.
- Text-guided 3D content generation: Earlier text-guided 3D methods optimize NeRFs or meshes using pretrained image-text models, with CLIP-forge relying on a computationally expensive normalizing flow model.
- Text-guided 3D content generation: DreamFusion introduced diffusion-based Score Distillation Sampling, followed by methods targeting finer meshes, efficiency, diversity, texture, or disentangled geometry and texture.
- 3D human generative models: Many 3D human generative models use voxel grids, point clouds, meshes, NeRFs, tri-planes, or GANs and often require expensive, limited 3D datasets.
- 3D human generative models: Human radiance-field methods trained on 2D human datasets produce convincing results but cannot generate new subjects absent from their training data.
3. Methodology
DreamAvatar generates text-and-shape guided 3D human avatars by optimizing a trainable NeRF with diffusion-based 2D self-supervision, SMPL guidance, dual observation spaces, and head-focused losses. These components target controllable poses, complete textures, pose-faithful geometry, and improved facial details.
- Shape and pose guidance: SMPL supplies shape and pose guidance, with A-pose used as the canonical model because it is more natural for diffusion models to understand.The framework derives density fields from signed distances to SMPL surfaces in canonical and target poses.
- Dual observation spaces: DreamAvatar jointly optimizes canonical and posed observation spaces connected by an SMPL-based learnable deformation field.Canonical observations reduce self-occlusion for more complete textures, while posed observations support target-pose-faithful geometry and provide additional supervision.
- Neural representation and deformation: The extended NeRF queries density, color, and normal features while using SMPL parameters to handle different body parts and initialize density values.The posed-to-canonical deformation combines inverse articulated SMPL skinning with an MLP-predicted corrective offset.
- Head refinement: Full-body and zoomed-in head losses are jointly optimized with landmark-based ControlNet and a special <back-view> token to alleviate Janus faces and improve facial details.The head loss is computed from a zoomed-in head render at each iteration.
- Comparisons: Compared with text-to-3D and avatar-specified baselines, DreamAvatar produces higher-quality geometry and texture, including higher-resolution avatar outputs.The comparisons specifically emphasize structural and topological correctness and improved geometry and texture resolution.
4. Experiments
DreamAvatar is evaluated across text, pose, shape, style, and head–body integration, with comparisons, user studies, and ablations examining its generation quality and design choices.
- 4.1. Qualitative Evaluations: DreamAvatar generates diverse human avatars across real-world, movie, anime, and video game styles.
- 4.1. Qualitative Evaluations: The method maintains high-quality texture and geometry for various poses, including extreme poses with severe self-occlusion.
- 4.1. Qualitative Evaluations: Text manipulation produces avatars that embody added descriptions and capture the unique characteristics of the main subject.
- 4.1. Qualitative Evaluations: Editing SMPL shape parameters generates avatars with different sizes, including thin, short, tall, and fat shapes.
- 4.2. User Studies: In a study of 25 volunteers rating 20 examples, DreamAvatar received the best rankings for geometry quality, texture quality, and text consistency.
- 4.3. Further Analysis: Removing SMPL-derived densities causes low-quality, outlier-prone, and view-inconsistent avatars with unreasonable human shapes.
- 4.3. Further Analysis: Removing the head VSD loss lowers head quality and makes the multi-face Janus problem more pronounced.
- 4.3. Further Analysis: Neither canonical-only nor posed-only modeling without deformation matches the performance of the dual-observation-space design.
5. Conclusions
DreamAvatar combines SMPL guidance, dual observation spaces, and joint full-body and head losses for text-and-shape guided 3D human avatar generation. The paper reports more complete textures, pose-faithful geometry, improved facial details, alleviated Janus artifacts, and state-of-the-art results, while noting limitations in animation and inherited diffusion-model biases.
- SMPL provides shape and pose guidance through a rough human-body prior.
- The dual-observation-space design facilitates more complete textures and geometry faithful to the target pose.
- Joint full-body and zoomed-in head losses alleviate the multi-face Janus problem and improve facial details.
- Extensive experiments report state-of-the-art 3D human avatar generation.
- The current implementation does not consider animation and inherits biases from the pretrained diffusion model's text-image data distribution.