PaperlayerView exact revision on arXiv

Source-linked AI summary

TADA! Text to Animatable Digital Avatars

Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, Michael J. Black

arXiv:2308.10899v1cs.AI

TL;DR

Text-based character-generation methods have limitations in geometry and texture quality and do not reliably yield realistically animatable avatars. TADA optimizes an SMPL-X-based avatar using displacement, texture, and geometry-texture learning with animation training. The paper reports diverse avatar generation, graphics-pipeline compatibility, and component ablations supporting its approach.

  • Problem

    Existing text-based character methods are limited in geometry and texture quality, and geometry-texture misalignment can hinder realistic animation.

    Source: §1 · p. 2

  • Method

    TADA extends subdivided SMPL-X with learnable displacements and texture, optimizes geometry and texture using RGB and normal-image SDS losses, and trains with animations to preserve SMPL-X correspondence.

    Source: §4.1 · p. 4, §4.2 · p. 5, §4.3 · p. 5

  • Results

    TADA generates avatars across celebrities, customized characters, and cartoon characters, and its ablations report improved geometry-texture alignment and face geometry from consistency loss and animation training.

    Source: §5.2 · p. 6, §5.5 · p. 7

  • Takeaways & Limitations

    The SMPL-X-based output is described as graphics-ready and compatible with traditional pipelines, while avatar creation and editing are controlled through text.

    Source: §1 · p. 2, §8 · p. 9

    Relighting across environments needs improvement, and original skinning weights may yield unrealistic animation for avatars that deviate substantially from SMPL-X.

    Source: §7 · p. 8

Abstract

from arXiv · show

We introduce TADA, a simple-yet-effective approach that takes textual descriptions and produces expressive 3D avatars with high-quality geometry and lifelike textures, that can be animated and rendered with traditional graphics pipelines. Existing text-based character generation methods are limited in terms of geometry and texture quality, and cannot be realistically animated due to inconsistent alignment between the geometry and the texture, particularly in the face region. To overcome these limitations, TADA leverages the synergy of a 2D diffusion model and an animatable parametric body model. Specifically, we derive an optimizable high-resolution body model from SMPL-X with 3D displacements and a texture map, and use hierarchical rendering with score distillation sampling (SDS) to create high-quality, detailed, holistic 3D avatars from text. To ensure alignment between the geometry and texture, we render normals and RGB images of the generated character and exploit their latent embeddings in the SDS training process. We further introduce various expression parameters to deform the generated character during training, ensuring that the semantics of our generated character remain consistent with the original SMPL-X model, resulting in an animatable character. Comprehensive evaluations demonstrate that TADA significantly surpasses existing approaches on both qualitative and quantitative measures. TADA enables creation of large-scale digital character assets that are ready for animation and rendering, while also being easily editable through natural language. The code will be public for research purposes.

1. Introduction

Text-based 3D generation has yet to reliably produce high-quality, animatable avatars compatible with traditional graphics workflows. TADA combines an optimizable SMPL-X-based representation with geometry-texture alignment and animation training to address these gaps.

1. Introduction

  • Existing text-based character methods are limited in geometry and texture quality, often lack animation, or are incompatible with traditional CG workflows.Single- or sparse-image reconstruction methods described here are also limited to real humans and are hard to edit and control.Source: §1 · p. 2
  • TADA optimizes a subdivided SMPL-X mesh with learnable displacements and a texture map, using hierarchical rendering at different focal lengths for detailed body and face views.Its geometry-texture alignment addresses a mismatch that the paper says can make animation infeasible.Source: §1 · p. 2
  • Training with sampled SMPL-X poses and facial expressions preserves semantic correspondence, allowing finished avatars to use novel SMPL-X parameters for animation.The paper notes that this can be combined with text-to-motion or text-to-audio-to-motion methods.Source: §1 · p. 2
  • TADA is presented as a text-controlled avatar creation and editing tool whose SMPL-X, displacement, and texture representation fits traditional graphics pipelines.The paper reports ablations, qualitative comparisons, and a user study of high-quality 3D avatar generation.Source: §1 · p. 2

2. Related Work

Related work extends text-to-image generation to 3D, including via CLIP guidance and diffusion-based score distillation. Prior avatar approaches face trade-offs in rendering quality, animation readiness, or character-shape flexibility.

2. Related Work

  • TADA’s overview initializes from SMPL-X, adds subdivision and personalized displacements, then jointly optimizes geometry and texture from posed RGB and normal renders using SDS.Poses and expressions are resampled during optimization, and each rendered view uses a novel camera view.Source: Figure 3 · p. 3
  • CLIP-guided shape and texture optimization can fail to produce convincing renderings, while full-body appearance GANs are challenging to learn across human and cartoon avatars.The cited discussion describes combining CLIP optimization with a generative appearance model as one way to improve quality.Source: §2 · p. 3
  • SDS uses a pretrained 2D text-to-image diffusion model to optimize 3D representations without training a 3D generative model, while DreamFusion faces slow NeRF optimization and low-resolution supervision.TEXTure optimizes texture maps on an input mesh; DreamFusion optimizes a NeRF representation.Source: §2 · p. 3
  • Many SDS object methods target static content; avatar NeRFs have quality and CG-workflow limitations, while fixed-topology displacement methods struggle with diverse shapes.The passage contrasts these approaches with jointly optimizing shape, expression, and displacement for SMPL-X animation and CG workflows.Source: §2 · p. 3

3. Preliminaries

The preliminaries introduce SMPL-X as a parametric body model controlled by shape, pose, and expression, then describe SDS as a way to optimize a rendered 3D model using a pretrained diffusion model.

3. Preliminaries

  • SMPL-X represents the body, hands, and face with 10,475 vertices and 54 joints, and is parameterized by shape, pose, and expression.Pose includes body-joint, jaw, and finger poses.Source: §3 · pp. 3–4
  • SMPL-X combines a mean template with shape, expression, and pose blend shapes, then uses linear blend skinning to transform the body to a target pose.The skinning function uses the skeleton joints and skinning weights.Source: §3 · p. 4
  • SDS optimizes 3D-model parameters by comparing injected noise in a rendered image with noise predicted by a pretrained diffusion model conditioned on text.The rendered image is noised, and the weighting function depends on the noise level.Source: §3 · p. 4

4. Method

TADA builds text-prompted avatars on an upsampled SMPL-X model with learnable displacements, then jointly optimizes geometry and texture using diffusion losses. Training on varied poses and expressions is used to preserve SMPL-X correspondence for animation.

4.1. SMPL-X+D Representation

  • Adaptive subdivision and learnable displacements refine SMPL-X while allowing varied human-like and anime character shapes.Subdivision targets low-density areas such as the body and back of the head, producing more uniformly distributed vertices and smoother skinning weights.Source: §4.1 · p. 4

4.2. Consistent Geometry and Texture Learning

  • Stable Diffusion guides texture from rendered images, with camera views alternating between full-body and zoomed-in head perspectives.The head view supports detailed face reconstruction; texture optimization increases image resolution from 32 × 32 to 512 × 512, while normal and color images for the geometry loss remain at 512 × 512.Source: §4.2 · pp. 4–5

4.3. Training with Animations

  • TADA supports holistic face, body, and hand animation, with examples driven by TalkSHOW/TTS and priorMDM motions.The examples animate generated avatars of Lionel Messi and Mabel Pines in Gravity Falls.Source: Figure 4 · p. 5

4.2. Consistent Geometry and Texture Learning

  • SDS on rendered RGB and normal-image latents combines texture guidance with geometry consistency to align appearance and shape.The method interpolates the RGB and normal latents for the geometry-consistency loss, alongside the texture SDS objective.Source: §4.2 · pp. 4–5

4.3. Training with Animations

  • Training with sampled jaw poses helps preserve facial semantic correspondence to SMPL-X and avoid artifacts when the avatar is animated.The authors report little visible difference from animating SMPL-X expression parameters and suspect they may matter with a higher-resolution face mesh.Source: §4.3 · p. 5

5. Experiments

The experiments evaluate TADA’s animation, avatar quality, and component effects through demonstrations, qualitative comparisons, a user study, and ablations.

5.1. Expressive Holistic Body Animation

  • TADA supports natural full-body animation of the face, body, and hands, including with recent text-to-animation methods that output SMPL-X.The demonstrations use text-to-audio with TalkSHOW or text-to-motion with priorMDM, converted to SMPL-X animations.Source: §5.1 · p. 5

5.2. Diverse Range of Avatars

  • TADA generates avatars spanning animated-film characters, celebrities, and custom characters, with varied shapes, appearances, and clothing styles.The custom characters include prompts generated by ChatGPT.Source: §5.2 · pp. 6–7, Figure 5 · p. 6

5.4. Quantitative Evaluation.

  • TADA receives significantly higher user preference than the baselines for geometry, texture, and consistency with input prompts.The A/B study used 17 CV students and 27 randomly selected prompts, comparing TADA with AvatarCLIP and DreamAvatar.Source: Table 1 · p. 7, §5.4 · p. 7

5.3. Qualitative Comparison

  • Against full-body and head-avatar baselines, TADA produces realistic textures, varied body shapes, fewer geometric artifacts, and well-aligned head geometry and textures.DreamFace is biased toward natural head shapes, while HeadSculpt produces noisy geometry and texture that make its output less useful for animation.Source: Figure 6 · p. 6, §5.3 · p. 7, Figure 7 · p. 7

5.5. Ablation Study

  • Geometry consistency loss improves geometry-texture alignment, while animation training improves face geometry and SMPL-X semantic correspondence, especially around the mouth.The reported effects include alignment on Superman’s backside and improved mouth-region geometry.Source: §5.5 · p. 7, Figure 8 · p. 7

5. Experiments

  • Figure 9 depicts five individuals, each shown in two outfits, as a virtual try-on demonstration.The outfits are presented as typical styles personalized from each person’s name and alternative attire imagined by ChatGPT.Source: Figure 9 · p. 8

6. Editing Applications

TADA supports text-controlled virtual try-on, outfit-texture changes, and localized shape editing.

Virtual Try-on

  • Virtual try-on generates personalized outfit textures while preserving the person’s identity.The described outfits can be designed by ChatGPT for a specific person.Source: §6 · p. 8

Texture Editing

  • Changing the input text modifies outfit textures, including clothing color.The authors describe this as useful for quickly exploring character color palettes in film or game design.Source: §6 · p. 8, Figure 10 · p. 8

Local Shape Editing

  • SMPL-X body-part segments support direct face or body swapping between avatars, as well as clothing transfer and prompt-driven local sculpting.The paper presents face editing across four individuals as an example.Source: §6 · p. 8, Figure 11 · p. 8

7. Discussion

TADA’s limitations include incomplete relighting across environments, possible unrealistic animation for avatars far from the base SMPL-X model, and limits in text-only appearance control. The authors also flag deep-fake, intellectual-property, gender, and cultural-diversity concerns.

7. Discussion

  • Relighting across environments, including generated indoor rooms, needs improvement for photo-realistic rendering with human-scene interactions.The discussion identifies separating material, albedo, and lighting with BRDF as a possible direction.Source: §7 · p. 8
  • For avatars that deviate substantially from base SMPL-X, original skinning weights may produce unrealistic animations.The authors suggest jointly learning adaptable skinning weights tailored to text input.Source: §7 · p. 8
  • Text alone may not capture nuanced character appearance; controllable text-to-image models and compositional hair, accessories, and outfits are proposed for further exploration.The discussion specifically identifies greater control over faces and clothing as a potential benefit of controllable text-to-image models.Source: §7 · pp. 8–9
  • Iconic-character generation raises deep-fake and intellectual-property concerns, while stereotyped outputs such as male-only police officers raise gender and cultural-diversity concerns.The authors call for regulation and emphasize inclusivity and avoiding stereotypes.Source: §7 · p. 9

8. Conclusion

TADA generates text-conditioned, textured 3D avatars using a subdivided SMPL-X model and hierarchical optimization. The authors report superiority over other state-of-the-art approaches and compatibility with traditional graphics pipelines.

8. Conclusion

  • TADA generates high-quality, animatable 3D textured avatars from text and is designed to integrate with existing computer-graphics pipelines.The method uses subdivided SMPL-X with learned displacements and UV texture, and trains with animation to preserve semantic correspondence.Source: §8 · p. 9
  • Its contributions include hierarchical optimization with adaptive focal lengths and a geometric consistency loss to align geometry and texture.The underlying avatar uses a subdivided SMPL-X model with learned displacements and UV texture.Source: §8 · p. 9
  • The authors validate the components through ablations and report qualitative and quantitative superiority over other state-of-the-art methods.Source: §8 · p. 9

A. Additional Qualitative Comparisons

Additional comparisons report TADA’s geometry and texture advantages for head, full-body iconic, and customized avatars. The iconic-avatar results are also described as suitable for conventional graphics workflows.

Head Avatar Generation

Full-body Iconic Avatar Generation

  • For full-body iconic avatars, TADA is reported to produce superior geometry and well-aligned textures, with outputs suitable for conventional graphics workflows.NeRF-based methods are described as producing low-quality geometric fields incompatible with traditional rasterization and animation; AvatarClip is described as producing minimally clothed geometry and artifact-heavy textures.Source: A. Additional Qualitative Comparisons · p. 13

Full-body Customized Avatars Generation

B. Implementation details

The implementation samples full-body and zoomed head camera views during optimization, while supplementary comparisons and ChatGPT-generated prompts cover several avatar categories and imagined outfits.

B. Implementation details

  • Optimization uses full-body camera views 70% of the time and zoomed head views 30%, with separately sampled elevation and azimuth ranges for each view.Full-body elevation is sampled from 60°–90° and head-view elevation from 75°–85°; azimuth ranges are −180°–180° and −30°–30°, respectively.Source: B. Implementation details · p. 13
  • Figure 13 reports higher geometry and texture quality than baselines for iconic avatars; Figures 12 and 14–16 show head, iconic, and customized comparisons.Source: Figure 12 · p. 13, Figure 13 · p. 14, Figure 14 · p. 15

C. Prompts used by ChatGPT

  • ChatGPT-generated prompts cover celebrities, fictional characters, general job descriptions, and imagined outfits for virtual try-on.Source: C. Prompts used by ChatGPT · p. 18

Related papers

Loading 2308.10899v1…