PaperlayerView exact revision on arXiv

Source-linked AI summary

MegaAvatar: Controllable Talking Avatar Generation

Junyao Gao, Sibo Liu, Weidong Zhang, Cairong Zhao, Jun Zhang

arXiv:2609.39273v1cs.CV

TL;DR

Existing talking-avatar methods often lack explicit control over global body pose and head motion. MegaAvatar combines SMPL-X, audio, and face conditioning with an audio-to-SMPL-X model; experiments report controllable motion, speech-related facial expressions, and identity preservation.

  • Problem

    Many existing talking-avatar methods rely mainly on audio for mouth movement and synchronization, without explicit control over global body pose and head motion.

    Source: §1 · p. 2

  • Method

    MegaAvatar adds SMPL-X motion guidance and audio and face cross-attention to Wan2.2-TI2V-5B, alongside an audio-to-SMPL-X model conditioned on a reference image and audio.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Experiments report high-quality videos with controllable body and head motion, fine-grained audio-driven facial expressions, and stable identity preservation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    MegaAvatar supports both audio-driven generation without user-provided SMPL-X frames and custom motion control with supplied SMPL-X frames.

    Source: §1 · p. 2, §4 · p. 6

Abstract

from arXiv · show

This report presents \textbf{MegaAvatar}, a controllable talking avatar generation framework built on top of the Wan2.2-TI2V-5B model. Compared with previous talking-avatar methods that mainly rely on audio or reference-image conditioning, we introduce additional SMPL-X-derived 3D guidance, enabling global control over body pose and head motion. Specifically, we render the driving SMPL-X sequence into dense mesh frames and encode them with a lightweight 3D convolutional encoder, whose outputs are injected into the latent tokens to provide overall motion control. Furthermore, we extend Wan2.2-TI2V-5B with additional audio and face cross-attention modules to enable fine-grained expression control and preserve the input identity, respectively. In addition, we implement an audio-to-SMPL-X model to predict an SMPL-X sequence conditioned on the reference image and input audio, allowing MegaAvatar to support audio-driven inference without user-provided SMPL-X frames. Experiments show that MegaAvatar achieves high-quality talking avatar generation with controllable body and head motion, speech-synchronized facial expressions, and consistent identity preservation. MegaAvatar also supports inference with flexible resolutions and video lengths. Codes, dataset, models will be avaliable in https://github.com/Jeoyal/MegaAvatar

1. Introduction

Earlier talking-avatar methods often synchronized mouth motion from audio but lacked explicit control over body pose and head motion. MegaAvatar adds SMPL-X motion guidance and audio and face conditioning; experiments report controllable motion, speech-related expressions, and stable identity.

  • Previous methods often relied on audio for mouth movement and speech synchronization, leaving global body pose and head motion difficult to control precisely.Source: §1 · p. 2
  • SMPL-X-derived 3D guidance provides global body-pose and head-motion control, while audio and face cross-attention support expression control and consistent facial appearance.Source: §1 · p. 2
  • MegaAvatar generates high-quality talking-avatar videos with controllable body and head motion, fine-grained audio-driven expressions, and stable identity preservation.Source: §1 · p. 2

2. Method

MegaAvatar combines SMPL-X, audio, and face conditions in Wan2.2-TI2V-5B to guide motion, speech-related facial movement, and identity. An audio-to-SMPL-X model predicts motion from a reference image and audio when user-provided motion is unavailable.

  • SMPL-X mesh guidance, audio, and face conditions provide global body and head motion control, speech-related facial expressions, and stable facial appearance, respectively.Source: §2 · p. 2
  • Rendered driving-video SMPL-X meshes are encoded by a 3D convolutional encoder and added to latent tokens; a reference mesh feature is added to the reference latent for appearance-motion alignment.Source: §2 · pp. 2–3
  • Frame-level audio features attend to temporally aligned audio windows through cross-attention modules to guide speech-synchronized mouth movement and facial expressions.Source: §2 · p. 3
  • ArcFace identity tokens enter through face cross-attention modules to reduce identity drift and maintain consistent facial appearance across frames.Source: §2 · p. 3
  • The audio-to-SMPL-X model uses part-specific VQ latents and flow matching to predict a motion sequence from the reference image and input audio, then renders it as MegaAvatar motion guidance.Source: §2 · p. 3, Figure 3 · p. 3
  • The audio-to-SMPL-X overview depicts motion prediction from reference SMPL-X and audio through flow matching in a part-specific VQ latent space.Source: Figure 3 · p. 3

3. Experiments

The implementation section describes a talking-video dataset and begins outlining a staged training strategy.

3.1. Implementation Details

  • SpeakerVid-1M contains about 1M clips and 2,000 hours of talking-human video; filtering for single-person videos with paired audio yields SpeakerVid-400K for audio and face training.Source: §3.1 · p. 3

3.2. Qualitative Evaluation

Qualitative evaluations report temporally smooth, speech-aligned predicted SMPL-X sequences and final videos with consistent appearance, motion guidance, and identity preservation.

3.2. Qualitative Evaluation

  • Predicted SMPL-X frames are temporally smooth and align with speech rhythm, supporting audio-driven talking-avatar generation.Figure 4 visualizes the predicted sequences as rendered dense mesh frames.Source: §3.2 · p. 6
  • Final videos follow SMPL-X guidance for global body and head motion while audio controls fine-grained mouth movement and expressions, with stable identity and facial details.Figures 5 and 6 show the final video results.Source: §3.2 · p. 6

4. Conclusion

MegaAvatar combines SMPL-X, audio, and face conditioning to control body and head motion, speech-related expressions, and facial appearance. Its audio-to-SMPL-X model supports both audio-driven generation and user-provided motion control.

4. Conclusion

  • SMPL-X, audio, and face conditions provide global body and head motion control, fine-grained speech-related expressions, and stable facial appearance, respectively.Source: §4 · p. 6
  • The audio-to-SMPL-X model supports both audio-driven generation and user-provided SMPL-X control.Source: §4 · p. 6
  • Together, these components form a framework for high-quality, controllable talking-avatar generation.Source: §4 · p. 6
Loading 2609.39273v1…