Source-linked AI summary
OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animation
Qijun Gan, Ruizi Yang, Jianke Zhu, Shaofei Xue, Steven Hoi
TL;DR
Existing audio-driven animation methods often emphasize facial motion, leaving full-body synchronization and precise prompt control limited. OmniAvatar combines pixel-wise multi-hierarchical audio embeddings with LoRA-based adaptation to generate synchronized, adaptive full-body videos. Experiments report state-of-the-art facial and semi-body results and precise text control across diverse domains, while long-video stability, complex multi-character control, and inference speed remain limitations.
Problem
Existing audio-driven human-animation methods largely focus on facial movements, while full-body animation and precise text-based control remain challenging.
Method
OmniAvatar uses pixel-wise multi-hierarchical audio embedding, LoRA-based DiT training, and frame-overlap with reference-image embedding for audio-driven full-body video generation.
Results
OmniAvatar achieves state-of-the-art facial and semi-body portrait video generation and supports precise text-based control across diverse video domains.
Takeaways & Limitations
The model supports human-animation videos with natural body movements, controllable emotions and gestures, dynamic backgrounds, and varied application scenarios.
Takeaways & Limitations
The model inherits Wan’s color shifts and long-video error propagation, struggles with complex multi-character text control, and has long diffusion-based inference times.
Abstract
from arXiv · showhide
Significant progress has been made in audio-driven human animation, while most existing methods focus mainly on facial movements, limiting their ability to create full-body animations with natural synchronization and fluidity. They also struggle with precise prompt control for fine-grained generation. To tackle these challenges, we introduce OmniAvatar, an innovative audio-driven full-body video generation model that enhances human animation with improved lip-sync accuracy and natural movements. OmniAvatar introduces a pixel-wise multi-hierarchical audio embedding strategy to better capture audio features in the latent space, enhancing lip-syncing across diverse scenes. To preserve the capability for prompt-driven control of foundation models while effectively incorporating audio features, we employ a LoRA-based training approach. Extensive experiments show that OmniAvatar surpasses existing models in both facial and semi-body video generation, offering precise text-based control for creating videos in various domains, such as podcasts, human interactions, dynamic scenes, and singing. Our project page is https://omni-avatar.github.io/.
1 Introduction
OmniAvatar addresses limitations in audio-driven human animation by generating full-body videos with synchronized lip movements, adaptive body motion, and more precise text control. Experiments report leading facial and semi-body results across varied video domains.
- Most existing audio-driven animation methods focus on facial movements, limiting natural full-body animation.
- OmniAvatar maps audio into the latent space pixel-wise and uses multi-hierarchical embeddings to improve spatial audio perception and lip synchronization.LoRA-based training preserves foundation-model capabilities while adapting the model to audio features.
- The model generates audio-driven full-body videos with natural and adaptive body movements and accurate lip-syncing.
- OmniAvatar provides more precise text-based control for generating human animation videos.
- Extensive experiments show leading results in facial and semi-body portrait video generation on test datasets.
- The model supports videos across podcasts, human-object interactions, dynamic scenes, and singing, with controllable emotions, gestures, and dynamic backgrounds.
2 Related Work
Related work progresses from image- and video-diffusion foundations to end-to-end audio-driven human animation. Existing systems improve realism, temporal coherence, lip-syncing, motion, controllability, and efficiency through varied conditioning and attention mechanisms.
- Image-pretrained diffusion models were extended temporally, while later transformer architectures targeted long-range dependencies and semantic consistency.
- Recent text-to-video systems use vision-language pretraining, hierarchical token fusion, and multimodal prompts for fine-grained conditional control.
- Early audio-driven animation used cascaded motion prediction and rendering pipelines, but these methods had limited expressiveness and temporal drift.
- Diffusion-based human-animation systems increasingly use multi-stage refinement and long-range attention to improve realism and coherence.
- Audio-driven methods balance weak audio cues with strong visual cues and use long-term motion, emotion-aware objectives, and end-effector guidance to refine animation.
- Localized attention and gesture-adaptive conditioning are used to improve controllability and efficiency for human animation.
3 OmniAvatar
OmniAvatar combines latent diffusion with pixel-wise, multi-hierarchical audio conditioning and LoRA adaptation for audio-driven avatar videos. Its pipeline also uses reference embeddings and frame overlap to support identity preservation and long-video consistency.
- 3.1 Preliminaries: OmniAvatar uses latent diffusion with Wan2.1’s transformer-based denoising network as its foundation for video generation.The model denoises video latents progressively and uses full attention in latent space.
- 3.2 Audio Embedding Strategy: Audio Pack rearranges padded audio features and maps them into audio latents with a linear layer aligned to the video latent space.Audio features are grouped according to the 3D VAE’s compression rate before linear projection.
- 3.2 Audio Embedding Strategy: Pixel-wise audio embedding directly fuses audio features into the video latent, distributing audio information across video pixels for synchronized lips and holistic body movements.The approach avoids relying solely on cross-attention between audio and visual features.
- 3.2 Audio Embedding Strategy: Multi-hierarchical audio embedding inserts audio features at different DiT stages while restricting injection to selected layers to limit excessive influence on latent features.This design is intended to preserve audio information through deep networks without overwhelming the visual latents.
- 3.3 LoRA Training: LoRA adapts attention and feed-forward layers to learn audio-conditioned behavior without altering the foundation model’s underlying capacity.The strategy addresses degradation and overfitting observed when all model layers are updated.
- 3.4 Long Video Generation: Reference image embeddings preserve avatar identity, while frame overlap supports temporal consistency during long-video inference.The reference frame’s latent representation is repeated and concatenated with video latents at each time step.
4 Experiments
Experiments evaluate OmniAvatar on facial and semi-body generation, ablations, and controllable animation scenarios. The model is assessed against existing methods using qualitative and quantitative comparisons, with additional analysis of training and audio-embedding strategies.
- Experimental Setup: Experiments use AVSpeech and HDTF test data for facial generation, including cropped square faces for AVSpeech.AVSpeech also supplies the semi-body test set described in the comparison setup.
- Talking-Face Comparison: Qualitative facial comparisons report higher image quality, more natural facial movements, improved aesthetics, and more accurate lip-syncing.The reported alignment links the pixel-wise audio-embedding strategy with closer audio and facial-expression correspondence.
- Talking-Face Comparison: OmniAvatar achieves leading talking-face performance in Sync-C and competitive FID, FVD, and IQA results.The reported metrics reflect lip-sync accuracy, image quality, video quality, and perceptual accuracy.
- Semi-Body Comparison: OmniAvatar excels in semi-body generation, particularly for audio-lip synchronization, overall video quality, realistic body movements, and seamless audio-visual synchronization.Table 2 compares OmniAvatar with existing audio-driven semi-body video-generation methods on the AVSpeech test set.
- Controllability: Prompt control supports human-object interactions, gesture manipulation, dynamic backgrounds, and controllable character emotions.Figure 6 illustrates scene, interaction, gesture, and background control, while Figure 7 illustrates facial-expression control.
- Ablations: Full training converges faster and adapts scenes better, but can reduce video quality and cause distortions when training data lacks high-resolution portrait content.The reported degradation includes image-quality loss and motion-related problems associated with low-quality data.
- Ablations: Multi-hierarchical audio embedding improves audio synchronization relative to single-layer embedding in the ablation study.The comparison applies single-layer audio embedding at the model’s middle layer.
5 Conclusion
OmniAvatar is presented as an audio-driven full-body video-generation model combining pixel-wise multi-hierarchical audio embedding with LoRA-based training. Experiments report state-of-the-art facial and semi-body generation, precise text control, and videos spanning multiple domains.
- Conclusion: OmniAvatar combines pixel-wise multi-hierarchical audio embedding with LoRA-based training to synchronize lip movements and generate dynamic body movements.The conclusion frames these components as addressing simultaneous lip-sync and realistic-body-animation challenges.
- Conclusion: Experiments on test datasets report state-of-the-art results for both facial and semi-body portrait video generation.The conclusion also reports precise text-based control for high-quality videos across various domains.
- Conclusion: OmniAvatar supports precise text-based control while generating high-quality videos across various domains.The conclusion identifies this capability as an additional outcome beyond facial and semi-body generation.
6 Appendix
OmniAvatar demonstrates results across additional visual styles and interaction scenarios, while its limitations include long-video degradation, difficult multi-character control, and slow diffusion inference.
- OmniAvatar supports additional scenarios including realistic, plain, oil-painting, and cartoon styles, as well as human-object interaction and moving backgrounds.
- The model inherits Wan’s color shifts and error propagation during long-video generation as inaccuracies accumulate over time.
- Complex text control remains challenging for identifying the speaking character and handling multi-character interactions.
- Diffusion inference requires many denoising steps, producing long inference times that make real-time video generation challenging.