Source-linked AI summary
AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
Huawei Wei, Zejun Yang, Zhisheng Wang
TL;DR
AniPortrait targets the challenge of generating realistic, temporally consistent portrait animations from audio and a reference image. It predicts 3D facial motion, projects it to 2D landmarks, and uses diffusion-based video generation; experiments report natural faces, varied poses, and strong visual quality, with editable representations supporting reenactment. A stated limitation is the cost of obtaining large-scale, high-quality 3D data and the resulting uncanny-valley effect in expressions and head postures.
Problem
Generating realistic, expressive, and temporally consistent portrait animations from audio and static images requires coordinated lip movements, facial expressions, and head positions.
Method
AniPortrait extracts audio-driven 3D facial meshes and head poses, projects them into 2D landmarks, and uses a diffusion model with a motion module to generate reference-consistent portrait videos.
Results
Experiments report superior facial naturalness, pose diversity, and visual quality for AniPortrait animations.
Takeaways & Limitations
Editable intermediate 3D representations support applications including facial motion editing and face reenactment.
Takeaways & Limitations
The method requires costly large-scale, high-quality 3D data, and generated expressions and head postures can still exhibit an uncanny-valley effect.
Abstract
from arXiv · showhide
In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait
1 Introduction
AniPortrait addresses the challenge of producing expressive, temporally consistent portrait animations from audio and a static image. It combines audio-derived facial representations with diffusion-based video generation and reports strong facial naturalness, pose diversity, and visual quality.
- High-quality portrait animation requires coordinated lip movements, facial expressions, and head positions while preserving temporal consistency.
- Existing GAN-, NeRF-, and motion-decoder-based methods often have limited generalization and stability for high-quality visual generation.
- AniPortrait extracts 3D facial meshes and head poses from audio, projects them into 2D landmarks, and uses diffusion to generate portrait video.
- Experiments report facial naturalness, varied poses, and excellent visual quality, while the editable intermediate representations support facial motion editing and reenactment.
- The framework uses concurrent training for its audio-to-landmark and landmark-to-video stages.
2 Method
The method has two modules: Audio2Lmk predicts 3D facial motion from speech and projects it into 2D landmarks, while Lmk2Video renders those landmarks as a reference-consistent portrait animation. The video module combines diffusion, temporal motion modeling, appearance conditioning, and multi-scale landmark guidance.
- Audio2Lmk: Wav2vec features are converted into 3D meshes with two fully connected layers, while a separate transformer decoder predicts pose using cross-attention.The audio-to-mesh and audio-to-pose modules use the same wav2vec backbone but do not share weights; both are trained with L1 loss.
- Lmk2Video: Lmk2Video takes a reference portrait and landmark sequence to generate temporally consistent frames aligned with the target motion.The module is designed to preserve appearance consistency with the reference image across the animation.
- Lmk2Video: Lmk2Video uses Stable Diffusion 1.5, a temporal motion module, and ReferenceNet to combine multi-frame generation with reference-image appearance information.ReferenceNet mirrors the Stable Diffusion 1.5 structure and helps maintain consistent face identity throughout the output video.
- Lmk2Video: A ControlNet-style multi-scale PoseGuider injects landmark features into different backbone blocks to better capture intricate lip movements.The design also adds the reference image’s landmark as an input, with cross-attention relating it to each frame’s target landmarks.
3 Experiments
The experiments use speech, facial-video, and landmark-processing pipelines to train AniPortrait. Qualitative results show realistic animations, and editable intermediate representations enable face reenactment.
- Training: Audio2Lmk training uses wav2vec2.0, MediaPipe-derived 3D annotations, a nearly hour-long single-speaker speech dataset, and HDTF for Audio2Pose.
- Training: Lmk2Video training uses VFHQ and CelebV-HQ, with MediaPipe processing to extract 2D facial landmarks.Training follows two steps: first the backbone, ReferenceNet, and PoseGuider are trained; then the motion module is trained while other components are frozen.
- Results: AniPortrait generates animations described as striking in quality and realism in the reported qualitative results.
- Results: Editing the intermediate 3D representation enables manipulation of the final output, including face reenactment by changing landmark identity.
4 Conclusion and Future Work
The framework generates portrait videos from audio and a reference image, producing smooth lip motion, natural head movements, realistic image quality, and lifelike motion. However, its reliance on intermediate 3D representations and costly high-quality 3D data leaves generated expressions and head postures susceptible to the uncanny valley effect.
- Audio and a reference image produce portrait videos with smooth lip motion and natural head movements.
- The diffusion model's generalization abilities support realistic image quality and convincingly lifelike motion.
- The method requires intermediate 3D representations, while large-scale, high-quality 3D data are costly to obtain.
- Generated facial expressions and head postures remain susceptible to the uncanny valley effect.
- Future work will predict portrait videos directly from audio, following the approach of EMO.