Source-linked AI summary
FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesis
Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, Mu Xu
TL;DR
Realistic talking portraits from a single static image remain difficult because existing methods struggle with subtle expressions, global body movements, and dynamic backgrounds. FantasyTalking uses a pretrained video diffusion transformer with dual-stage audio-visual alignment, facial-focused identity preservation, and motion-intensity control. The authors report improvements in video quality, motion diversity, and identity consistency, while identifying real-time acceleration and interactive dialogue as future directions.
Problem
Generating realistic, audio-controlled talking portraits from a single static portrait remains challenging because subtle expressions, global body movements, and dynamic backgrounds are difficult to capture.
Method
FantasyTalking combines a pretrained video diffusion transformer with clip-level global alignment, frame-level lip refinement, facial-focused cross-attention, and motion-intensity modulation.
Results
FantasyTalking outperforms existing state-of-the-art methods in video quality, motion diversity, and identity consistency.
Takeaways & Limitations
The framework supports synchronized portraits with controllable facial and body movements while preserving identity consistency.
Takeaways & Limitations
Further acceleration is needed for scenarios with higher real-time requirements, including live streaming and interactive real-time applications.
Abstract
from arXiv · showhide
Creating a realistic animatable avatar from a single static portrait remains challenging. Existing approaches often struggle to capture subtle facial expressions, the associated global body movements, and the dynamic background. To address these limitations, we propose a novel framework that leverages a pretrained video diffusion transformer model to generate high-fidelity, coherent talking portraits with controllable motion dynamics. At the core of our work is a dual-stage audio-visual alignment strategy. In the first stage, we employ a clip-level training scheme to establish coherent global motion by aligning audio-driven dynamics across the entire scene, including the reference portrait, contextual objects, and background. In the second stage, we refine lip movements at the frame level using a lip-tracing mask, ensuring precise synchronization with audio signals. To preserve identity without compromising motion flexibility, we replace the commonly used reference network with a facial-focused cross-attention module that effectively maintains facial consistency throughout the video. Furthermore, we integrate a motion intensity modulation module that explicitly controls expression and body motion intensity, enabling controllable manipulation of portrait movements beyond mere lip motion. Extensive experimental results show that our proposed approach achieves higher quality with better realism, coherence, motion intensity, and identity preservation. Ours project page: https://fantasy-amap.github.io/fantasy-talking/.
1 INTRODUCTION
FantasyTalking targets realistic, audio-controlled talking portraits by coordinating scene-wide motion, precise lip synchronization, identity preservation, and controllable expression and body-motion intensity.
- Generating an animatable avatar from one static portrait remains challenging, especially for subtle expressions and realistic motions.
- FantasyTalking uses a pretrained video diffusion transformer and clip-level audio-visual alignment to establish coherent global motion across the scene.The alignment covers the portrait, contextual objects, and background.
- Frame-level alignment refines lip movements with a mask so they adhere more closely to the audio content.
- A facial-focused cross-attention module preserves identity while avoiding the motion restrictions associated with full-image reference networks.
- Motion intensity modulation controls facial expression and body-motion intensity beyond lip motion.
- Extensive experiments report new state-of-the-art performance in video quality, temporal consistency, and motion diversity.
2 RELATED WORK
Diffusion-based video generation has progressed from image-model extensions toward DiT architectures, while audio-driven talking-head methods still face challenges in expressive motion and identity preservation.
- Diffusion video methods initially relied mainly on UNet architectures with temporal layers added to pretrained image-generation models.
- DiT-based models have more recently advanced video generation technology.
- Earlier talking-head methods used 3DMM-derived facial animation parameters, but limited 3DMM expressiveness constrained facial and head-motion authenticity.
- End-to-end audio-to-video methods improve potential but commonly use full-image reference networks that restrict broader motion ranges.
- Prior work emphasizes audio-lip synchronization, whereas correlations between audio and other facial expressions and body movements remain underexplored.
3 METHOD
FantasyTalking adapts a pretrained DiT-based video generator for audio-synchronized talking portraits by combining global and lip-level alignment with facial identity preservation and controllable motion intensity.
- Framework overview: FantasyTalking generates audio-synchronized videos from a single reference image while maintaining the person’s identity during motion.The framework combines dual-stage audio-visual alignment, identity learning, and motion control.
- Framework overview: The method uses a pretrained video diffusion transformer to establish unified dynamics across the portrait, contextual objects, and background.Its multi-modal alignment framework is built on a DiT-based video generation model.
- Dual-stage audio-visual alignment: Clip-level training establishes global audiovisual dependencies and learns both non-verbal cues and audio-synchronized lip dynamics across full-length sequences.The learned cues include eyebrow movements, shoulder motions, and lip dynamics.
- Dual-stage audio-visual alignment: Frame-level training refines lip motion through one-to-one audio-video alignment and a latent-space lip mask derived from precise pixel-space lip masks.MediaPipe extracts the lip masks, which are projected into latent space by trilinear interpolation.
- Dual-stage audio-visual alignment: The lip constraint is applied probabilistically because exclusive lip-specific regularization can suppress natural head movements and background dynamics.The probability η balances lip-motion focus against overall movement naturalness.
- Identity preservation: Facial identity is preserved by cropping facial regions, extracting ArcFace features, aligning them with Q-Former, and injecting the resulting identity embedding through cross-attention.The facial-focused design avoids the full-frame reference processing used by conventional reference networks, which can restrict dynamic effects.
- Motion intensity modulation: A motion intensity modulation network controls facial expression and body movement amplitudes using normalized landmark and joint-variance coefficients.The facial expression and body movement coefficients are normalized to [0, 1].
4 EXPERIMENTS
Experiments compare FantasyTalking with prior methods on tame and wild talking-head datasets, assess controllable motion intensity, and validate video quality, synchronization, identity preservation, and motion diversity through user studies.
- Evaluation Setup: The evaluation uses FID, FVD, Sync-C, Sync-D, IDC, ES, and aesthetic-related measures to assess quality, synchronization, and identity characteristics.The selected baselines include UNet-based Aniportrait, EchoMimic, and Sonic, plus the DiT-based Hallo3.
- Dataset Comparisons: FantasyTalking achieves the best scores on the tame dataset for FID, FVD, IDC, ES, and Aesthetic score, with best or second-best synchronization results.The tame dataset emphasizes lip synchronization and facial-expression accuracy.
- Dataset Comparisons: On the wild dataset, FantasyTalking achieves the best results across all metrics while producing more natural foreground and background variations.The wild dataset contains substantial variation in foreground and background elements.
- Motion Intensity Control: At subtle and natural motion levels, FantasyTalking and Sonic both control motion intensity while maintaining lip synchronization; FantasyTalking performs better for intense movements.The evaluated levels are subtle (β=0.5, ω_l=0.1, ω_b=0.1), natural (β=1.0, ω_l=0.5, ω_b=0.5), and intense (β=2.0, ω_l=1.0, ω_b=1.0).
- Qualitative Comparisons: Compared with Hallo3, FantasyTalking produces more authentic expressions, head movements, and background dynamics, while Hallo3 shows facial artifacts and unrealistic background motion.The comparison is presented for wild talking-head videos.
- User Studies: In a 24-participant user study, FantasyTalking outperforms baseline methods in lip synchronization, video quality, identity preservation, and motion diversity.Participants rated each dimension from 0 to 10, with particularly notable gains in motion diversity.
5 ABLATION STUDIES AND DISCUSSION
Ablation studies examine dual-stage audio-visual alignment, identity preservation, and motion intensity modulation. The discussion also identifies real-time acceleration and interactive dialogue as directions for broader deployment.
- DAVA Ablation: Using only clip-level alignment significantly reduces Sync-C, showing that clip-level training alone does not learn precise audio-to-lip correspondence.Combining clip-level and frame-level alignment achieves precise synchronization while enhancing character and background dynamics.
- Identity Preservation Ablation: Removing identity preservation sharply decreases IDC and introduces facial artifacts and distortions, whereas focused facial knowledge maintains identity alongside lip synchronization and rich motion.The proposed facial-focused identity method improves identity retention and overall video quality.
- Motion Intensity Ablation: The motion intensity modulation ablation varies ω_l and ω_b against FVD and SD while fixing the other parameter at the neutral value 0.5.Figure 10 reports the quantitative effects of adjusting the limb- and body-related motion intensity coefficients.
- Discussion: Acceleration strategies remain an open direction for scenarios with higher real-time requirements, including live streaming and interactive real-time applications.The paper also points to interactive portrait dialogue with real-time feedback as a broader application direction.
6 CONCLUSIONS
FantasyTalking captures audio-linked lip movements, facial expressions, and body motions while preserving facial identity for natural, varied portrait animation.
- FantasyTalking uses dual-stage audio-visual alignment to capture relationships between audio signals, lip movements, facial expressions, and body motions.
- A facial-focused identity-preservation approach retains facial features accurately throughout generated videos.
- A motion network controls the magnitude of facial expressions and body movements, supporting natural and varied animations.
- Qualitative and quantitative experiments show improvements over existing SOTA methods in video quality, motion diversity, and identity consistency.