Source-linked AI summary
EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
Zhiyuan Chen, Jiajiong Cao, Zhiquan Chen, Yuming Li, Chenguang Ma
TL;DR
Portrait animation methods often condition on audio or facial landmarks separately, with audio-only driving potentially unstable and landmark-only driving potentially unnatural. EchoMimic jointly trains audio and landmark conditions, supports either or both at inference, and demonstrates superior quantitative and qualitative performance across diverse datasets. The authors note that its current video architecture is adapted from image-processing techniques and could be accelerated for real-time generation.
Problem
Existing portrait-animation methods commonly use audio or facial landmarks separately, while image-level evaluation can overlook facial structure and dynamics.
Method
EchoMimic jointly trains audio signals and facial landmarks, using a Denoising U-Net with reference, landmark, audio, and temporal modules.
Results
EchoMimic demonstrates superior performance in quantitative and qualitative evaluations across diverse public datasets and a collected dataset.
Takeaways & Limitations
The framework supports portrait-video generation from audio, facial landmarks, or their combination, including selected facial landmarks.
Takeaways & Limitations
The current architecture extends Stable Diffusion image-processing techniques to video rather than constituting a genuine video-processing framework.
Abstract
from arXiv · showhide
The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.
1. Introduction
Portrait animation seeks realistic talking-head videos but remains challenged by motion representation, artifacts, and the limited control flexibility of audio-only or landmark-only conditioning. EchoMimic addresses these issues by jointly training audio and facial-landmark inputs, supporting each modality separately or together, and reports superior quantitative and qualitative performance.
- Portrait animation transfers motion and expressions from a source video or conditioning signal to a target portrait image for video generation.
- Diffusion-based portrait animation improves generation capacity but can still produce distortions and artifacts when motion representation or losses are inadequate.
- Audio-only and facial-landmark-only approaches offer different trade-offs in stability and naturalness, motivating more flexible conditioning.
- EchoMimic is jointly trained with audio signals and facial landmarks, enabling portrait-video generation from either modality independently or their combination.
- EchoMimic demonstrates superior performance in quantitative and qualitative evaluations across public datasets and a collected dataset.
2. Related Works
Related work progresses from video-based lip synchronization toward diffusion-based image animation, but existing methods commonly condition on audio or pose separately. Evaluation also often emphasizes image-level metrics, leaving facial structure and dynamics less fully represented.
- Video-based talking-head methods such as Wav2Lip synchronize lip movements with audio but can produce blurred visuals and distorted dental structures.
- Diffusion-based image methods improve portrait-video realism and expressiveness while enabling control over facial motion, pose, and expression.
- SadTalker, AniPortrait, V-Express, and Hallo model audio-conditioned motion through 3D coefficients, landmarks, layered alignment, or hierarchical synthesis.
- Existing image-based methods typically condition synthesis on either audio or pose separately rather than integrating both concurrently.
- Evaluation protocols often rely on FID and E-FID, potentially overlooking facial structure and dynamics.
3. Method
EchoMimic extends Stable Diffusion with reference-image, audio, landmark, and temporal modules for portrait-video generation. Its training combines staged video learning with landmark selection, audio augmentation, spatial supervision, and part-aware motion synchronization.
- 3.1. Preliminaries: EchoMimic builds on Stable Diffusion and uses a VAE to encode images into latent representations before decoding them back to images.
- 3.2. Model Architecture: The framework combines a Denoising U-Net with Reference U-Net, Landmark Encoder, and Audio Encoder modules for multi-input portrait-video generation.
- 3.2. Model Architecture: Reference-Attention preserves reference-image information, Audio-Attention integrates visual and audio content, and Temporal-Attention models dependencies across frames.
- 3.2. Model Architecture: The Landmark Encoder maps facial landmark images into latent-aligned features and adds them element-wise to multi-frame latents.
- 3.2. Model Architecture: Temporal-Attention reshapes hidden states to apply self-attention across frames, learning motion patterns for smoother and more coherent sequences.
- 3.3. Training Details: Training uses two stages: single-frame image-audio and image-pose learning, followed by 12-frame video training where only the temporal model is optimized.
- 3.3. Training Details: Random Landmark Selection drops facial parts during training, while spatial loss and audio augmentation improve spatial learning and robustness.
- 3.4. Inference: Part-aware Motion Synchronization combines a full-face transformation with residual transformations computed separately for facial parts.
4. Experiments
EchoMimic is evaluated across datasets, driving modes, and quantitative and qualitative settings. It shows strong visual quality, temporal consistency, lip synchronization, and controllable facial animation, while the authors identify video-processing and speed-up improvements for future work.
- Quantitative Results: EchoMimic achieves the best HDTF scores across FID, FVD, SSIM, and E-FID, including FID 29.136 and SSIM 0.812.It also records FVD 492.784 and E-FID 1.112.
- Quantitative Results: On CelebV-HQ, EchoMimic attains the lowest FID at 63.258 and the most favorable E-FID at 2.723 among compared methods.The authors describe these results as comparable with other methods on the lower-quality dataset.
- Quantitative Results: On the collected dataset, EchoMimic obtains the lowest FID 43.272 and FVD 988.144, favorable E-FID 1.421, and comparable SSIM 0.691 versus the best 0.699.The findings are reported under challenging and diverse scenarios.
- Qualitative Results: Audio-driven results across portrait styles and audios produce high-resolution, visually consistent videos with synchronized lip movements.The qualitative results are presented as evidence of adaptability across diverse audio inputs and acoustic scenarios.
- Qualitative Results: Landmark-driven animation aligns driving landmarks with the reference face, preserves portrait identity, and handles pose variation and nuanced expressions.The motion synchronization method facilitates control results closer to the reference face shape.
- Ablation Study: Audio plus selected landmarks balances freedom and similarity while maintaining lip synchronization and enabling finer control of expressions such as blinking or closing the eyes.The third driving mode yields intermediate outcomes between audio-only and fully landmark-constrained driving.
5. Conclusions
EchoMimic is presented as a portrait animation approach using audio signals and facial landmarks to generate expressive talking head videos. Evaluations, comparisons, methodology, and ablations support its reported performance and potential impact.
- EchoMimic leverages audio signals and facial landmarks to generate high-quality, expressive talking head videos.
- A novel training strategy enables EchoMimic to produce authentic and visually appealing portrait animations.
- Evaluations across diverse public datasets and comparisons with alternative algorithms report superior performance and robustness.
- Methodology, qualitative and quantitative assessments, and ablation studies collectively reinforce EchoMimic’s efficacy and potential impact in portrait image animation.