Source-linked AI summary
Audio-Driven Emotional Video Portraits
Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chen Change Loy, Xun Cao, Feng Xu
TL;DR
Existing audio-driven talking-head methods emphasize speech-to-mouth synchronization while often neglecting facial emotion. EVP disentangles emotion and content, predicts emotional landmarks, and aligns them to target-video poses for high-fidelity portrait synthesis. Qualitative and quantitative experiments validate the method’s effectiveness.
Problem
Previous audio-driven portrait methods generally synchronize mouth movements with speech but do not model emotion, a key factor in naturalistic portraits.
Method
EVP disentangles audio into content-agnostic emotion and speech-content embeddings, predicts emotional 2D landmarks, and aligns them with target-video poses for rendering.
Results
Qualitative and quantitative experiments validate EVP’s effectiveness, with user evaluations reporting the highest visual-quality and audio-visual-synchronization scores apart from real data.
Takeaways & Limitations
EVP enables emotion control in video-based editing talking-face generation and produces high-fidelity emotional video portraits.
Takeaways & Limitations
The method uses 3D models for pose alignment because inaccurately fitted facial expressions are insufficient to represent detailed emotions.
Abstract
from arXiv · showhide
Despite previous success in generating audio-driven talking heads, most of the previous studies focus on the correlation between speech content and the mouth shape. Facial emotion, which is one of the most important features on natural human faces, is always neglected in their methods. In this work, we present Emotional Video Portraits (EVP), a system for synthesizing high-quality video portraits with vivid emotional dynamics driven by audios. Specifically, we propose the Cross-Reconstructed Emotion Disentanglement technique to decompose speech into two decoupled spaces, i.e., a duration-independent emotion space and a duration dependent content space. With the disentangled features, dynamic 2D emotional facial landmarks can be deduced. Then we propose the Target-Adaptive Face Synthesis technique to generate the final high-quality video portraits, by bridging the gap between the deduced landmarks and the natural head poses of target videos. Extensive experiments demonstrate the effectiveness of our method both qualitatively and quantitatively.
1. Introduction
EVP addresses the limited emotional control of audio-driven video portrait generation by disentangling speech content from emotion and adapting synthesized faces to target-video poses. Its pipeline predicts emotional landmark dynamics and renders photo-realistic portraits.
- Prior audio-driven portrait methods largely synchronize lip movements but omit emotion modeling, limiting portrait naturalism.
- EVP introduces audio-based emotion control for video-edited talking faces, animating full portraits with emotion dynamics matched to speech intonation.
- Cross-Reconstructed Emotion Disentanglement separates audio into duration-independent emotion and duration-dependent speech-content spaces for controllable representations.Dynamic Time Warping forms pseudo-pairs from uneven-length speech because matched sentences with identical content, emotion variation, and duration are difficult to obtain.
- An audio-to-landmark module predicts emotional 2D landmark dynamics, while Target-Adaptive Face Synthesis bridges inferred landmarks with target-video head poses in 3D.A 3D-aware keypoint alignment algorithm projects the landmarks into the target video before rendering.
- The system combines predicted landmarks with target-video edge maps and uses an Edge-to-Video rendering network to produce photo-realistic animations.
2. Related Work
Prior talking-face systems primarily target lip synchronization, while emotional generation remains limited by expression, realism, and scale constraints. Related work spans image-based synthesis, video editing, and emotion-conditioned image translation.
- 2.1. Audio-Driven Talking Face Generation: Image-based talking-face methods improve lip synchronization but generally produce little facial expression or head movement.
- 2.2. Emotional Talking Face Generation: Emotional talking-face approaches add expression modeling, but reported limitations include limited experimental scale and less realistic results.
- 2.1. Audio-Driven Talking Face Generation: Video-based methods improve applicability and photorealistic editing, but commonly edit only mouth regions and leave broader portrait emotion unmodeled.
- 2.3. Facial Expression Editing: Unsupervised emotion-conditioned image translation methods learn compact expression codes or generate continuous facial expressions for specified emotions.
3. Method
The method disentangles audio emotion from speech content, predicts emotional landmark dynamics, and adapts them to target-video poses for portrait synthesis.
- Cross-Reconstructed Emotion Disentanglement: Cross-Reconstructed Emotion Disentanglement learns separate emotion and content information from audio signals.The system uses duration-independent, content-agnostic emotion features and duration-dependent speech-content features.
- Cross-Reconstructed Emotion Disentanglement: Dynamic Time Warping aligns uneven-length speeches with the same content but different emotions to create pseudo training pairs.DTW stretches or shrinks MFCC sequences along time so corresponding speech samples can be aligned.
- Cross-Reconstructed Emotion Disentanglement: Cross-reconstruction combines content and emotion embeddings from different audio samples to reconstruct corresponding swapped samples.Training uses cross-reconstruction, self-reconstruction, classification, and content losses.
- Target-Adaptive Face Synthesis: An audio-to-landmark module predicts landmark displacements from disentangled audio embeddings, while 3D-aware alignment adapts them to target-video poses.The predicted landmarks are aligned using detected target landmarks and 3D face-model parameters before projection to the image plane.
- Target-Adaptive Face Synthesis: The adapted landmarks and target-frame edge map guide a conditional GAN that generates the final portrait frames.The guidance map replaces original facial landmarks with aligned landmarks while retaining edges outside the face region.
- Evaluation: Qualitative and quantitative comparisons evaluate competing solutions using different target-video or target-image protocols and landmark and video-quality measures.Table 1 reports mouth-region and face-region results, while Figure 4 compares examples across speech content and emotions.
4. Experiment
Experiments evaluate EVP against prior methods, analyze emotion disentanglement and emotion editing, and ablate the cross-reconstruction and 3D-aware alignment components. Results indicate improved emotional fidelity, user-rated quality and synchronization, and smoother, more realistic synthesis with the proposed modules.
- Qualitative and quantitative comparisons: EVP is compared with three representative prior methods on emotional talking-face generation.The comparisons include Chen et al., Song et al., and Wang et al.
- Qualitative and quantitative comparisons: EVP generates high-fidelity emotional talking-face videos, while methods without emotion modeling produce neutral expressions and Wang et al. can show expression inconsistencies and artifacts.The reported issues for Wang et al. include mismatches between mouth emotion and facial expression, implausible expressions under head movements, and changes to characteristics such as hairstyles.
- User study: 50 participants rated 72 videos, and EVP achieved the highest visual-quality and audio-visual-synchronization scores among generated videos and the highest emotion-classification accuracy.Participants scored synchronization and video quality from 1 to 5 and classified emotions without audio; real data remained higher for the first two criteria.
- Emotion editing and disentanglement: EVP achieves 83.58% emotion accuracy, exceeding Wang et al.’s 76.00%, while the MEAD-trained classifier reaches 90.2% on the testing set.The classifier accuracy is reported as evidence that its emotion labels are reasonable, and EVP’s higher accuracy indicates better emotion maintenance than Wang et al.
- Emotion editing and disentanglement: Interpolating continuous emotion latent features enables smooth changes in emotion category and intensity during inference.The reported interpolation uses αE_s+(1−α)E_t to transition between source and target emotion features.
- Emotion editing and disentanglement: Cross-reconstruction clusters samples of the same emotion more tightly and raises synthesized-video emotion accuracy from 69.79% to 83.58%.The latent-space comparison uses t-SNE across eight emotion categories, while the accuracy comparison evaluates the final synthesized clips.
- Ablation study: 3D-aware landmark alignment makes head motions consistent with target videos and avoids contour displacements and visible synthesis artifacts caused by direct 2D landmark use.The ablation contrasts cases with and without 3D keypoint alignment.
5. Conclusion
The paper presents EVP, an audio-driven video-editing system that synthesizes emotional video portraits using disentangled audio representations and target-adaptive landmark alignment. Qualitative and quantitative experiments validate the effectiveness of the approach.
- Conclusion: EVP decomposes audio into disentangled content and emotion embeddings, generates emotional 2D facial landmarks, and aligns them with target-video head poses for high-fidelity portraits.The system combines Cross-Reconstructed Emotion Disentanglement with Target-Adaptive Face Synthesis.
- Conclusion: Qualitative and quantitative experiments validate the effectiveness of EVP for synthesizing emotional video portraits.