Source-linked AI summary
EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation
Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, Zhaoxin Fan
TL;DR
Existing speech-driven 3D face animation methods often neglect emotional variation and struggle to disentangle emotion from speech content. EmoTalk uses an emotion disentangling encoder, an emotion-guided decoder, and the 3D-ETF dataset; experiments and user studies report stronger and more diverse emotional facial movements.
Problem
Existing methods often neglect emotional facial expressions and do not clearly disentangle speech emotion from content.
Method
EmoTalk disentangles speech emotion and content, then uses emotion-guided feature fusion to generate 3D facial animation trained with the 3D-ETF dataset.
Results
The method outperforms existing state-of-the-art methods and exhibits more diverse facial movements in experiments and user studies.
Takeaways & Limitations
EmoTalk generates 3D facial animation that conveys emotional expressions while retaining speech-driven content.
Takeaways & Limitations
Inference is slowed by reliance on a large-scale audio pre-training model, and pseudo-3D data cannot represent skin micro-expressions.
Abstract
from arXiv · showhide
Speech-driven 3D face animation aims to generate realistic facial expressions that match the speech content and emotion. However, existing methods often neglect emotional facial expressions or fail to disentangle them from speech content. To address this issue, this paper proposes an end-to-end neural network to disentangle different emotions in speech so as to generate rich 3D facial expressions. Specifically, we introduce the emotion disentangling encoder (EDE) to disentangle the emotion and content in the speech by cross-reconstructed speech signals with different emotion labels. Then an emotion-guided feature fusion decoder is employed to generate a 3D talking face with enhanced emotion. The decoder is driven by the disentangled identity, emotional, and content embeddings so as to generate controllable personal and emotional styles. Finally, considering the scarcity of the 3D emotional talking face data, we resort to the supervision of facial blendshapes, which enables the reconstruction of plausible 3D faces from 2D emotional data, and contribute a large-scale 3D emotional talking face dataset (3D-ETF) to train the network. Our experiments and user studies demonstrate that our approach outperforms state-of-the-art methods and exhibits more diverse facial movements. We recommend watching the supplementary video: https://ziqiaopeng.github.io/emotalk
1. Introduction
EmoTalk addresses the neglect and entanglement of emotion in speech-driven 3D facial animation with an end-to-end emotion-aware method and a large-scale pseudo-3D dataset.
- Motivation: Current speech-driven methods prioritize lip synchronization while neglecting emotional facial variation, which is important for natural communication.
- Motivation: Because speech content and emotion are entangled, manually controlling emotion can contradict the speech, such as producing happiness from angry speech.
- Method: Its emotion-guided feature fusion decoder combines identity, emotional, and content features to output 52 emotion-enhanced blendshape coefficients.
- Dataset: The 3D-ETF dataset provides blendshape coefficients and mesh vertices, constructed from emotional 2D data using semantic FLAME templates and blend linear skinning.
- Results: The proposed end-to-end network achieves varied emotional expressions and outperforms existing state-of-the-art methods.
- Method: EmoTalk introduces an emotion disentangling encoder that separates speech content and emotion to provide facial animation with clear emotional information.
2. Related Work
Speech-driven 3D facial animation has expanded beyond 2D talking heads, but emotional expression remains limited by datasets and by the difficulty of separating speech content from emotion.
- 3D facial animation methods target character models used in applications such as games and virtual reality, unlike many 2D talking-head approaches.
- VOCA and FaceFormer improve speech-aligned mouth movements but produce limited upper-face or emotional expression because they rely on VOCASET.
- MeshTalk synthesizes audio-uncorrelated movements such as blinking and eyebrow motion, yet emotional expression remains insufficient in current 3D methods.
- Speech emotion recognition is essential but challenging for generating realistic talking-head animations from speech.
- Emotion disentanglement separates emotion from speech content so neural networks can learn more specific features and reconstruct speech representations.
3. Method
EmoTalk disentangles speech content and emotion, then fuses those representations with personal-style and emotional-level controls to generate 52-dimensional facial blendshape sequences. Cross-reconstruction, temporal and classification losses, and a pseudo-3D emotional dataset support training.
- 3. Method: EmoTalk maps speech, emotional level, and personal style to 52-dimensional facial blendshape coefficients for controllable 3D facial animation.
- 3.1. Emotion disentangling encoder: The emotion disentangling encoder uses separate content and emotion latent spaces to address the difficult speech-to-emotional-expression mapping.
- 3.1. Emotion disentangling encoder: Pseudo-training pairs combine different contents and emotions, requiring reconstruction of corresponding ground-truth blendshapes to enforce disentanglement.
- 3.2. Emotion-guided feature fusion decoder: The emotion-guided decoder fuses 256-dimensional emotion, 512-dimensional content, 32-dimensional personal-style, and 32-dimensional emotional-level features.
- 3.2. Emotion-guided feature fusion decoder: Emotion-guided attention combines decoder features with emotion-latent outputs, and a final fully connected layer produces 52 blendshape coefficients.
- 3.3. Loss function: Training combines cross-reconstruction, self-reconstruction, velocity, and classification losses, with weights 1.0, 1.0, 0.5, and 0.1, respectively.
- 3.3. Loss function: Velocity loss compares predicted and ground-truth frame differences to encourage smoother facial-expression sequences and reduce jitter.
- 3.4. Datasets construction: The 3D-ETF dataset contains approximately 700,000 blendshape frames spanning over 6.5 hours, with blendshape coefficients and mesh vertices derived from 2D emotional data.
4. Experiments
Experiments evaluate EmoTalk on reconstructed emotional talking-face data using quantitative metrics, qualitative comparisons, user studies, and ablations. The method shows lower lip and emotional errors, strong zero-shot generalization, favorable perceptual judgments, and component-specific benefits.
- Datasets: RAVDESS and HDTF were used to construct the 3D-ETF training and evaluation data, with HDTF also supporting mouth-shape generalization.RAVDESS contains emotional speech from 24 actors; five hours of HDTF video were selected and partitioned for generalization experiments.
- Quantitative evaluation: LVE measures lip synchronization error, while EVE measures emotional-expression error in the eye and forehead regions.Both metrics use maximum ℓ2 vertex displacement per frame, with average test-set error reported.
- Quantitative evaluation: EmoTalk achieved lower LVE and EVE than VOCA, MeshTalk, and FaceFormer on RAVDESS and HDTF.The blendshape outputs were converted to FLAME mesh vertices for comparison against ground truth.
- Quantitative evaluation: EmoTalk outperformed competing methods on VOCA-Test, including in zero-shot settings.The authors attribute this generalization partly to blendshape-based control and learning from sufficient 2D datasets.
- Qualitative evaluation: Qualitative comparisons showed expressive emotion-matched movements on RAVDESS and noticeable motion range with near-ground-truth performance on VOCA-Test.The model also showed more pronounced lip movements and improved alignment with speech patterns in perceptual comparisons.
- User study: In full-face user voting, EmoTalk surpassed MeshTalk and FaceFormer by 65.9% and 64.6%, respectively.The study separately evaluated full-face, lip-synchronization, and emotion-expression comparisons across 120 questions.
- Ablation experiment: Removing the emotion disentangling encoder increased emotional-expression error, while removing emotion-guided attention increased EVE.Removing velocity loss caused noticeable animation jitter, and omitting HDTF increased LVE by about 0.5mm.
5. Limitations
The method has limitations in inference speed, motion coverage, and training-data precision. Its pseudo-3D data cannot represent skin micro-expressions, restricting outputs to overall emotional state.
- Limitations: A large-scale audio pre-training model increases inference time and hinders real-time applications.This is identified as a limitation planned for future work.
- Limitations: The network outputs 52 blendshape coefficients but excludes head movements such as shakes and rotations.The authors suggest combining blendshape coefficients with FLAME to control facial expressions and head movements.
- Limitations: Because training data is derived from 2D images, the pseudo-3D data is less precise than 3D-scanned data and cannot represent skin micro-expressions.Consequently, the method reflects only the animated face’s overall emotional state.
6. Conclusion
EmoTalk generates emotional speech-driven 3D face animation using disentangled speech representations and emotion-guided feature fusion. The paper also introduces the 3D-ETF dataset and reports stronger methods and user feedback.
- The emotion disentangling encoder separates speech into emotional and content components for facial animation.
- The emotion-guided feature fusion decoder emphasizes emotion-related features to enhance facial expressiveness.
- The 3D-ETF dataset contains blendshape coefficients and mesh vertices for training emotional talking-face models.
- Parameterized transformations support conversion between blendshape coefficients, FLAME models, and different facial animations.
- Experiments report that EmoTalk outperforms existing state-of-the-art methods and receives better user-experience feedback.
A. Implementation details
The supplementary implementation description focuses on EmoTalk’s architecture and its two key components: emotion disentangling and emotion-guided feature fusion.
- EmoTalk’s implementation is organized around an emotion disentangling encoder and an emotion-guided feature fusion decoder.
A.1. Training details
Training uses preprocessed audio and video, end-to-end optimization, cross-reconstruction for emotion disentanglement, and concatenated emotion and personal-style features. Blendshape outputs are transformed into FLAME vertex parameters.
- Video is processed at 30 frames per second, audio at 16 kHz, and facial capture provides 52 blendshape coefficients per frame.
- The network is trained end-to-end with Adam at a 1e - 4 learning rate and batch size 8 for approximately 8 hours over 80 epochs.
- Emotion disentanglement uses a temporal convolutional network, length interpolation, and cross-reconstruction constraints for training.
- Emotion intensity and personal style are encoded as 32-dimensional embeddings, representing binary intensity and 24 speakers, respectively.
- Blendshape-to-FLAME conversion enables facial-expression transfer across virtual characters using 52 semantically meaningful FLAME head templates.
- Linear weighting of 52 FLAME templates converts EmoTalk’s blendshape coefficients into 5023*3-dimensional vertex parameters.
C. Baseline methods
EmoTalk is compared with VOCA, MeshTalk, and FaceFormer across RAVDESS, HDTF, and VOCASET evaluation settings, using vertex-based comparisons where needed.
- The baseline comparison includes VOCA, MeshTalk, and FaceFormer as three state-of-the-art approaches.
- RAVDESS and HDTF are processed with facial blendshape capture to obtain ground-truth coefficients and corresponding FLAME vertex parameters.
- On VOCASET, pre-trained VOCA and FaceFormer models are used, while MeshTalk is retrained to evaluate vertex error on VOCA-Test.
- EmoTalk cannot be trained on VOCASET because the official data lack blendshape coefficients and contain incompatible marked-face images.
- A transfer module converts EmoTalk’s blendshape outputs into mesh vertices so its errors are comparable with the other methods.
D. Dataset construction details
The 3D-ETF dataset uses facial blendshapes as supervision to reconstruct 3D faces from 2D emotional talking-face data. It is built from processed RAVDESS and HDTF videos, with filtering and mesh conversion applied to the captured coefficients.
- Professional animators fine-tune the facial blendshape capture method to create numerous 3D facial animations from RAVDESS and HDTF data.
- The dataset combines 1,440 RAVDESS videos and 385 HDTF videos, converted to 30 frames per second for per-frame blendshape capture.
- A Savitzky-Golay filter with window length 5 and polynomial order 2 smooths the output coefficients and reduces frame-to-frame jitter.
- RAVDESS contributes 159,702 frames, approximately 1.5 hours of video, while HDTF contributes 543,240 frames, approximately 5 hours.
- The captured blendshape coefficients are converted into mesh vertices using a transform module and included in the dataset.
E. Blendshape capture method
The blendshape capture method predicts 52 facial blendshape values from processed face images and uses animator refinement for realistic emotional facial reconstruction. These coefficients correspond to semantically meaningful FLAME head templates.
- A neural network predicts blendshape coefficients from input video streams, after which professional animators manually fine-tune them for realistic emotional facial reconstruction.
- A ResNet processes cropped and transformed face images paired with blendshape data to regress 52 specific blendshape values.
- An L2 loss constrains the ResNet outputs to support precise regression of facial blendshapes.
- The 52 FLAME head templates correspond to blendshape coefficients and transform those coefficients into the FLAME head model.