Emotional Speech-Driven Animation with Content-Emotion Disentanglement

1. Introduction

• The paper identifies a gap in speech-driven 3D animation: lip-sync has received attention, but modeling emotion during speech remains missing.Such avatars are intended for uses including films and games, telepresence, and digital assistants.Source: §1 · p. 2
• EMOTE targets the conflict between emotion-conveying face shapes and audio-matched lip motion with semantic control over expressed emotion.The approach includes per-frame lip-reading consistency, sequence-level dynamic emotion consistency, and an emotion-content disentanglement mechanism.Source: §1 · p. 2
• Because no large-scale 3D scan dataset with emotional speech is available, EMOTE uses MEAD video and pseudo-ground-truth 3D reconstructions.Directly applying a FaceFormer-like architecture to that pseudo-ground truth produces mediocre motion and does not provide emotion control.Source: §1 · p. 2
• The method treats speech articulation as fast and mouth-localized, while emotion changes more slowly and may affect the whole face.The authors hypothesize that these temporal and spatial differences make the two phenomena disentanglable.Source: §1 · p. 2
• A temporal transformer-based VAE motion prior supports natural, temporally consistent output, while the audio regressor maps speech to its latent space.The contributions also include a bidirectional nonautoregressive architecture described as more efficient than autoregressive transformer-based state-of-the-art methods, and release of pseudo-ground-truth data, trained EMOTE, and code.Source: §1 · p. 2
• EMOTE produces high-quality emotional 3D facial animation with accurate lip-sync and enables emotion editing, demonstrated qualitatively and quantitatively in perceptual studies.The paper also reports better lip-sync than state-of-the-art methods trained on the same data.Source: §1 · p. 2

Semantic Control

• Prior semantic-control approaches offer style interpolation or latent editing, but the cited methods lack EMOTE’s combination of semantic emotion-type control and speech-emotion disentanglement.EmoTalk requires artist-curated training data and controls emotion intensity rather than emotion type.Source: §2 · pp. 2–3

Supervision

• Recent speech-driven 3D methods are generally supervised with paired scans and audio using frame-level vertex losses; EMOTE adds perceptual lip- and emotion-consistency losses with disentanglement.Related work also includes cross-modal and bilabial mouth-closure losses.Source: §2 · p. 3

Motion Prior

• Motion-prior approaches include categorical latent and VQ-VAE priors, while a transformer autoencoder for facial motion was not investigated for speech-driven regression.The cited work also studies motion compression, retargeting, and unconditional generation.Source: §2 · p. 3

2D Talking Head Generation

• Most speech-driven 2D talking-head methods focus on lip-sync; expression-focused approaches operate on 2D video and do not target high-quality 3D output shapes.The passage also notes that some use 3D parametric models to guide expressions.Source: §2 · p. 3

3D Datasets

• Existing speech-driven 3D datasets are limited in size, emotional richness, speaking styles, and subject-shape diversity, while video datasets offer more data for pseudo-ground-truth reconstruction.MEAD is described as suitable because it is lab-captured and has high emotional variety; reconstruction methods can nevertheless be vulnerable to occlusion, shape or expression errors, and temporal instability.Source: §2 · p. 3

Face model

• FLAME takes identity shape β, facial expression ψ, and pose θ (with k = 4 joints) and outputs a 3D mesh of vertices and triangles; EMOTE predicts its expression and jaw-pose parameters.Source: §3 · p. 4

Emotion feature extraction

• The emotion pipeline uses EMOCA’s image network features and a transformer classifier that aggregates them across a video into an emotion class vector and a 256-dimensional feature.The classifier aggregates across time because single-frame features can confuse speech articulation with emotional cues.Source: §3 · p. 4

Speech feature extraction

• Speech is encoded from raw 16 kHz audio with pretrained Wav2Vec 2.0 features, downsampled to 25 Hz to match the video frame rate.The resulting per-frame speech feature has dimension 768.Source: §3 · p. 4

4.2. Emotional Speech-Driven Animation: EMOTE

• EMOTE combines Wav2Vec 2.0 speech features with speaker, emotion, and intensity styling, then decodes latent sequences through a frozen FLINT decoder into FLAME parameters.The styling condition can be set manually at test time to control output emotion.Source: Figure 3 · p. 5, §4.2 · pp. 5–6

4.1. Facial Motion Prior: FLINT

• FLINT is a temporal VAE motion prior over FLAME sequences that reduces high-frequency jitter and unnatural jaw rotations.Its encoder compresses non-overlapping groups of eight frames into latent frames.Source: §4.1 · p. 5

4.2. Emotional Speech-Driven Animation: EMOTE

• EMOTE uses non-autoregressive bidirectional decoding, calling the decoder once with O(1) decoding complexity rather than the O(T) loops of FaceFormer and CodeTalker.Bidirectional decoding also allows consideration of future context.Source: §4.2 · p. 6
• Per-frame lip-reading consistency and sequence-level video-emotion consistency losses supervise speech articulation and emotional content at different temporal scales.The lip-reading loss compares features from cropped mouth regions; the video-emotion loss compares emotion features of the original and rendered sequences.Source: §4.2 · p. 6
• Emotion-content disentanglement swaps emotion conditions between sequences and trains outputs to retain original lip-reading features while matching the new emotion.The mechanism supports changing emotion while preserving the articulation associated with the audio.Source: §4.2 · pp. 6–7

5. Implementation Details

• MEAD training and validation use 32 and 7 different subjects, respectively, while evaluation uses speaker-disjoint LRS3 test audio.MEAD contains 48 subjects; the paper uses 39, with different subjects in its training and validation splits.Source: §5 · p. 7
• Because MEAD has no 3D meshes, the authors refine a reconstruction model combining EMOCA, SPECTRE, MICA, and Mediapipe supervision to produce reference face shapes.They finetune the image encoders on MEAD with added keypoint and lip-articulation losses.Source: §5 · p. 7
• EMOTE training first uses vertex-loss supervision, then freezes Wav2Vec 2.0 and adds differentiable rendering, perceptual losses, and disentanglement for two finetuning epochs.The first stage runs for 20 epochs; FLINT is trained on MEAD for 500 epochs.Source: §5 · p. 7

6. Experiments

• The evaluation combines perceptual and qualitative assessments because lip-sync and emotional-content quality are difficult to evaluate automatically.The lip-articulation study compares EMOTE with CodeTalker, FaceFormer, MeshTalk, and VOCA using 15 audio sequences; the ablation study evaluates emotion communication and lip-sync.Source: §6 · p. 7, §6.1 · pp. 7–8

6.1. Perceptual Studies

• In a fair comparison, EMOTE outperforms FaceFormer retrained on MEAD, despite being trained on pseudo-ground-truth rather than high-quality scan data.EMOTE’s lip-sync is also preferred over scan-trained MeshTalk, while VOCASET-trained methods benefit from higher-quality training scans.Source: §6.1 · p. 8
• The perceptual ablation study finds that the design choices matter, while the static per-frame emotion-loss variant performs similarly to EMOTE.Participants judged both emotion communication and lip-sync across ablated models.Source: §6.1 · p. 8

6.2. Qualitative Results

• The qualitative comparison shows that EMOTE generates accurate lip animations, while baselines can have inferior lip-sync and FaceFormer trained on MEAD struggles with neutral expression.FaceFormer trained on MEAD can produce emotional faces, but its emotions are not well disentangled and it lacks emotional control.Source: Figure 5 · p. 8, §6.2 · p. 8
• Removing disentanglement terms weakens lip-sync, especially for high-intensity emotions; removing lip-reading loss also harms lip-sync, while removing video emotion loss weakens emotional cues.Removing FLINT makes motion temporally unstable and produces artifacts; removing perceptual losses reduces emotional expressivity, and static emotion loss can produce artifacts such as eye closure.Source: §6.2 · pp. 8–9

6.3. Emotion editing

• EMOTE lets animators edit emotions over the course of a sequence.Figure 9 demonstrates emotion editing within a sequence.Source: §6.3 · p. 9

6.4. Limitations

• The 25-fps pseudo-ground-truth can limit performance on very fast speech, and EMOTE does not model eye blinks because they are weakly correlated with speech and emotion.The authors note that higher-rate 3D scans or data augmentation could improve results on fast speech.Source: §6.4 · p. 9
• EMOTE omits paralinguistic motion and mouth interiors, and its emotion set is limited to eight basic emotions; automatic emotion control also requires a user-provided label.The paper notes that broader emotion coverage would require richer large-scale datasets and a more granular emotion model.Source: §6.4 · p. 9

• EMOTE’s lip-sync is better than that of state-of-the-art methods trained on the same data, despite using pseudo-ground-truth rather than high-quality 3D ground truth.The conclusion attributes this advantage to perceptual video-emotion and lip-reading losses.Source: §7 · p. 9
• EMOTE provides explicit control over the type and intensity of emotional expression in 3D talking-head avatars.Source: §7 · p. 9
• A content-emotion exchange mechanism separates speech-driven lip articulation from expression controlled by the specified emotion.Source: §7 · p. 9
• FLINT is a temporal transformer-based VAE motion prior, and EMOTE uses a computationally efficient feedforward architecture.Its decoder maps speech features and emotion condition into the prior’s latent space.Source: §7 · p. 9

EMOTE w/o FLINT

• Without FLINT, the visual ablation shows uncanny artifacts, while removing lip-reading supervision can produce inferior lip-sync, including incomplete bilabial mouth closure.Source: Figure 6 · p. 10
• Participants prefer EMOTE on both emotion and lip-sync tasks to all other ablated models.Source: Figure 7 · p. 11
• Participants prefer EMOTE over MeshTalk for lip-sync, while VOCA, FaceFormer, and CodeTalker trained on VOCASET are preferred over EMOTE.FaceFormer retrained on MEAD has considerably lower lip-sync preference than the VOCASET-trained comparison.Source: Figure 8 · p. 11

A.1. Motion Prior: FLINT

• FLINT uses modified ALiBi in both transformers, three-layer convolutional blocks, and a bottleneck dimension of 128.Training sets λ_rec to 1000000 and λ_KL to 0.001.Source: §A.1 · p. 16

A.2. Face Reconstruction Network

• The face-reconstruction network combines DECA with MICA for face shape, EMOCA for expression, and SPECTRE’s lip-reading loss for lip articulation.Source: §A.2 · p. 16
• The reconstruction network passes images through MICA, DECA, and EMOCA encoders to predict shape, other face parameters, and expression, then reconstructs geometry and renders an image for supervision.Source: §A.2 · p. 16
• Training proceeds by finetuning DECA with MICA shape predictions, training EMOCA with lip-reading loss, then finetuning EMOCA on MEAD for pseudo-ground-truth.The MEAD stage uses the previous stage’s losses.Source: §A.2 · pp. 16–17

A.3. Limitations of Static Emotion Recognition

• Static emotion recognition can produce different classifications across frames even when a person maintains one emotion throughout a video.The paper attributes the instability to missing temporal context and possible misinterpretation of speech-induced expressions.Source: §A.3 · p. 17

A.4. Video Emotion Recognition

• A lightweight single-layer transformer processes static emotion features into sequence-level features for emotion and intensity classification.It is trained on MEAD using cross-entropy losses and the dataset’s emotion and intensity labels.Source: §A.4 · p. 17

A.5. Dynamic vs Static Emotion Loss

• Video emotion classification is significantly superior to the static emotion classifier.Source: §A.5 · p. 17

B. Datasets

• The video emotion architecture encodes framewise features from a frozen pretrained recognizer into one video feature for emotion and intensity classification.It uses a single-layer transformer encoder with modified ALiBi that considers future frames.Source: Figure 12 · p. 18
• Dynamic emotion recognition reaches 90.8% accuracy on MEAD validation, compared with 57.9% for static emotion recognition.The static result uses the most occurring classification in each video.Source: Figure 13 · p. 18
• MEAD was selected for its scale, emotion intensities, and high-quality constrained video that is relatively easy to reconstruct.The authors note that lab capture avoids some in-the-wild reconstruction challenges, including occlusions and difficult illumination.Source: B. Datasets · pp. 17–18

C. Perceptual studies

• The perceptual-study materials show pairwise video comparisons for lip synchronization, articulation, and emotion quality.The emotion assessment uses muted videos; the articulation assessment is audible and reuses the same pair.Source: C. Perceptual studies · pp. 18–19, Figure 14 · p. 19

D. Additional Results

• Figures 17–18 show emotional animations with a different speaking style in each figure and the same input audio paired with different emotion labels.The captions identify the same-audio, different-emotion setup for both figures.Source: D. Additional Results · p. 19, Figure 17 · p. 22, Figure 18 · p. 23
• Emotion quality is judged from muted video pairs against the specified emotion, while articulation is judged from audible pairs with the audio considered.Both videos must be watched in full before the participant can answer.Source: Figure 15 · p. 20, Figure 16 · p. 21