PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation

Yuan Gan, Zongxin Yang, Xihang Yue, Lingyun Sun, Yi Yang

arXiv:2309.04946v2cs.SDcs.CVcs.GReess.AS

TL;DR

Existing approaches make emotion control in audio-driven talking-head generation inflexible and costly to transfer. EAT adapts a pretrained emotion-agnostic model with lightweight guidance and outperforms previous methods in one-shot emotion manipulation and video quality.

  • Problem

    Existing approaches make emotional talking-head generation inflexible and require costly transfer from guidance videos to predictions.

    Source: §1 · pp. 1–2

  • Method

    EAT uses a two-stage approach that adapts a pretrained emotion-agnostic talking-head model with Deep Emotional Prompts, EDN, and EAM for emotional expression control.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    EAT outperforms previous methods in both emotion manipulation and video quality for one-shot talking-head generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Lightweight guidance makes pretrained talking-head models more adaptable to practical scenarios such as text-guided zero-shot expression editing.

    Source: §1 · p. 2

    Background and head-pose diversity in emotional training data can affect generalizability, and zero-shot generation requires effective guidance texts.

    Source: D. Limitations and Future Work · p. 16

Abstract

from arXiv · show

Audio-driven talking-head synthesis is a popular research topic for virtual human-related applications. However, the inflexibility and inefficiency of existing methods, which necessitate expensive end-to-end training to transfer emotions from guidance videos to talking-head predictions, are significant limitations. In this work, we propose the Emotional Adaptation for Audio-driven Talking-head (EAT) method, which transforms emotion-agnostic talking-head models into emotion-controllable ones in a cost-effective and efficient manner through parameter-efficient adaptations. Our approach utilizes a pretrained emotion-agnostic talking-head transformer and introduces three lightweight adaptations (the Deep Emotional Prompts, Emotional Deformation Network, and Emotional Adaptation Module) from different perspectives to enable precise and realistic emotion controls. Our experiments demonstrate that our approach achieves state-of-the-art performance on widely-used benchmarks, including LRW and MEAD. Additionally, our parameter-efficient adaptations exhibit remarkable generalization ability, even in scenarios where emotional training videos are scarce or nonexistent. Project website: https://yuangan.github.io/eat/

1. Introduction

Emotional talking-head generation can improve realism, but prior methods require costly full-network training and depend on suitable emotional driving videos. EAT adapts pretrained emotion-agnostic models with lightweight modules and flexible guidance, reporting improved one-shot results and rapid transfer.

1. Introduction

  • Prior emotional methods use driven emotional videos and require training or finetuning the entire network, while video selection and mouth-shape ambiguity remain practical concerns.GC-AVT replaces mouth regions with neighboring frames; EAMM ignores them through data augmentation.Source: §1 · pp. 1–2
  • EAT uses a two-stage framework: enhanced 3D latent keypoints and A2ET first support emotion-agnostic generation, then learnable guidance and adaptation steer emotional expression.The adaptations are Deep Emotional Prompts, EDN for facial latent deformation, and EAM for visual quality.Source: §1 · p. 2
  • EAT outperforms previous methods in one-shot talking-head generation for both emotion manipulation and video quality.The introduction also reports state-of-the-art performance in 2 hours using 25% training data.Source: §1 · p. 2
  • Deep Emotional Prompts, EDN, and EAM enable efficient transfer from talking-head generation without emotional expression to generation with emotional expression.Source: §1 · p. 2
  • EAT introduces flexible guidance for talking-head adaptation, including text-guided image-text models for zero-shot expression editing.The authors describe this as the first such flexible guidance approach to their knowledge.Source: §1 · p. 2

2. Related Work

Prior audio-driven talking-head work emphasizes synchronization and pose or identity control, while recent emotional methods use emotional or pose-guiding videos. EAT instead pursues emotion control through efficient adaptation of a pretrained talking-head model.

Audio-driven Talking Head Generation

  • Prior audio-driven approaches use facial landmarks, disentangled speech and speaker representations, or pose control, and AVCT applies an audio-visual correlation transformer.Source: §2 · p. 2

Emotion-aware Talking Head Generation

  • Audio-driven methods have advanced emotion-agnostic generation, while emotional talking-head work includes pose-fixed generation, granular control, and augmented emotional source videos.The cited examples include Sinha, GC-AVT, and EAMM.Source: §2 · p. 3

Efficient Finetuning

  • Efficient finetuning includes adapters, bias tuning, side tuning, and visual prompt methods, while EAT combines deep emotional prompts with EDN and EAM for emotion-related transfer.The related work names residual adapter, bias tuning, side tuning, CoOp, and VPT.Source: §2 · p. 3

3. Method

EAT reuses an emotion-agnostic pretrained talking-head model and adapts it with lightweight modules for emotional generation. Its method combines enhanced 3D expression representations, audio-to-expression prediction, emotional guidance, and supervised objectives.

3. Method

  • EAT adapts a pretrained emotion-agnostic model through deep emotional prompts, an Emotional Deformation Network (EDN), and an Emotional Adaptation Module (EAM).The method separates emotion-agnostic pretraining from parameter-efficient emotional adaptation.Source: §3 · p. 3, §3.2 · p. 5

3.1. Emotion-agnostic Pretraining

  • The emotion-agnostic pretraining reuses learned knowledge and enhances 3D latent keypoints to represent subtle facial expressions.A2ET maps audio to synchronized 3D latent keypoints using large-scale talking-head datasets.Source: §3.1 · pp. 3–4

3.1.1 Enhanced Latent Representation.

  • Enhanced latent representations remove the deformation prior loss, use paired neutral and emotional MEAD videos, and restrict losses to faces while augmenting data with AffectNet.The changes target subtler expression changes and more expressive faces; the authors state that they enhance the representation ability of the learned keypoints.Source: §3.1.1 · p. 4

3.1.2 Audio-to-Expression Transformer.

  • A2ET predicts expression deformation rather than full 3D keypoint sequences, using audio semantics, acoustic features, and pose context.PCA reduces the deformation dimensionality because optimizing 3D expression motions directly leads to convergence issues.Source: §3.1.2 · pp. 4–5

3.2. Efficient Emotional Adaptation

  • Emotional guidance is supplied as transformer prompts, with deep prompts added at every layer after the first; the authors report better expression transfer than with shallow prompts, but note a synchronization trade-off.The EDN predicts emotion-related deformation ΔE_i from emotional guidance and a source latent token, then adds it to A2ET's speech-related deformation E_i.Source: §3.2 · p. 5
  • EAM uses guidance embedding e to produce channel weights γ and bias β, yielding emotion-conditioned features through channel-wise multiplication and addition.The module is described as lightweight and plug-and-play; γ and β are confined to [-1, 1] with tanh.Source: §3.2 · p. 6
  • For zero-shot text-guided expression editing, CLIP loss aligns generated-face and text embeddings while finetuning only the guidance mapping network and EAM.The paper states that this editing approach does not require emotional training data and is intended to generalize to rare expressions.Source: §3.2 · p. 6

3.3. Training Objectives

  • Training combines latent, synchronization, and reconstruction losses; zero-shot editing replaces reconstruction loss with CLIP loss because no ground-truth video is available.EDN training uses only the 3D-keypoint term of the latent loss because emotional deformation is not included in the predicted PCA expression representation.Source: §3.3 · p. 6

4. Experiments

EAT is evaluated across emotion-agnostic and emotional talking-head generation, including user ratings, adaptation efficiency, zero-shot editing, and component ablations. The reported results favor EAT on most measures while revealing trade-offs among synchronization, fidelity, and emotion intensity.

4.1. Experimental Setup

  • The evaluation uses identity-disjoint MEAD training and test sets, VoxCeleb2 videos for head-pose changes, and LRW and MEAD test sets for baseline comparisons.Metrics cover image quality, audio-visual synchronization, mouth and face landmark distances, and emotional accuracy.Source: §4.1 · p. 7

4.2. Talking-head Generation

  • On LRW's 25,000 neutral test videos, EAT outperforms other methods in visual quality and improves the pretrained talking-head model.The authors note that Wav2Lip and PC-AVS score above ground truth on synchronization, and Wav2Lip generates only mouth regions.Source: §4.2 · pp. 7–8
  • EAT achieves the best performance on most MEAD metrics, with better video quality and emotion accuracy than competing approaches.The authors also report that EAT produces authentic, comprehensive expressions in the one-shot setting.Source: §4.2 · p. 8

4.3. User Study

  • In a 14-participant study, EAT receives the best scores for lip-sync, video quality, and emotion accuracy on generated in-the-wild videos.The authors caution that SyncNet scores in Table 1 are inaccurate for emotional talking heads because SyncNet was trained only on neutral videos.Source: §4.3 · p. 8

4.4. Tuning Efficiency

  • EAT surpasses state-of-the-art results within one hour using full or half the data, and reaches comparable performance in two hours with one-quarter.The adaptations add 6.85% parameters: deep emotional prompts account for 5.30%, EDN for 1.17%, and EAM for 0.38%.Source: Figure 5 · p. 7, Figure 6 · p. 7, §4.4 · p. 9

4.5. Zero-shot Expression Editing

  • With CLIP guidance, EAT edits expressions from neutral videos using text descriptions, and learned latent codes can transfer to another video.Editing performance depends on the design of the text descriptions.Source: §4.5 · p. 9

4.6. Ablation Study

  • Ablations associate deep prompts with stronger expressions but weaker synchronization, while adding EDM and EAM improves image quality and ground-truth fidelity at the expense of emotion intensity and accuracy.The study also reports that sync loss contributes to synchronization, while the other tested losses contribute to expression fidelity.Source: §4.6 · p. 9

5. Conclusion

The conclusion presents EAT as a two-stage paradigm for adapting emotion-agnostic talking-head generation to emotional expression control. It identifies variation in emotional training data backgrounds and head poses as a constraint on generalization.

5. Conclusion

  • EAT first enhances 3D latent representations and uses A2ET for emotion-agnostic generation, then adds learnable guidance through deep emotional prompts, EDN, and EAM.The authors describe the resulting paradigm as parameter-efficient and effective, and note that emotional-data variation in backgrounds and head poses affects generalizability.Source: §5 · p. 9

Supplementary Material

The supplementary material presents additional emotional expressions generated by EAT from emotional guidance.

Supplementary Material

  • Additional examples show EAT generating facial expressions from emotional guidance using source images from CREMA-D and MakeItTalk.The caption describes the expressions as realistic and diverse.Source: Figure 9 · p. 13

A. The Networks Details

The network details describe how audio features are mapped to expression deformations and how learned emotional guidance steers generation. They specify the roles of the emotion mapper, EDN, and EAM.

A. The Networks Details

  • A2ET predicts expression deformation from audio features, and PCA projects the predicted deformation into 32 dimensions before reconstruction with the principal components and mean.The passage says direct optimization of 3D expression motions causes convergence problems; the deformation modifies neutral 3D keypoints.Source: A. The Networks Details · p. 13
  • The emotion mapper takes a 16-dimensional latent code and outputs seven emotional tokens, assigning one to EAM and six as prompts to A2ET layers.EDN combines the source representation with all seven tokens to produce emotional deformation.Source: A. The Networks Details · p. 13
  • EDN pools emotion-related token features and uses an MLP to produce deformation, while EAM derives channel-wise scale and bias values to modify input features.The passage reports that the adaptation modules use about 7% of the whole network's parameters; EDN and EAM together use less than 2%.Source: A. The Networks Details · p. 13

B. Training and Testing Details

EAT is trained in two stages using emotional video and image data, then evaluated on LRW and MEAD with dataset-specific test protocols. The section also specifies face preprocessing and evaluation metrics.

Training Details

  • A2ET is first trained with latent loss for initialization, then with the full loss; adaptation modules are fine-tuned on MEAD.The first stage uses MEAD and 8k Voxceleb2 clips for enhanced latent keypoints, plus roughly 21k AffectNet images for emotional expression generation; neutral faces are generated with Ganimation.Source: B. Training and Testing Details · pp. 13–14

Testing Details and Protocol

  • LRW testing starts from the first frame and uses relative latent-keypoint offsets, whereas MEAD testing uses predicted latent keypoints.The protocols account for variation in MEAD facial expressions unrelated to the neutral source image.Source: B. Training and Testing Details · p. 14

Evaluation

  • Evaluation uses aligned, cropped faces for PSNR, SSIM, FID, M-LMD, and F-LMD, with synchronization confidence preprocessing following PC-AVS.Table 8 concerns EDN weight initialization and states that initializing it with the A2ET encoder promotes EAT performance.Source: B. Training and Testing Details · p. 14, Table 8 · p. 14

C. Additional Experimental Results

Additional experiments examine baseline comparisons, emotion-class performance, interpolation, component effects, and profile-view generation. They report trade-offs between emotion intensity, fidelity, and image quality.

Additional baseline results

Various emotional expressions

  • EAT performs significantly better on Happy, Sad, Disgusted, and Contempt than other methods, while all methods perform poorly on Fear.EAMM performs well on Neutral, Angry, and Surprised; the text attributes the difficulty with Fear to its similarity to Surprise.Source: C. Additional Experimental Results · p. 14

Emotion interpolation

  • Emotion-guidance interpolation between Angry and Surprised, and between Contempt and Sad, is used to assess continuity of the learned emotion-mapper latent space.The figure’s top and bottom rows show those respective pairs; its neutral faces come from MEAD [54].Source: C. Additional Experimental Results · p. 15

Additional ablation and component visualization

  • Using A2ET encoder weight initialization for EDN leads to quicker convergence and improved video quality and audio-visual synchronization.The fear-expression component visualization reports accuracy rates of 38.46%, 30.77%, and 15.38% for (A), (B), and (C), respectively.Source: C. Additional Experimental Results · p. 15

Visual analysis on each component

  • Deep emotional prompts intensify expressions but can deviate from ground truth; adding EDM and EAM improves fidelity and PSNR/SSIM while reducing emotion intensity and accuracy.The component visualization presents the comparison across the EAT components.Source: C. Additional Experimental Results · p. 15

Visualization on profile faces

  • Profile-face visualizations test generated frames from left 30-degree and right 60-degree MEAD faces under Surprised and Happy emotions.The passage describes the tested views and emotions but does not state a comparative outcome.Source: C. Additional Experimental Results · p. 15

D. Limitations and Future Work.

EAT has limitations tied to emotional training-data diversity and the guidance needed for zero-shot generation. The authors also warn of potential misuse and recommend labeling generated videos as fake.

Limitations and Future Work

  • Background and head-pose diversity in emotional training data can affect generalizability; in-the-wild examples show forehead wrinkles that are not obvious.The authors suggest more naturalistic, non-acted emotional data and representations with a head prior such as FLAME [62].Source: D. Limitations and Future Work · p. 16, Figure 15 · p. 16
  • Zero-shot generation requires effective guidance texts, a constraint the authors associate with the limited ability of image-text-trained models.The passage presents this as a possible explanation, not a definitive cause.Source: D. Limitations and Future Work · p. 16

Ethical considerations

  • Because emotional talking-head generation could be abused, the authors recommend labeling generated videos as “fake”.The stated intended uses are virtual human research and entertainment.Source: E. Ethical Considerations · p. 16
Loading 2309.04946v2…