PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video

Chanhyuk Choi, Taesoo Kim, Donggyu Lee, Siyeol Jung, Taehwan Kim

arXiv:2604.07786v2cs.CVcs.LG

TL;DR

Existing emotion-editing methods have limited expressive flexibility because labels are discrete, audio entangles emotion with linguistic content, and image references are difficult to obtain for extended emotions. C-MET maps emotion semantic vectors from speech to facial expressions using separate audio and visual embedding spaces. Experiments on MEAD and CREMA-D report stronger emotion accuracy while preserving visual attributes.

  • Problem

    Existing emotion-editing methods provide limited expressive flexibility: labels use discrete categories, audio entangles emotion with linguistic content, and image methods require difficult-to-obtain reference data for extended emotions.

  • Method

    C-MET maps emotion semantic vectors from speech to facial expressions using a pretrained audio encoder and a disentangled facial-expression encoder.

  • Results

    Experiments on MEAD and CREMA-D show that C-MET significantly outperforms state-of-the-art emotion-editing methods in emotion accuracy while preserving visual attributes.

  • Takeaways & Limitations

    C-MET can synthesize unseen emotional expressions from expressive speech and integrates as a plug-and-play module into disentanglement-based generators.

  • Takeaways & Limitations

    The method does not yet handle multi-view identity images, limiting editing across diverse viewpoints; current emotional talking-face datasets also support only English.

Abstract

from arXiv · show

Talking face generation has gained significant attention as a core application of generative models. To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role. However, existing approaches often limit expressive flexibility and struggle to generate extended emotions. Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions. Audio-based methods can leverage emotionally rich speech signals - and even benefit from expressive text-to-speech (TTS) synthesis - but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches. Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm). To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces. C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities. Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos - even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/

1. Introduction

Emotion editing aims to make talking-face videos more expressive, but existing label-, audio-, and image-based controls limit flexibility or struggle with complex emotions. C-MET transfers emotion semantics from speech to facial-expression spaces, enabling expressive editing including unseen extended emotions.

  • Motivation: Complex and subtle emotional expressions are important for believable virtual agents and more immersive, empathetic communication.The motivation includes applications such as education, therapy, and virtual assistants.
  • Limitations of Existing Methods: Label-based methods restrict expressiveness through categorical emotion sets, while audio-based methods struggle to disentangle emotion from linguistic content.The introduction describes these as two common limitations among existing emotion-editing approaches.
  • Open Problem: Cross-modal mapping remains difficult because audio and visual emotion representations occupy different domains.The paper identifies bridging this domain gap as a key open problem.
  • C-MET: C-MET explicitly models relationships between emotion semantic vectors in audio and visual feature spaces.Each semantic vector is obtained by subtracting embeddings of two different emotional expressions.
  • C-MET: C-MET extracts audio emotion vectors with a pretrained speech model and predicts corresponding visual vectors using a disentangled facial-expression encoder.The resulting mapping transfers emotion semantics across modalities and extends voice emotion control to visual generation.
  • Evaluation: Experiments on MEAD and CREMA-D show improved emotion editing and generation of extended emotions, including emotions unseen during training.The paper reports quantitative and qualitative validation, with the method designed as a plug-and-play component for disentanglement-based generators.

2. Related Work

Related work develops audio-driven talking-face generation around realism, identity preservation, and lip synchronization, then adds emotion control through labels, text, audio, or images. C-MET addresses the limitations of these controls by learning facial emotion semantic vectors from audio emotion vectors.

  • Audio-Driven Generation: Audio-driven talking-face methods primarily target realism, precise audio–lip synchronization, and preservation of reference-image identity.Early work such as Wav2Lip blends synthesized lip movements into existing frames, while later approaches use two-stage pipelines.
  • Emotion Control: Emotional talking-face generation adds target-emotion synthesis after producing a lip-synchronized talking-face video.Existing approaches condition generation on emotion labels, emotional audio, or driving signals.
  • Label and Text Controls: Label-based methods use discrete categories, making extended or subtle emotional states difficult to represent.EAT is described as a lightweight transformer-based adaptation network controlled by discrete emotion labels.
  • Cross-Modal Emotion Transfer: C-MET uses a large-scale audio encoder, a disentangled facial-expression encoder, and an intermediate network to generate visual emotion semantic vectors from audio vectors.The design is intended to reduce the modality gap for cross-modal emotion regression.

3. Methodology

C-MET transfers emotion semantics from speech to facial-expression representations by aligning multimodal tokens and predicting visual emotion vectors for video reconstruction.

  • Cross-Modal Emotion Transfer Learning: C-MET extracts modality-specific embeddings, aligns them in a shared latent space, and models cross-modal correspondence with a Transformer encoder.The method uses pretrained audio and visual encoders, learnable tokenizers, and Transformer layers to connect speech and facial-expression representations.
  • Contrastive Learning on Multimodal Tokens: Contrastive learning aligns visual and audio tokens using bidirectional visual-to-audio and audio-to-visual objectives.Visual tokens are temporally pooled from facial-expression embeddings, while audio tokens are produced from pretrained audio embeddings; cosine similarity, batch negatives, and temperature guide the loss.
  • Evaluation Protocol: Table 1 evaluates emotion editing from neutral videos across MEAD and CREMA-D, prioritizing emotion classification accuracy while preserving perceptual attributes.The comparison varies emotion-source modality across image-, label-, and audio-based methods.
  • Cross-Modal Emotion Transfer Learning: Emotion semantic vectors represent differences between two emotional states in the audio and visual representation spaces.Input and target emotions are selected separately, and their embedding differences define the transfer target.
  • Cross-Modal Emotion Transfer Learning: The Transformer predicts target visual semantic vectors from reference visual, target speech, and input visual tokens, then projects the final visual tokens.The input tokens are concatenated and processed jointly to model intra-modal and inter-modal dependencies.
  • Cross-Modal Emotion Transfer Learning: Training minimizes reconstruction error while adding contrastive and direction losses to the objective.The direction term encourages forward and reverse semantic vectors to be opposite, with λcnt and λdir weighting the auxiliary losses.

4. Experiment

Experiments evaluate C-MET across datasets, emotion settings, baselines, ablations, qualitative examples, user preferences, and continuous editing. Results show strong emotion accuracy, expressive motion, generalization to extended emotions, and integration with disentanglement networks.

  • Experimental Settings: The model is trained with emotion2vec+large and EDTalk encoders, using averaged neutral–emotional speech and video semantic vectors to stabilize learning.The implementation uses 5-frame reference visual semantics and evaluates MEAD and CREMA-D, with HDTF and generated portraits used qualitatively.
  • Experimental Settings: Evaluation measures video quality with FID, temporal coherence with FVD, synchronization with Syncconf, and emotional accuracy with Emotion-FAN.Experiments include basic emotions and extended emotions, for which user studies are required because ground-truth videos are unavailable.
  • Quantitative Results: C-MET achieves the highest Accemo across all benchmarks, while EDTalk is slightly better on FID, FVD, and Syncconf.The comparison uses identical neutral videos and varies only the emotion-source modality across EAMM, EDTalk, EAT, FLOAT, and C-MET.
  • Quantitative Results: Stronger and more diverse expressions can increase motion and pixel deviations, creating a trade-off between emotional accuracy and reconstruction-based visual metrics.Despite this trade-off, C-MET produces more dynamic and emotionally expressive facial motions while remaining marginally comparable in visual quality.
  • Qualitative Results: C-MET generates dynamic frowning and eyebrow contraction for anger and asymmetric one-sided smiles for sarcasm, unlike the other baselines.The method generalizes across emotions and identities by learning emotion semantic vectors disentangled from audio content.
  • User Study and Further Analysis: User studies show C-MET consistently outperforms all baselines for emotional expression, visual quality, and lip synchronization across basic and extended emotions.Continuous editing applies speech-derived semantic vectors over five-frame intervals to produce smooth facial-expression transitions and fine-grained intensity.

5. Conclusion

The conclusion presents C-MET as a speech-to-expression method that transfers emotion semantic vectors across separate audio and visual spaces. Experiments on MEAD and CREMA-D show improved emotion accuracy while preserving visual attributes and supporting unseen expressions.

  • Conclusion: C-MET maps speech emotion semantic vectors to facial expressions for emotion editing in talking face videos.It learns the vectors in separate audio and visual embedding spaces.
  • Conclusion: C-MET synthesizes unseen emotional expressions from expressive speech despite training only on existing audio–visual datasets.The method also operates as a plug-and-play module in disentanglement-based generators.
  • Conclusion: Experiments on MEAD and CREMA-D show that C-MET outperforms state-of-the-art emotion-editing methods in emotion accuracy while preserving visual attributes.The conclusion also reports reduced inference latency through integration with disentanglement-based generators.

Supplementary Material

The supplementary material adds expressive-speech generation details, visualizations, experimental analyses, human-evaluation templates, and limitations.

  • Supplementary Material: The supplement describes expressive speech generation, additional visualizations, further experiments, human-evaluation templates, and limitations.Its additional experiments cover speech-shot impact, emotion consistency, audio-encoder ablations, and full ablation metrics.

A. Expressive Text-to-Speech

Expressive speeches for extended emotions are synthesized through a multi-stage Gemini 2.5 Flash TTS process. The process generates affective sentences, selects expressive voice identities, and injects the selected identity into TTS generation.

  • Expressive Text-to-Speech: Gemini 2.5 Flash TTS generates extended-emotion speech by first creating affective sentences and then selecting suitable expressive voice identities.The selected voice identity is passed into the TTS pipeline through the API's voice config parameter.

B. More Visualization Results

Additional qualitative examples visualize C-MET across various emotions, with interactive sample-video playback available on the project page.

  • Figures 9 and 10 provide additional visualizations of C-MET across various emotions.
  • Interactive playback of sample videos is available on the project page.

C. Additional Experimental Results

Additional experiments examine speech-shot aggregation, emotion consistency, encoder and loss choices, model integration, and emotion-wise accuracy across datasets.

  • Impact of speech-shot: Emotion accuracy steadily improves as more speech-shots are aggregated, surpassing all baselines with only two samples.Averaging speech-derived semantic vectors suppresses speaker-specific variation and stabilizes emotion representations.
  • Impact of speech-shot: Ten speech-shots are used in the main experiments, where performance saturates.
  • Emotion consistency evaluation: C-MET shows the most concentrated diagonal confusion-matrix patterns across all seven categories, indicating accurate and consistent emotion control.Compared models show more scattered or category-biased predictions.
  • Ablation studies: emotion2vec+large provides higher emotion accuracy and lower inference latency than Qwen2.5-Omni in the audio-encoder ablation.Its pretraining on large-scale emotion-specific speech corpora aligns representations with affective cues.
  • Ablation studies: Adding contrastive loss improves visual quality and temporal consistency, while direction loss yields the highest emotion accuracy.The full loss setup is adopted because emotion accuracy is the task’s most important metric.
  • Model integration: Integrating C-MET into PD-FGC and EDTalk consistently improves inference speed and emotion accuracy.EDTalk shows slight degradations in FID, FVD, and Syncconf, but these remain comparable and have negligible human-perception impact.
  • Emotion-wise accuracy: C-MET achieves the highest average accuracy on MEAD and CREMA-D, including 55.91% versus EDTalk’s 41.99% on MEAD and 43.47% on CREMA-D.It also reports 78.57% for Happy and 88.64% for Sad, supporting more balanced performance across positive and negative emotions.

D. Human Evaluation Template

The human-evaluation setup compares edited videos using speech-conditioned emotional expression, visual quality and realism, and lip synchronization criteria.

  • Evaluation setup: Each evaluation presents an emotional speech sample alongside a video concatenating neutral, edited A, and edited B clips.
  • Evaluation setup: Participants choose Edited Video A, Edited Video B, or Tie after listening to the audio.
  • Evaluation criteria: Ten Amazon Mechanical Turk participants evaluate emotional expression, visual quality and realism, and lip synchronization.
  • Sampling and template: Fifty randomly sampled test-set outputs are used to support diversity and fairness in the evaluation.The human-evaluation template is illustrated in Figure 12.

E. Limitations

The paper identifies boundaries involving paired speech data, multi-view identity images, and the English-only scope of current emotional talking-face datasets.

  • Data requirements: C-MET requires at least three pairs of neutral and emotional speech samples for stable performance.Expressive TTS can synthesize paired data, and existing neutral or basic-emotion recordings can be reused.
  • Multi-view limitation: The approach does not yet handle multi-view identity images, limiting editing across diverse viewpoints.The authors propose a multi-view facial-expression encoder as a future extension.
  • Language scope: Current emotional talking-face datasets support only English, limiting the present language scope.Future work plans to extend semantic-vector modeling to multilingual emotional speech data.
Loading 2604.07786v2…