Source-linked AI summary
EmoFI-TTS: Face-Conditioned Expressive Voice Synthesis with Intensity Modulation
Yejin Jeon, Luel Hagos Beyene, Marie Maltais, Hyounghun Kim, David Ifeoluwa Adelani
TL;DR
Emotive TTS must control emotional intensity even when explicit labels are unavailable, while maintaining naturalness and intelligibility. EmoFI-TTS infers emotion and intensity from a face and speaker characteristics from reference audio, then refines speech using intensity conditioning. On speaker-disjoint MEAD evaluation, it improves intensity accuracy over evaluated baselines while retaining intelligibility and naturalness.
Problem
Expressive TTS must control emotion across content and speaker identity without sacrificing naturalness or intelligibility, while common conditioning approaches rely on explicit labels or emotional reference speech.
Source: §1 · p. 1Method
Results
64.4 vs. 39.1 unweighted intensity accuracy; EmoFI-TTS also achieved the lowest WER (6.9%) and highest UTMOS score (3.14) among the baseline systems.
Source: §4 · p. 4Takeaways & Limitations
Speaker-disjoint MEAD evaluations support facial affect as a conditioning signal for fine-grained emotional speech synthesis without emotion annotations at inference.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Controlling emotional intensity remains challenging in expressive text-to-speech synthesis, particularly when emotion must be inferred without explicit labels. We propose EmoFI-TTS, a face-conditioned framework that derives emotion category and intensity from a single facial image while extracting speaker characteristics from reference audio. Modality-specific representations are fused to condition a pretrained speech synthesizer, with cross-speaker training encouraging separation of affect and vocal identity. An intensity-conditioned refinement module enhances the generated mel-spectrogram, while interpolation of learned intensity embeddings enables continuous control without additional training. On speaker-disjoint MEAD evaluation, EmoFI-TTS achieves on average 64.4% and 75.2% objective and subjective emotion intensity accuracies, while retaining low word error rate and high naturalness.
1. INTRODUCTION
Expressive TTS still faces the challenge of controlling emotion across linguistic content and speaker identity without sacrificing naturalness or intelligibility. EmoFI-TTS uses facial expression to infer emotion and intensity, while reference audio supplies speaker characteristics, addressing the limited affective and intensity modeling of prior face-to-speech work.
1. INTRODUCTION
- Precise emotive control that generalizes across content and speaker identity, without sacrificing naturalness or intelligibility, remains an open TTS challenge.Existing systems commonly rely on manually specified labels, learned style embeddings, or emotional reference speech.Source: §1 · p. 1
- Text-based emotion inference is ambiguous, and a single emotion category can span intensities that call for markedly different prosody, energy, and spectral emphasis.Identical sentences can express conflicting emotional intent, while prior approaches do not capture the continuum from mild to exuberant emotion.Source: §1 · p. 1
- Facial expression conveys intended emotion and intensity, but prior face-to-speech work has mainly treated faces as proxies for speaker attributes and has not modeled fine-grained intensity.A mild and exuberant expression of the same category may warrant different acoustic realizations.Source: §1 · p. 1
- EmoFI-TTS derives affect from a facial image and timbre from reference audio, then fuses these representations to condition a speech synthesizer.An intensity-conditioned module refines the mel-spectrogram while preserving speaker consistency.Source: §1 · pp. 1–2
- The framework targets expressive generation and emotion transfer to unseen speakers without requiring emotional recordings from the target speaker, while evaluations report preserved intelligibility and naturalness.Its stated contribution is visually conditioned, disentangled control of facial emotion and intensity and reference-speaker identity.Source: §1 · p. 2
2. METHOD
EmoFI-TTS conditions pretrained FastSpeech2 on facial affect and reference-audio timbre, then refines its mel-spectrogram using facially inferred intensity. Cross-speaker training and interpolated intensity embeddings support separation of vocal identity from affect and continuous intensity control.
2. METHOD
- EmoFI-TTS maps a face to affect and reference audio to speaker timbre, fuses them into a style vector, and conditions pretrained FastSpeech2.The fused style vector modulates the phoneme encoder and mel decoder.Source: §2 · p. 2
- Training combines the backbone’s mel, pitch, energy, duration, and adversarial losses with emotion and intensity classification losses.The emotion and intensity losses use classification heads reading only the face embedding.Source: §2 · p. 2
- Half of training swaps in a different speaker’s face matched on emotion and intensity, discouraging face-based shortcuts for target voice identity.The reference audio continues to identify the target speaker.Source: §2 · p. 2
- A post-generation refiner predicts level-conditioned mel residuals, with level 1 bypassed because refinement over-energized calm speech.The refinement module uses learned intensity embeddings, and the lowest level bypasses it entirely.Source: §2 · p. 3
- Refiner training combines multiscale LSGAN, spectral-variance matching, an L1 content anchor, and speech-emotion feature matching.The feature-matching loss compares frozen wav2vec 2.0 representations after the same frozen vocoder.Source: §2 · p. 3
- Continuous control interpolates adjacent learned intensity embeddings; at intensity 1, the refinement gate is bypassed.The embeddings were trained on three discrete levels.Source: §2 · p. 3
3. EXPERIMENTAL SETTINGS
The experiments use speaker-disjoint MEAD data and evaluate intelligibility, naturalness, emotion and intensity accuracy, and human judgments. Training separates the face-conditioning and refinement stages, with adapted systems from adjacent tasks as baselines.
Dataset
- MEAD yields 18,270 utterances across three intensity levels after excluding disgust, contempt, and two speakers, with train/val/test split speaker-disjointly at approximately 8:1:1.Because MEAD lacks ground-truth transcripts, the authors manually identified 141 canonical sentences, transcribed each utterance with Whisper-large, then corrected transcriptions by matching them against that set.Source: §3 · p. 3
Training and evaluation
- The style-conditioning branches are trained for 30k steps and frozen before the refinement module is trained for 15k steps.Evaluation covers WER, UTMOS, emotion and intensity classification accuracy, and human judgments of TTS quality, emotion synthesis, and speaker transfer.Source: §3 · p. 3
Baselines
- EmoSpeech uses oracle emotion labels at inference, while FaceTTS is repurposed to predict emotion intensity from face input.Neither baseline was designed for face-conditioned synthesis at varying intensities.Source: §3 · pp. 3–4
Ablation tables
- The experimental tables cover intensity-refinement and fusion ablations; the fusion table marks the adopted methodology in grey and reports speaker transfer in percentages.Source: Table 2 · p. 4, Table 3 · p. 4
4. RESULTS
On speaker-disjoint MEAD evaluation, EmoFI-TTS outperforms the evaluated baselines on intensity accuracy while also achieving the lowest WER and highest UTMOS. Human evaluations and ablations report naturalness, emotion, speaker-transfer, and intensity-control findings.
Baseline comparison
- 64.4 vs. 39.1 unweighted intensity accuracy; EmoFI-TTS also has the lowest WER (6.9%) and highest UTMOS (3.14) among the baselines.Unlike EmoSpeech, it infers emotion category and intensity from a facial image rather than receiving a ground-truth emotion label at synthesis time.Source: §4 · p. 4
Human evaluation and continuous control
- 3.25 naturalness MOS and 4.13 speaker-fidelity score; human evaluators also rated EmoFI-TTS highest for emotion-synthesis accuracy at each intensity level.Interpolating learned intensity-level embeddings enables continuous adjustment without additional training; increasing t raises level-3 recall and lowers level-1 recall.Source: §4 · p. 4
Oracle conditioning
- 64.4% intensity accuracy with face-predicted levels versus 67.2% with oracle conditioning, a gap of 2.8 percentage points.Source: §4 · p. 4
Fusion ablation
- Concatenation followed by linear projection achieves 56.7 intensity accuracy and 92.1% speaker transfer before refinement, outperforming additive and attention-based fusion on those objectives.Source: §4 · p. 4
Cross-speaker training
- Cross-speaker training raises level-3 intensity accuracy from 65.9% to 74.1% over paired training.This training pairs reference audio with a different speaker’s face matched on emotion category and intensity.Source: §4 · p. 4, Table 4 · p. 4
5. CONCLUSION
EmoFI-TTS infers emotion category and intensity from a face while using reference audio for speaker characteristics. On speaker-disjoint MEAD data, it improves intensity accuracy, intelligibility, and naturalness over evaluated baselines.
Conclusion
- EmoFI-TTS uses facial input for emotion category and intensity and reference audio for speaker characteristics; cross-speaker training encourages affect-identity separation, while intensity-conditioned refinement supports continuous adjustment without additional training.Speaker-disjoint MEAD evaluations show improved intensity accuracy, intelligibility, and naturalness over the evaluated baselines.Source: §5 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The paper reports that no ethical approval was required for this modeling study.
Ethical standards
- No ethical approval was required for this modeling study.Source: §7 · p. 5