Source-linked AI summary
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild
Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman
TL;DR
Large labelled speech-emotion datasets are costly, ambiguous, and difficult to collect at scale. The paper transfers facial-expression supervision to speech through cross-modal distillation on unlabelled videos, and the learned speech embeddings generalize across external benchmarks despite limitations in the source data and teacher setup.
Problem
Large human-labelled speech emotion datasets are difficult to obtain because annotation is costly, labels are ambiguous, and available data often have limited speaker diversity.
Method
A facial emotion teacher labels unlabelled VoxCeleb face-tracks, and a speech student learns emotion representations by transferring this supervision across modalities.
Results
The learned speech embeddings generalize across corpora with different dataset types and labelling schemes, outperform chance on external benchmarks, while remaining below the teacher’s performance.
Takeaways & Limitations
Unlabelled video can provide cross-modal supervision for learning speech emotion embeddings that achieve reasonable performance on standard benchmarks.
Takeaways & Limitations
The source dataset is heavily unbalanced, and facial emotion interpretation may be affected by missing contextual cues such as body language.
Abstract
from arXiv · showhide
Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.
1 INTRODUCTION
The paper addresses speech emotion representation learning under noisy, unconstrained conditions without labelled audio by transferring emotional knowledge from facial expressions in unlabelled audiovisual data.
- Motivation: Natural speech emotion datasets are costly to label, often acted, ambiguously annotated, and limited in speaker diversity.These constraints make broad generalization difficult and encourage speaker adaptation.
- Motivation: Fully supervised models trained on such datasets often achieve high accuracy only within the training corpus and tend to overfit.
- Motivation: The central question is whether emotional speech representations can be learned from unlabelled audio-visual speech by transferring knowledge from facial expressions.
- Approach: The method uses large unlabelled speech videos and the hypothesis that speech emotion correlates with the speaker’s facial expression.A visual model supplies discriminative emotional knowledge to an audio network through the video data.
- Motivation: The approach seeks annotation-free supervision, uncertainty-aware emotion labelling, and robustness to noisy correspondence between facial and vocal emotion.
- Contributions: The paper develops a facial teacher, labels VoxCeleb face-tracks, transfers supervision to speech, and evaluates the resulting model on two external datasets.The automatically labelled VoxCeleb data are released as EmoVoxCeleb.
2 RELATED WORK
Prior work includes teacher-student transfer within and across modalities, while speech emotion recognition remains constrained by acted, clean, and speaker-limited datasets.
- Teacher-student methods: Teacher-student methods transfer supervision through logits or probabilistic outputs, whereas cross-modal distillation transfers supervision between modalities.
- Teacher-student methods: Cross-modal research has used paired RGB-depth, RGB-flow, and synchronous audio-visual data in teacher-student or self-supervised architectures.
- Speech representations: The paper uses minimally processed magnitude spectrograms with CNN max pooling to identify emotionally salient regions in speech.
- Existing datasets: Most speech emotion datasets use acted or elicited studio speech, which may not generalize well to natural speech.
- Existing datasets: VoxCeleb instead contains real-world noise such as chatter, laughter, overlapping speech, room acoustics, and channel variation.
- Existing datasets: VoxCeleb has two orders of magnitude more speakers than existing public speech emotion datasets and requires no emotion labels for training.
- Evaluation assumptions: Emotion recognition is evaluated against human annotator judgement because the speaker’s true emotional state is typically unavailable.
3 CROSS MODAL TRANSFER
The paper transfers facial-expression supervision to speech through paired speaking face-tracks, training a voice model without labelled audio. It builds a facial teacher, aggregates frame predictions, and distills them into speech embeddings.
- Cross-modal transfer: Cross-modal distillation trains a speech student to reproduce supervision from a facial teacher using paired audio-visual inputs without labelled speech.The teacher operates on faces while the student operates on voices, with correspondence between synchronous facial expressions and speech utterances.
- Cross-modal transfer: The method assumes emotional content is redundant across a speaker’s concurrent facial expression and speech, requiring many speaking face-tracks with known correspondence.
- Cross-modal transfer: Distillation minimizes cross entropy between temperature-softened teacher and student predictions, with temperature T=2 used in all reported experiments.
- The teacher: Teacher training uses a Squeeze-and-Excitation ResNet-50-based face network, evaluated on the FERPlus facial-expression benchmark.The model is pretrained on VGG-Face2 and trained to match the distribution of annotator responses with cross entropy.
- The teacher: Face-track supervision is formed by max-pooling each emotion’s frame-level response across the track as a simple approximation to peak-frame selection.Average pooling was slightly inferior but not dramatically different, while prior peak-frame methods require identifying frames where expression is at its apex.
- The student: The speech student uses a batch-normalized VGG-M architecture and receives minimally processed audio represented directly as magnitude spectrograms.The segment duration is chosen to capture emotion-related prosody while avoiding entire sentences and excessive lexical content.
4 EMOVOXCELEB DATASET
EmoVoxCeleb is built from large-scale, unlabeled speaking face-tracks by transferring frame-level facial emotion predictions to synchronous speech segments. The resulting dataset contains diverse speakers and languages but has strongly imbalanced, automatically generated emotion labels.
- VoxCeleb contributes over 100,000 speech segments from interviews with 1,251 celebrities spanning ages, nationalities, professions, accents, genders, and languages.The dataset contains no human emotion labels before teacher-based annotation.
- The Heard-Val split shares identities with training, whereas Unheard-Val contains identities disjoint from training.This split design supports evaluation under heard and unheard speaker identities.
- Teacher predictions are aggregated by max-pooling across frames into eight-dimensional emotion vectors for synchronous speech segments, producing the EmoVoxCeleb annotations.The authors release frame-level predictions because the best aggregation strategy remains open; automated labels may be noisier than manual annotations.
- EmoVoxCeleb is heavily skewed toward a small number of emotions, particularly neutral, although it retains some emotional diversity and resembles the broader trend in Afew 6.0.Afew 6.0 has a significantly more balanced label distribution; Figure 4 uses a logarithmic y-axis.
- Approximately 5 million individual frames are annotated from face tracks sampled every 0.24 seconds.
5 EXPERIMENTS
The experiments distill facial emotion predictions into speech embeddings using paired four-second audio-visual segments, then evaluate those embeddings on external acted speech corpora. The student beats chance but remains below the facial teacher, while cross-corpus and identity-generalization issues remain important discussion points.
- 5.1 Implementation Details: The student learns to match max-pooled eight-dimensional teacher responses for randomly sampled four-second speech segments using temperature-scaled cross-entropy.Teacher and student predictions are passed through softmax layers before the loss is computed.
- 5.1 Implementation Details: The student uses a VGG-M/VGGVox-style speech architecture, while the input is a minimally processed amplitude spectrogram derived from four seconds of raw audio.The spectrogram uses a 25 ms Hamming window, 10 ms hop, and size 512 × 400; the network accepts variable-length inputs.
- 5.2 Results on external datasets: Evaluation uses the acted, human-labelled, class-balanced RML and eNTERFACE datasets, which are relatively clean and differ from noisy EmoVoxCeleb conditions.RML is specifically included to assess generalization to multilingual speech.
- 5.2 Results on external datasets: Student predictions are treated as eight-dimensional embeddings rather than directly evaluated because training and test emotion sets differ and predictions skew toward neutral.A learned map converts the embeddings into the target emotion labels.
- 5.2 Results on external datasets: The student performs significantly better than chance but below the facial teacher, indicating that facial supervision can transfer to speech despite differing evaluation conditions.Performance is reported using 10-fold cross-validation alongside random-classifier, teacher, and prior speech-classifier baselines.
- 5.3 Discussion: EmoVoxCeleb embeddings generalize across corpora with differences in dataset nature and labeling scheme, although performance remains below the supervising teacher.The discussion frames this as progress toward speech emotion embeddings usable across multiple corpora without speech annotation.
- 5.3 Discussion: A potential student shortcut is learning speaker identity instead of emotion, but heardVal and unheardVal performance is similar at 0.71 versus 0.69 mean ROC AUC.This comparison is used to examine identity bias in matching teacher predictions.
6 CONCLUSIONS
Using emotion-unlabelled video, the authors transfer emotional information from faces to speech and obtain reasonable benchmark performance. The approach achieves state-of-the-art facial emotion recognition and establishes cross-modal distillation benchmarks for speech emotion recognition.
- The speech emotion model learned from emotion-unlabelled video achieves reasonable classification performance on standard benchmarks, far above random.
- The approach achieves state-of-the-art performance on facial emotion recognition on the supervised FERPlus benchmark.
- The work sets benchmarks for cross-modal distillation methods on the RML and eNTERFACE speech emotion recognition datasets.
- Because video data is almost limitless and freely available from YouTube and other sources, future work can scale to larger unlabelled datasets.