Source-linked AI summary
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, Xie Chen
TL;DR
Emotion representation from speech is important for emotional tasks, yet existing SSL models and task-specific approaches leave a need for universal emotion representations. emotion2vec uses self-supervised online distillation with utterance- and frame-level losses, achieving broad performance across datasets, languages, and emotion tasks while requiring only linear downstream layers for SER.
Problem
Existing SSL models are not entirely suitable for emotional tasks, while fine-tuning is computationally costly and prior conclusions may be data-specific or model-constrained.
Method
emotion2vec is pre-trained on 262 hours of unlabeled emotion data through online distillation combining utterance-level and frame-level losses.
Results
emotion2vec outperforms mainstream SSL and specialist SER models on IEMOCAP with frozen features and linear downstream layers, and generalizes across 10 languages, datasets, environments, and other emotion tasks.
Takeaways & Limitations
The experiments support emotion2vec as a universal speech emotion representation across different tasks, languages, and scenarios.
Takeaways & Limitations
The paper identifies scaling emotion representation with more data and larger parameters as future work.
Abstract
from arXiv · showhide
We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and frame-level loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field.
1 Introduction
Speech emotion representation supports tasks including SER and sentiment analysis, but traditional features lack semantic richness and existing SSL models are not fully suited to emotion. emotion2vec addresses this need as a universal representation model trained on unlabeled emotion data.
- Traditional FBank and MFCC speech features provide limited semantic information, restricting performance on emotional tasks.
- Existing speech SSL models improve emotional-task performance but remain not entirely suitable for emotion-focused applications.
- Fine-tuning SSL models for SER requires substantial computation, while reported conclusions may be data-specific or model-constrained.
- Vesper is an SER specialist model whose universal representation capability remains to be demonstrated.
- emotion2vec uses 262 hours of open-source emotion data and combines utterance-level and frame-level losses through online distillation.
2 Related Work
Speech emotion representation research has largely reused general speech pre-training models or adapted them for specific emotional tasks. emotion2vec is presented as a universal speech emotion representation model spanning emotion-related tasks.
- Self-supervised learning has become successful across natural language processing, computer vision, and speech processing.
- The paper presents the first universal speech emotion representation model.
- Prior work generally uses speech pre-training models directly or fine-tunes them on task-specific emotional data, mostly for SER.
- For other emotion tasks, including multimodal emotion recognition, common practice similarly relies on general speech representation approaches.
3 Methods
emotion2vec is trained with online teacher–student distillation that combines utterance-level and frame-level objectives. The framework then supports frozen representations with lightweight downstream models.
- 3.1 Model Pipeline: Online distillation trains emotion2vec with teacher and student networks sharing feature-extractor and Transformer-backbone components.The student is optimized by backpropagation, while the teacher is updated using exponential moving average.
- 3.1 Model Pipeline: The student masks consecutive feature frames and prepends learnable utterance embeddings before backbone processing.The teacher processes unmasked downsampled features, providing targets for the student outputs.
- 3.1 Model Pipeline: For downstream tasks, emotion2vec is frozen and a lightweight downstream model is trained on its representations.This separates representation pre-training from task-specific modeling.
- 3.2 Utterance-level Loss: Utterance-level loss uses temporal pooling to align teacher and student outputs and learn global emotion information.The paper evaluates token, chunk, and global embedding strategies for computing this loss.
- 3.3 Frame-level Loss: Frame-level loss learns contextual emotion by comparing teacher and student frame outputs only at masked positions.This objective follows a masked-prediction setup, restricting the loss to the masked frame indices.
- 3.4 Online Distillation: The total training loss combines utterance-level and frame-level losses, while the teacher parameters are updated through EMA during pre-training.The teacher starts from the student initialization, and its update coefficient increases linearly during training.
4 Experiments Setup
The experiments use multiple initial architectures, broad multilingual and multitask emotion datasets, and simple downstream prediction heads to evaluate emotion2vec representations.
- 4.1 Initial Model: The initial models include data2vec and data2vec 2.0, which share a feature extractor design but use different backbone network designs.Both alternatives are used to assess emotion2vec with different initial architectures.
- 4.1 Initial Model: The feature extractor is a seven-layer 1-D convolutional network that downsamples 16000 Hz audio by 320× and produces 512-dimensional representations.A linear projection transforms the representation dimension from 512 to 768 before masking and backbone processing.
- 4.1 Initial Model: The data2vec backbone combines learnable convolutional positional encoding with a 12-layer Transformer, while data2vec 2.0 adds a CNN decoder for masked-feature reconstruction.The data2vec 2.0 encoder processes only non-masked downsampled features before Gaussian-noise completion and CNN decoding.
- 4.2 Pre-training: Pre-training uses 262 hours of unlabeled emotion data for 100 epochs on four NVIDIA A10 GPUs, with simulated 16-GPU updates and Adam optimization.The training configuration includes a maximum token batch size of 1 × 10^6 and a learning rate of 7.5 × 10^-5.
- 4.4 Supervised Fine-tuning: Downstream architectures are intentionally simple: non-sequential tasks use two linear layers with ReLU, whereas sequential tasks use two GRU layers.This setup is designed to demonstrate the representation ability of the pre-trained model.
- 4.3 Datasets and Evaluation: The evaluation covers 18 emotional datasets across 10 languages, with IEMOCAP serving as the main four-class cross-validation benchmark after merging excited with happy.Additional evaluations test language generalization on nine out-of-domain datasets and task generalization for song emotion recognition, conversational emotion prediction, and sentiment analysis.
5 Results
emotion2vec demonstrates strong performance across speech emotion recognition settings, languages, and related emotion tasks using simple downstream models. Ablations further support the contribution of its initialization and combined loss design.
- Main Results: emotion2vec outperforms existing SSL models and performs on par with or better than specialist SER models using only linear downstream layers.The specialist models use downstream networks with 2x, 135x, and 114x more parameters than emotion2vec, respectively.
- Language Generalization: emotion2vec achieves state-of-the-art performance across English datasets, including noisy and out-of-domain recording environments.MELD tests performance in complex environments, while RAVDESS and SAVEE represent out-of-domain settings.
- Language Generalization: emotion2vec captures emotion patterns across languages and shows state-of-the-art performance under WA, UA, and WF1 evaluation.The experiments target out-of-domain language SER datasets using frozen models and downstream linear layers.
- Task Generalization: emotion2vec generalizes beyond SER, outperforming known SSL models without finetuning on song emotion recognition and improving both speech-only and speech-text EPC performance.Additional evaluations cover song emotion recognition, emotion prediction in conversation, and sentiment analysis; it also outperforms data2vec, WavLM, and supervised Whisper Encoder baselines for sentiment analysis.
- Ablation Studies: Initializing from a pretrained model improves performance, with data2vec 2.0 initialization outperforming data2vec initialization.Both are representative online-distillation models pretrained on LibriSpeech 960 hours.
- Ablation Studies: Frame-level loss is necessary for reasonable performance, while combining utterance-level and frame-level losses yields the strongest results.Chunk embeddings perform best for utterance-level loss, and a 1:1 weighting of utterance-level and frame-level losses works best.
6 Conclusion
emotion2vec is presented as a universal emotion representation model trained through self-supervised online distillation on unlabeled emotion data. Extensive experiments support its use across different tasks, languages, and scenarios, while future work will study scaling with more data and larger models.
- emotion2vec is pre-trained on 262 hours of unlabeled emotion data through self-supervised online distillation.
- Combining utterance-level and frame-level losses during emotion pretraining is shown to be effective.
- Extensive experiments demonstrate emotion2vec can extract emotion representations across different tasks, languages, and scenarios.
- Future work will explore the scaling law of emotion representation models using more data and larger parameters.