Source-linked AI summary
The Role of Phonetic Units in Speech Emotion Recognition
Jiahong Yuan, Xingyu Cai, Renjie Zheng, Liang Huang, Kenneth Church
TL;DR
Emotion recognition must classify speech despite acoustic variability across phonetic units, and this paper bridges emotion classification with speech recognition. It fine-tunes Wav2vec 2.0 for emotion-dependent phonetic units; these models outperform utterance models, with broad phonetic classes performing best.
Problem
Acoustic variability across phonetic units complicates emotion recognition, while emotion recognition classifies utterances and speech recognition predicts sequences.
Source: §1 · p. 1Method
Results
Phoneme, broad-phonetic-class, and syllable models significantly outperform the utterance model, with broad phonetic classes performing best and significant improvement over most previously reported IEMOCAP results.
Source: §5 · p. 4Takeaways & Limitations
Phonetic units are helpful and should be incorporated in speech emotion recognition.
Source: §5 · p. 4Mandarin results may be affected by 8-to-16 kHz upsampling that leaves 4–8 kHz frequencies absent; the authors call for robustness work across recording-condition and sampling-rate mismatches.
Source: §4.2 · p. 4, §5 · p. 4
Abstract
from arXiv · showhide
We propose a method for emotion recognition through emotiondependent speech recognition using Wav2vec 2.0. Our method achieved a significant improvement over most previously reported results on IEMOCAP, a benchmark emotion dataset. Different types of phonetic units are employed and compared in terms of accuracy and robustness of emotion recognition within and across datasets and languages. Models of phonemes, broad phonetic classes, and syllables all significantly outperform the utterance model, demonstrating that phonetic units are helpful and should be incorporated in speech emotion recognition. The best performance is from using broad phonetic classes. Further research is needed to investigate the optimal set of broad phonetic classes for the task of emotion recognition. Finally, we found that Wav2vec 2.0 can be fine-tuned to recognize coarser-grained or larger phonetic units than phonemes, such as broad phonetic classes and syllables.
1. Introduction
The paper addresses phonetic variability as a confound in speech emotion recognition by treating emotion recognition as emotion-dependent speech recognition. It explores phonetic units and pretrained Wav2vec 2.0 to handle model-data demands and cross-lingual questions.
1. Introduction
- Earlier speech emotion recognition work shifted from feature engineering toward deep-learning architectures, with IEMOCAP serving as a common benchmark.The passage reports that several ICASSP 2020 papers reached at least 70% cross-validation accuracy on IEMOCAP using acoustic data only.Source: §1 · p. 1
- Phonetic units vary acoustically, complicating emotion recognition; the study adopts phonetically aware models rather than making emotion models independent of that variability.The cited example compares different vowel spectra within the same speaker and emotion.Source: §1 · p. 1
- Prior studies examined articulatory cues, phonetic-class features, phoneme-specific models, and attended phonemes in relation to emotion.Reported findings included emotion-related spectral information in consonant regions and higher accuracy from phoneme-specific models.Source: §1 · p. 1
- The method combines speech recognition and emotion classification by recognizing emotion-dependent phonetic units such as phonemes.The authors contrast this simpler one-step approach with prior high- and low-arousal HMM/GMM acoustic models.Source: §1 · p. 1
- Emotion-dependent models may require more training data because they increase the number of units or models with the number of emotions.The authors argue that pretrained Wav2vec 2.0 can reduce the data needed for speech recognition models, enabling fine-tuning with small amounts of data.Source: §1 · p. 1
- The study asks whether pretrained acoustic representations can recognize coarser or larger units, and whether broad classes or syllables improve cross-lingual robustness over phonemes.The comparison reflects that phonemes are language-specific, while broad phonetic classes and syllables are language-general.Source: §1 · pp. 1–2
2. Fine-tuning Wav2vec2.0 for emotion recognition
The procedure fine-tunes Wav2vec 2.0 to recognize emotion-dependent units, then maps recognized units to an utterance-level emotion. The paper compares phonemes, broad phonetic classes, syllables, and entire utterances as units.
2.1. The procedure
- The approach builds on Wav2vec 2.0, a self-supervised speech-representation framework pretrained on audio alone.Its architecture includes convolution and self-attention layers; pretraining masks speech input in latent space and solves a contrastive task over quantized representations.Source: §2.1 · p. 2
- A randomly initialized linear projection maps Wav2vec 2.0 contextual representations to emotion-dependent unit classes, and the model is fine-tuned by minimizing CTC loss.Figure 2 illustrates the fine-tuning procedure.Source: §2.1 · p. 2, Figure 2 · p. 2
- Recognized emotion-dependent units are mapped to an utterance emotion by majority vote.For example, an utterance is classified as happy when most recognized units are labeled happy.Source: §2.1 · p. 2
2.2. Phonetic units
- The compared recognition targets are phonemes, broad phonetic classes, syllables, and entire utterances.Figure 4 summarizes the four types with examples.Source: §2.2 · p. 2
3. Experiments on IEMOCAP
On IEMOCAP, phonetic-unit emotion models outperform most previously reported results, with broad phonetic classes performing best. The experiments also show that Wav2vec 2.0 can recognize phonetic units coarser than phonemes.
3.1. Data
- IEMOCAP is an English benchmark dataset; the study used 5,531 utterances across neutral, happy, angry, and sad categories.The dataset comprises 12 hours of speech from 10 professional actors.Source: §3.1 · p. 2
3.2. Fine-tuning
- Ten-fold cross-validation held out one speaker for testing in each fold and used the other nine speakers for fine-tuning.The model was fine-tuned for 15k updates; the output classifier alone was trained for the first 10k updates before the Transformer was also updated.Source: §3.2 · p. 2
3.3.1. Classification of emotion
- Emotion models using phonetic units significantly outperform most previously reported results, except the utterance model; broad phonetic classes perform best, while phonemes and syllables perform equally well.The utterance model uses one emotion-dependent unit per emotion and does not consider phonetic content.Source: Table 1 · p. 3, §3.3.1 · p. 3
3.3.2. Recognition of phonetic units
- 11.5% (broad phonetic classes), 14.9% (phonemes), and 16.3% (syllables) are the recognition error rates when emotion type is excluded.The results show Wav2vec 2.0 can recognize broad phonetic classes and syllables without a language model.Source: Table 3 · p. 3, §3.3.2 · p. 3
4. Experiments on datasets in other languages
IEMOCAP-trained models performed reasonably well on the German and Arabic datasets, while performance on Mandarin was poor; sampling-rate mismatch was identified as a probable main factor. The experiments tested phonetic-unit models on three other language datasets.
4.1. Data
- Models were tested on EmoDB, KSUEmotions, and Mandarin Affective Speech after being retrained on the full IEMOCAP dataset.The experiments used the same hyperparameters as the earlier experiments.Source: §4.1 · p. 4
- The evaluation used neutral, happy, angry, and sad utterances from each of the three datasets.EmoDB is German, KSUEmotions contains Modern Standard Arabic, and Mandarin Affective Speech contains Mandarin; Table 4 lists the utterance counts by emotion and dataset.Source: §4.1 · p. 4, Table 4 · p. 4
4.2. Results
- 75.2% weighted accuracy was achieved using initials and finals in seven-fold cross-validation on Mandarin Affective Speech.This result indicates that the poor performance of IEMOCAP models on Mandarin was not due to the dataset's nature.Source: §4.2 · p. 4
- The authors identify sampling-rate mismatch as probably the main factor in Mandarin performance and note that linguistic and cultural differences may also contribute.The Mandarin recordings were upsampled from 8 kHz to 16 kHz, but still lacked frequency components between 4 and 8 kHz; direct testing of 8 kHz audio with 16 kHz models performed even worse.Source: §4.2 · p. 4
- IEMOCAP-trained models performed reasonably well on German and Arabic, whereas they performed poorly on Mandarin.The authors suggest the method has potential for cross-lingual emotion recognition; they attribute Mandarin performance probably mainly to sampling-rate mismatch, alongside possible linguistic and cultural differences.Source: §4.2 · p. 4
- The reasons broad phonetic classes and syllables may be more robust across languages remain hypotheses requiring further study.The proposed explanations include their language-general nature and the greater amount of training data available for these units.Source: §4.2 · p. 4
5. Conclusions and discussion
The paper fine-tunes Wav2vec 2.0 to recognize emotion-dependent phonetic units, and reports improved IEMOCAP results over most earlier reports. Phonetic-unit models outperform the utterance model, with broad phonetic classes performing best; cross-lingual results favor broader units, while the reasons remain uncertain.
5. Conclusions and discussion
- Phoneme, broad-phonetic-class, and syllable models all significantly outperform the utterance model.The authors conclude that phonetic units are helpful for speech emotion recognition and should be incorporated.Source: §5 · p. 4
- Broad phonetic classes achieved the best performance among the tested units.The authors suggest these classes may direct the model away from within-class phoneme differences and toward variability among classes, and call for study of the optimal class set.Source: §5 · p. 4
- Broad-phonetic-class and syllable models outperform phoneme models in cross-lingual emotion recognition.Possible explanations are that the former units are language-general or have more training data; further research is needed to test these hypotheses.Source: §5 · p. 4
- Wav2vec 2.0 can be fine-tuned to recognize broad phonetic classes and syllables without a language model.The study also fine-tuned it to recognize phonemes and entire utterances.Source: §5 · p. 4
- The proposed Wav2vec 2.0 method achieved significant improvement over most previously reported results on IEMOCAP.The authors identify robustness across datasets with recording-condition and sampling-rate mismatches as future work.Source: §5 · p. 4