PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion Recognition from Speech

Kannan Venkataramanan, Haresh Rengaraj Rajamohan

arXiv:1912.10458v1cs.SDcs.CLeess.AS

TL;DR

Speech assistants are not proficient at recognizing users’ moods, motivating research on speech-based emotion recognition. The study compares audio features and model architectures on RAVDESS, with Log-Mel spectrograms performing best in its reported comparison and feature choice mattering more than model complexity.

  • Problem

    Virtual assistants can understand commands but are not proficient at recognizing users’ moods, motivating speech-based emotion recognition.

    Source: §1 · p. 1

  • Method

    The study compares audio features and classifiers for speech emotion recognition using RAVDESS, including Log-Mel spectrograms and a four-layer 2D CNN.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    Log-Mel spectrogram features were reported as conclusively better for this task, and a four-layer 2D CNN achieved 70% accuracy on the test set.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that audio-feature choice affects emotion-recognition results more than model architecture, and that no single feature works well across all signal-processing tasks.

    Source: §1 · p. 2

    RAVDESS uses English-speaking actors from Toronto, acted emotions, and only two statements, limiting its language, cultural, and lexical coverage.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

In this work, we conduct an extensive comparison of various approaches to speech based emotion recognition systems. The analyses were carried out on audio recordings from Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). After pre-processing the raw audio files, features such as Log-Mel Spectrogram, Mel-Frequency Cepstral Coefficients (MFCCs), pitch and energy were considered. The significance of these features for emotion classification was compared by applying methods such as Long Short Term Memory (LSTM), Convolutional Neural Networks (CNNs), Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs). On the 14-class (2 genders x 7 emotions) classification task, an accuracy of 68% was achieved with a 4-layer 2 dimensional CNN using the Log-Mel Spectrogram features. We also observe that, in emotion recognition, the choice of audio features impacts the results much more than the model complexity.

1 Introduction

The study compares audio features and model architectures for speech emotion recognition, motivated by virtual assistants’ limited ability to recognize users’ moods. It reports its strongest performance with a four-layer 2D CNN using Log-Mel spectrograms, and argues feature choice matters more than model architecture.

1 Introduction

  • Speech conveys linguistic content and paralinguistic cues, including tone, pitch, and emotion, which can help identify a speaker’s mental state.The authors connect this information to more natural interactions with virtual assistants.Source: §1 · p. 1
  • Emotion can be represented continuously through activation and valence or discretely as categories such as angry, sad, and happy.Activation relates to energy and pitch; valence helps distinguish emotions such as anger and happiness.Source: §1 · p. 1
  • The authors argue that emotion-recognition performance depends more on audio-feature choice than on model architecture, with no single feature suited to every signal-processing task.They describe features as handcrafted for particular tasks and deep learning as extracting hierarchical speech representations from them.Source: §1 · p. 2
  • Prior speech-emotion studies cited here report 61% accuracy using MFCCs and LPCC with HMMs and SVMs on SAVEE, and 56% using a 1D CNN on Chinese-language MFCCs.The examples illustrate comparisons across features, models, and datasets.Source: §1 · p. 2
  • A four-layer 2D CNN with Log-Mel spectrogram features achieved 70% accuracy on the RAVDESS test set and 62% on TESS.The study assessed robustness by testing on a different dataset, TESS.Source: §1 · p. 2

2 Datasets

The paper uses RAVDESS, a gender-balanced emotional speech and song dataset, and describes its class distribution, validation, preparation, and cleaning. It notes limitations tied to the dataset’s actors, language and cultural setting, and restricted statement variety.

2.1 Data Selection

  • RAVDESS contains 1,440 speech and 1,012 song utterances from 24 professional actors, with gender balance and multiple emotions and intensities.Each actor performed 104 vocalizations; the dataset’s emotion classes are approximately equally distributed at 15% each.Source: §2.1 · p. 2

2.2 Selection Bias & Limitations

  • RAVDESS is limited to 24 English-speaking Toronto actors, acted emotions, and two statements, constraining language and cultural coverage and lexical variety.The authors say models trained on this dataset should be extensively validated before deployment.Source: §2.2 · p. 3

2.3 Data Preparation

  • Each audio file’s seven-part numerical identifier encodes modality, vocal channel, emotion, intensity, statement, repetition, and actor; emotion is the target label.Source: §2.3 · p. 3

2.4 Data Cleaning

  • The authors trimmed silence and applied Wiener filtering after trying noise-reduction techniques, reporting improved audio quality without losing important content.The recordings were approximately three seconds long and had minute noise patterns.Source: §2.4 · pp. 3–4

3 Feature Engineering and Modelling

The section compares time- and frequency-domain audio features and describes how MFCCs and log-Mel spectrograms are prepared for emotion recognition. It outlines CNN, recurrent, and HMM approaches for modeling speech.

3.1 Feature Selection

  • The study focuses its feature analysis on MFCCs and Mel-spectrograms, while distinguishing time-domain from frequency-domain audio features.Time-domain features include short-term energy and zero-crossing rate; frequency-domain features include spectrograms and MFCCs.Source: §3.1 · p. 4

3.1.1 Mel-Frequency Cepstrum (MFC)

  • MFCC extraction frames the signal, estimates each frame’s frequencies, applies Mel filtering and a logarithmic transform, then uses a discrete cosine transform to decorrelate energies.MFCCs capture the envelope of the short-time power spectrum.Source: §3.1.1 · pp. 4–5
  • MFCCs, delta coefficients, and delta-delta coefficients provide alternative feature configurations, with the latter capturing changes in MFCC trajectories.The study trained models using each of these three configurations.Source: §3.1.1 · p. 5

3.1.2 Mel-Spectograms

  • A log-Mel spectrogram represents audio over time and log-Mel frequency; MFCCs and log-Mel spectrograms can be supplied as image inputs to deep-learning classifiers.The authors motivate the logarithmic representation by noting that human sound perception is logarithmic.Source: §3.1.2 · p. 5

3.2.1 Convolutional Neural Networks

  • CNNs apply shared kernels to exploit two-dimensional image structure, and the authors also consider 3D CNNs to model short- and long-term dependencies in reshaped audio.The 3D-audio frame length was set to 250 ms.Source: §3.2.1 · p. 5

3.2.2 Recurrent Neural Networks

  • LSTMs maintain sequence context through hidden states, addressing exploding and vanishing gradients; the authors motivate them because emotion inference requires longer time steps than the 10 ms feature frames.They also implemented CNN-LSTM models with 250 ms frame lengths and CNN-LSTMs on raw audio.Source: §3.2.2 · pp. 5–6

3.2.3 Hidden Markov Models

  • The HMMs model phones as hidden states and classify utterances by the likelihood of their phone sequence under each emotion class.The study implemented Gaussian and Gaussian-mixture emissions, with one HMM per class.Source: §3.2.3 · p. 6

4 Experiments

The experiments compare engineered audio features and model architectures under speaker-independent data splits, with a separate TESS evaluation. A four-layer 2D CNN with global average pooling and log-Mel features achieved 88% accuracy on positive-versus-negative emotion classification.

4.1 Data

  • The experiments used MFCCs with 10 ms windows and 5 ms hops, and 128 log-Mel features with 14 ms windows and 3.5 ms hops.MFCC counts above 25 per frame gave no additional performance gain; pitch, magnitude, energy, and their deltas were also added to MFCCs.Source: §4.1 · p. 6

4.2 Train, Validation and Test Data Creation

  • Actors 1–20 were used for training, actors 21–22 for validation, and actors 23–24 for testing to avoid actor-characteristic leakage.Both validation and test sets included male and female actors.Source: §4.2 · p. 6
  • The authors assessed robustness on TESS, a dataset unrelated in origin to RAVDESS but recorded from North-American speakers.They also cleaned the TESS data before evaluation.Source: §4.2 · p. 6

4.3 Methodology

  • The MFCC DNN initially failed to recognize the surprised class, so subsequent analysis excluded that class; adding MFCCs and deltas produced the best DNN performance.The authors expected log-Mel features to perform better because MFCC visualizations contained little higher-frequency information.Source: §4.3 · p. 7
  • For 2D CNNs, increasing depth beyond four layers did not improve performance, while global average pooling reduced fully connected parameters and allowed variable-sized inputs.The 14-class champion used 12 × 12 and 7 × 7 filters in its first two layers.Source: §4.3 · p. 7
  • HMMs achieved around 20% accuracy with MFCCs and 32% with log-Mel spectrogram features.The authors attributed the weaker HMM results probably to their poor representative power.Source: §4.3 · p. 7
  • The authors compared MFCC and log-Mel inputs across 3D CNN, CNN-LSTM, and LSTM models, and selected models using unweighted accuracy.Fixed-size inputs were trimmed or padded to three seconds.Source: §4.3 · pp. 7–8
  • 88% accuracy was achieved on positive-versus-negative emotion classification by a four-layer 2D CNN with global average pooling and log-Mel features.The result is reported for unseen test and validation data.Source: §4.3 · p. 8

5 Results and Conclusions

The authors conclude that engineered features outperform raw audio in their emotion-recognition experiments, with log-Mel spectrograms outperforming MFCCs. Their results also favor 2D CNNs and report performance on TESS, while identifying a speaker-age and pitch-related gender-classification error as a scope concern.

5 Results and Conclusions

  • Log-Mel spectrogram features were conclusively better than MFCCs, and engineered features performed much better than raw audio, which the authors suggest may reflect limited data.Adding pitch and energy to MFCCs also improved model performance.Source: §5 · p. 8
  • 2D CNNs performed best among the compared 2D CNN, 3D CNN, LSTM, and CNN-LSTM models, and gender-specific emotion classification led to higher performance.The authors relate gender-specific patterns to average male-female differences in pitch and energy.Source: §5 · p. 8
  • The RAVDESS-trained seven-class model achieved 62% accuracy on TESS, where chance accuracy was 14%.The authors interpret this as evidence that the models were robust and had not overfit to RAVDESS’s two sentences.Source: §5 · p. 8
  • On TESS, the best model classified one 64-year-old female speaker as male; the authors connect this error to her lower pitch.They propose exploring a hierarchical system that first predicts age group and gender before emotion classification.Source: §5 · p. 9

6 Appendix

The appendix illustrates MFCC and Log-Mel features across selected utterances and emotions, reports that other features show clearer overall patterns than Chroma, and defines zero-crossing rate.

6.2 Mel Spectrum Cepstral Coefficients (MFCC)

6.3 Chroma Features

  • Chroma projects audio onto 12 semitone bins; initial analysis of RAVDESS songs found other features showed clearer patterns overall.Source: §6.3 · p. 13

6.4 Zero-crossing rate

  • Zero-crossing rate measures how often a signal changes sign, including transitions through zero.Source: §6.4 · p. 13

6.5 Log Mel Spectogram

Loading 1912.10458v1…