PaperlayerView exact revision on arXiv

Source-linked AI summary

An Ensemble 1D-CNN-LSTM-GRU Model with Data Augmentation for Speech Emotion Recognition

Md. Rayhan Ahmed, Salekul Islam, Ph. D, A. K. M. Muzahidul Islam, Ph. D, Swakkhar Shatabda, Ph. D

arXiv:2112.05666v2cs.SDeess.AS

TL;DR

SER requires models that can recognize emotions from speech despite limitations in available datasets and conventional feature-based classification. This paper combines CNN-based local feature extraction, recurrent contextual modeling, data augmentation, and weighted ensembling across five benchmark datasets. The ensemble is reported to achieve state-of-the-art weighted average accuracy across all evaluated datasets.

  • Problem

    Speech emotion recognition is challenging because datasets can be insufficient or imbalanced, while conventional classifiers have limitations in learning hidden patterns from speech features.

  • Method

    The paper combines CNN-based local feature-acquiring blocks with LSTM or GRU layers, handcrafted speech features, three audio augmentation techniques, and weighted ensembling across five benchmark datasets.

  • Results

    The ensemble model achieves state-of-the-art weighted average accuracy across all five evaluated SER datasets.

  • Takeaways & Limitations

    Adding recurrent global contextual modeling and weighted ensembling provides the paper's supported route to improved SER performance over the baseline and individual models.

  • Takeaways & Limitations

    The paper states that GRU is less complicated and faster to train for relatively small datasets, whereas LSTM should be applied to large-volume datasets.

Abstract

from arXiv · show

In this paper, we propose an ensemble of deep neural networks along with data augmentation (DA) learned using effective speech-based features to recognize emotions from speech. Our ensemble model is built on three deep neural network-based models. These neural networks are built using the basic local feature acquiring blocks (LFAB) which are consecutive layers of dilated 1D Convolutional Neural networks followed by the max pooling and batch normalization layers. To acquire the long-term dependencies in speech signals further two variants are proposed by adding Gated Recurrent Unit (GRU) and Long Short Term Memory (LSTM) layers respectively. All three network models have consecutive fully connected layers before the final softmax layer for classification. The ensemble model uses a weighted average to provide the final classification. We have utilized five standard benchmark datasets: TESS, EMO-DB, RAVDESS, SAVEE, and CREMA-D for evaluation. We have performed DA by injecting Additive White Gaussian Noise, pitch shifting, and stretching the signal level to generalize the models, and thus increasing the accuracy of the models and reducing the overfitting as well. We handcrafted five categories of features: Mel-frequency cepstral coefficients, Log Mel-Scaled Spectrogram, Zero-Crossing Rate, Chromagram, and statistical Root Mean Square Energy value from each audio sample. These features are used as the input to the LFAB blocks that further extract the hidden local features which are then fed to either fully connected layers or to LSTM or GRU based on the model type to acquire the additional long-term contextual representations. LFAB followed by GRU or LSTM results in better performance compared to the baseline model. The ensemble model achieves the state-of-the-art weighted average accuracy in all the datasets.

1. Introduction

SER is presented as an active HCI and digital-signal-processing problem, with deep learning and ensemble architectures proposed to improve speech-emotion classification across benchmark datasets.

  • Speech emotion recognition distinguishes emotions from speech signals and is identified as an active research area in HCI and digital signal processing.
  • Traditional SER systems classify handcrafted speech features with linear or nonlinear classifiers, while deep learning learns hidden patterns from extracted feature vectors.
  • The paper proposes baseline CNN-FCN, CNN-LSTM-FCN, CNN-GRU-FCN, and weighted-ensemble frameworks for SER.
  • Experiments use five publicly available benchmark datasets: TESS, RAVDESS, SAVEE, CREMA-D, and EMO-DB.
  • The study combines CNN-based local feature-acquiring blocks with LSTM- or GRU-based global feature acquisition, followed by fully connected and softmax layers.

2. Related Works

Related work spans handcrafted-feature classifiers, deep neural architectures, attention and transfer learning, and ensemble methods for improving SER representations and prediction stability.

  • SER research uses time-domain, frequency-domain, cepstral, and statistical features, including ZCR, RMS, Mel-scaled spectrograms, Chromagram, and MFCC.
  • Earlier SER studies applied SVM, MLP, GMM, Naïve Bayes, HMM, and related conventional classifiers to speech signals.
  • Deep learning architectures including CNN, RNN, GRU, LSTM, Bi-LSTM, and Bi-GRU learn discriminative speech representations with varying success.
  • Prior work combines network types to capture local and long-term contextual dependencies, while other studies use attention mechanisms to weight emotional frames or integrate signal information.
  • Transfer-learning approaches use pretrained neural networks to learn high-level feature representations and address SER cross-domain challenges.
  • Ensemble methods combine multiple estimators for more stable and accurate predictions, although training several architectures increases time requirements.
  • Comparative evaluations consider methodology, extracted features, datasets, feature dimensions, augmentation methods, results, and publication year.

3. Proposed Methods

The proposed SER methods combine CNN-based local feature extraction with LSTM or GRU global contextual modeling, and aggregate three models through a weighted ensemble.

  • Overall framework: The approach targets multilingual SER using five benchmark datasets, data augmentation, and combined time-, frequency-, and cepstral-based speech features.Augmentation uses AWGN addition, pitch shifting, and time stretching; extracted features feed the proposed models.
  • Baseline model-A: Model-A extracts hidden local patterns from ZCR, Chromagram, MFCC, RMS, and LMS feature vectors using seven sequential LFABs before FCN classification.Each LFAB contains convolutional, max-pooling, batch-normalization, and dropout layers.
  • Recurrent variants: Models-B and-C extend Model-A with a 512-unit LSTM or GRU GFAB after the final LFAB to learn global contextual correlations.Both variants retain Model-A’s FCN configuration and apply 50% dropout after the GFAB.
  • LSTM mechanism: The LSTM uses input, forget, and output gates to update cell and hidden states from current inputs and previous timestep information.The sigmoid and tanh operations regulate information incorporation, state updating, and output generation.
  • GRU mechanism: The GRU uses reset and update gates with one hidden state to retain or overlook information while modeling long-term and short-term dependencies.The GRU’s final hidden state is computed from the update gate, preceding hidden state, and candidate hidden state.
  • Weighted ensemble model-D: The weighted ensemble combines Models-A, B, and C by selecting model weights through grid search and applying them to prediction results.The weighted prediction is formed by summing the weighted model outputs before selecting the class.

4. Experimental Analysis

The study evaluates speech-emotion-recognition preprocessing across five datasets, using augmentation and handcrafted acoustic features before deep-learning classification. Feature extraction combines spectral, pitch-related, amplitude, and zero-crossing information into a 155-dimensional representation.

  • 4.1. Datasets: Five datasets covering English and German speech are used, with class-wise utterance distributions presented for CREMA-D, RAVDESS, SAVEE, EMO-DB, TESS, and their combination.
  • 4.2. Data augmentation: Data augmentation applies AWGN injection, time-stretching, and pitch shifting to expand the relatively small speech datasets.The reported augmented sample counts are 8400, 7200, 1920, 2140, and 44652 for TESS, RAVDESS, SAVEE, EMO-DB, and CREMA-D, respectively.
  • 4.3. Feature extraction: MFCC extraction frames speech over 20–30 ms windows advanced every 10 ms, applies DFT and Mel filtering, then produces 13 lower-dimensional MFCC features.The process uses 26 Mel-scaled filter-bank filters, logarithmic energies, and DCT; the sampling rate is 44.1 kHz with DCT-2.
  • 4.3. Feature extraction: The feature set includes 12 Chromagram bins and 128 Log-Mel Spectrogram features, representing pitch-related and time-frequency signal characteristics.Using multiple audio features integrates sound characteristics such as pitch, tone, and harmony into training samples.
  • 4.3. Feature extraction: ZCR measures amplitude crossings through zero, while RMS describes average signal amplitude and signal strength irrespective of sign.
  • 4.3. Feature extraction: The extracted representation contains 13 MFCC, 12 Chromagram, 128 LMS, one ZCR, and one RMS feature, forming a 155-dimensional vector.Features are normalized, split into training and testing data at an 80:20 proportion, and fed into the deep-learning model.

5. Results Analysis

The study evaluates four proposed models using accuracy and class-sensitive metrics across five datasets, with results reported before and after data augmentation. Data augmentation improved ensemble performance, while individual models showed dataset- and metric-dependent differences.

  • Evaluation metrics: Accuracy, precision, recall, F1-score, and macro-F1 are used to assess speech emotion recognition performance, including class-balanced evaluation.Macro-F1 averages per-class F1-scores without weighting classes by prevalence.
  • Overall performance: 99.46%, 95.42%, 95.62%, 93.22%, and 90.47% WAA are reported for ensemble model-D on TESS, EMO-DB, RAVDESS, SAVEE, and CREMA-D after data augmentation.These are the reported dataset-specific weighted average accuracies for the proposed ensemble.
  • Effect of data augmentation: Data augmentation further improved TESS performance, reaching 99.46% WAA for ensemble model-D, while EMO-DB and SAVEE were affected by small, imbalanced datasets.The paper associates the lower original performance on EMO-DB and SAVEE with overfitting, class imbalance, and limited training samples.
  • Performance curves: Figures 11–15 plot training and validation accuracy for models A, B, and C after augmentation over 1000 training epochs.The plots cover TESS, EMO-DB, RAVDESS, SAVEE, and CREMA-D.
  • Individual-model comparison: After data augmentation, model-A has the highest macro precision across datasets, whereas model-B generally leads macro recall, mean accuracy, and macro-F1.Model-B does not lead macro recall on CREMA-D, and model-A performs better there than the other models.
  • Individual-model comparison: Model-C performs lower than the other individual models, particularly on EMO-DB and SAVEE, while TESS and RAVDESS show higher macro precision and recall.The paper relates stronger convergence on TESS and RAVDESS to their more balanced class distributions.

6. Conclusion and Future Works

The conclusion identifies limited data as a challenge for deep-learning speech emotion recognition and summarizes a multi-dataset evaluation of augmented, feature-based CNN, LSTM, GRU, and ensemble systems.

  • Conclusion: The study addresses data scarcity and overfitting by evaluating handcrafted speech features, CNN-based LFABs, LSTM or GRU global blocks, and a weighted ensemble across five benchmark datasets.The datasets cover English and German speech.
  • Conclusion: With data augmentation, ensemble model-D achieves reported WAA values of 99.46%, 95.42%, 95.62%, 93.22%, and 90.47% across TESS, EMO-DB, RAVDESS, SAVEE, and CREMA-D.The paper describes these results as state-of-the-art weighted average accuracy.

CRediT authorship contribution statement

The authors’ contributions span conceptualization, methodology, analysis, validation, investigation, resources, data curation, and manuscript preparation.

  • Authorship contributions: Md. Rayhan Ahmed led conceptualization, methodology, formal analysis, validation, investigation, resources, data curation, and writing, while the other authors contributed analysis, validation, and writing.Salekul Islam, A.K.M. Muzahidul Islam, and Swakkhar Shatabda are each credited with formal analysis, validation, and manuscript writing.
Loading 2112.05666v2…