PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Modelling of Speech Emotion from Raw Speech

Siddique Latif, Rajib Rana, Sara Khalifa, Raja Jurdak, Julien Epps

arXiv:1904.03833v4cs.SDeess.AS

TL;DR

Speech emotion recognition relies on hand-engineered features that may not suit the classification objective, and it must model long temporal context. The paper proposes parallel CNN layers at multiple temporal resolutions, jointly trained with an LSTM-based classifier; the model performs better than existing raw-waveform methods and comparably to methods using state-of-the-art hand-engineered features on IEMOCAP and MSP-IMPROV.

  • Problem

    Hand-engineered features may not be best for emotion classification, and speech emotion recognition also faces the challenge of modelling long temporal context.

    Source: §1 · p. 1

  • Method

    The model uses parallel CNN layers with different filter lengths to extract multiple temporal resolutions from raw speech, jointly optimized with an LSTM-based classification network.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    The proposed network offers better accuracy than existing raw-speech waveform methods and comparable accuracy to methods using state-of-the-art hand-engineered features on IEMOCAP and MSP-IMPROV.

    Source: §7 · p. 4

  • Takeaways & Limitations

    Parallel convolutional layers with multiple filter lengths capture short- and long-term interactions from raw speech in the proposed emotion-recognition model.

    Source: §7 · p. 4

Abstract

from arXiv · show

Speech emotion recognition is a challenging task and heavily depends on hand-engineered acoustic features, which are typically crafted to echo human perception of speech signals. However, a filter bank that is designed from perceptual evidence is not always guaranteed to be the best in a statistical modelling framework where the end goal is for example emotion classification. This has fuelled the emerging trend of learning representations from raw speech especially using deep learning neural networks. In particular, a combination of Convolution Neural Networks (CNNs) and Long Short Term Memory (LSTM) have gained great traction for the intrinsic property of LSTM in learning contextual information crucial for emotion recognition; and CNNs been used for its ability to overcome the scalability problem of regular neural networks. In this paper, we show that there are still opportunities to improve the performance of emotion recognition from the raw speech by exploiting the properties of CNN in modelling contextual information. We propose the use of parallel convolutional layers to harness multiple temporal resolutions in the feature extraction block that is jointly trained with the LSTM based classification network for the emotion recognition task. Our results suggest that the proposed model can reach the performance of CNN trained with hand-engineered features from both IEMOCAP and MSP-IMPROV datasets.

1. Introduction

Speech emotion recognition has practical applications but is difficult because emotional expression varies with context. Hand-engineered features may not best serve classification, while long temporal context is important for recognizing emotion; the paper proposes parallel CNN layers with different filter widths alongside LSTM.

1. Introduction

  • Emotional expression varies with culture, age, gender, accent, and surrounding environment, complicating speech emotion recognition.The task also has applications such as diagnosing depression and distress and monitoring mood state for bipolar patients.Source: §1 · p. 1
  • Hand-engineered acoustic features and classifiers have often been designed separately, and perceptually motivated filter banks are not guaranteed to suit emotion classification.LogMel features are popular for DNNs and draw on auditory and physiological evidence about speech perception.Source: §1 · p. 1
  • Emotion recognition requires efficient modelling of long temporal context because emotions are context-dependent and emotion-specific information is embedded in long contexts.LSTM is frequently used for this purpose; parallel CNN layers with different filter widths are proposed to capture diverse contextual information from raw speech.Source: §1 · p. 1
  • The proposed network stacks a multi-temporal CNN on LSTM to capture contextual information at multiple temporal resolutions while modelling long-term context from raw speech.The CNN is designed to complement the LSTM in the jointly trained recognition network.Source: §1 · p. 1

2. Related Work

Prior work explored CNN and LSTM models for raw-speech tasks, while raw-speech emotion models had not matched feature-based results in the cited studies. This paper proposes parallel CNN layers with multiple filter lengths, jointly optimized with classification, to extract multiple temporal resolutions and contextual dependencies.

2. Related Work

  • CNN and LSTM combinations had been evaluated for raw-speech automatic speech recognition, with reported reductions in word error rate and competitive performance against feature-based approaches.Other cited raw-speech DNN work spans environmental sound recognition, speaker identification, and automatic tagging.Source: §2 · pp. 1–2
  • Raw-speech emotion studies had not matched feature-based methods in the cited work, while multi-width CNNs had shown competitive results on hand-engineered Mel Filterbank features.The proposed approach instead applies parallel convolutional layers with multiple filter lengths directly to raw speech and jointly optimizes feature extraction and classification.Source: §2 · p. 2
  • Unlike cited emotion studies using end-to-end CNN–LSTM, TDNN, or single-layer multi-width CNN approaches, this paper uses parallel filter lengths for raw-speech emotion recognition.The feature-extraction block is jointly optimized with the classification block.Source: §2 · p. 2

3. Model

The model has feature-extraction and classification blocks. Parallel convolutional layers extract features at multiple temporal resolutions from raw speech, and a CNN–LSTM–fully connected stack performs classification.

3. Model

  • The model comprises a feature-extraction block and a classification block.Figure 1 is identified as a schematic of the proposed model.Source: §3 · p. 2, Figure 1 · p. 2

3.1. Feature extraction block

  • Parallel convolutional layers with different filter lengths capture short- and long-term interactions from raw speech; max pooling reduces temporal resolution and selects salient features before their outputs are concatenated.The concatenated features carry multiple temporal resolutions into the classification block.Source: §3.1 · p. 2

3.2. Classification Block

  • The classification block stacks a CNN, LSTM, and fully connected layer to capture high-level abstractions, model long-term temporal structure, and learn discriminative representations, respectively.The LSTM receives feature maps, and the fully connected layers transform its outputs for target prediction.Source: §3.2 · p. 2

4. Experimental Setup

Experiments use audio from IEMOCAP and MSP-IMPROV with leave-one-speaker-out evaluation and UAR. Training data are augmented with speed changes, and the jointly optimized model uses three parallel convolutional filter widths followed by CNN–LSTM classification.

4.1. Dataset

  • The experiments use audio recordings from the dyadic-interaction datasets MSP-IMPROV and IEMOCAP.Both datasets contain interactions between actors.Source: §4.1 · p. 2

4.1.1. IEMOCAP

  • IEMOCAP experiments use four emotion classes, merging excitement with happiness to remain consistent with previous studies.The corpus has five sessions, each with two speakers, for ten unique speakers overall.Source: §4.1.1 · p. 2

4.1.2. MSP-IMPROV

  • MSP-IMPROV experiments use its four emotion categories: angry, neutral, sad, and happy.The dataset has six sessions with two speakers per session.Source: §4.1.2 · p. 2

4.2. Data Pre-processing and Augmentation

  • Training utterances were augmented with speed effects at factors 0.9 and 1.1, and non-speech intervals at utterance beginnings and ends were removed.Two copies of each training utterance were created using the approach in.Source: §4.2 · p. 2

4.3. Model Configuration

  • The front end uses three parallel striding convolutional layers with 15 ms, 25 ms, and 100 ms filters, jointly optimized with a CNN–LSTM classification block.Each front-end layer has 40 filters; the classification block includes a 128-cell LSTM and a 1024-unit fully connected layer before softmax.Source: §4.3 · p. 3

5. Experiments and Results

The experiments evaluate speech emotion recognition with leave-one-speaker-out testing and compare the proposed model with SVMs using established acoustic feature sets. Comparisons with prior studies are separated when their data-augmentation procedures differ.

5. Experiments and Results

  • UAR is reported on both datasets using leave-one-speaker-out testing, with one speaker held out for testing and another for validation and early stopping.The remaining speakers' utterances are used for training, and the same augmentation technique is used for fair comparison with.Source: §5 · p. 3
  • SVM baselines use MFCC, LogMel, GeMAPS, and eGeMAPS features, with RBF kernels and validation-based hyperparameter selection.The features are extracted with openSmile, and the SVM experiments use the same augmented data as the proposed model.Source: §5 · p. 3
  • Prior-study comparisons are separated from other results when augmentation differs, limiting direct comparability.The authors compare with those studies without augmentation and mark that group separately in Table 1.Source: §5 · p. 3
  • Table 1 compares UAR (%) across models and the proposed approach on raw speech.Source: Table 1 · p. 3

6. Analysis and Discussion

The analysis finds that the proposed raw-speech model compares favorably with SVMs using established features and reaches results comparable to CNNs trained on hand-engineered features. Ablations examine convolutional-layer count, pooling, classifier composition, and input length.

6.1. Convolutional Layers Analysis

  • Three parallel convolutional layers give the best results among the tested counts, outperforming a single-layer architecture.The tested counts are one, two, three, and four parallel layers; the suitable count should be determined empirically for each problem.Source: §6.1 · p. 3, Table 2 · p. 3

6.2. Pooling Strategies

6.3. Analysing Classification Block

  • The CNN-LSTM-DNN classification block performs best; using DNN layers alone hurts performance.The proposed block assigns high-level abstraction to CNN, long-term temporal modelling to LSTM, and discriminative representation to the fully connected layer.Source: §6.3 · p. 4, Table 4 · p. 4

6.4. Input Length Analysis

  • Six-second utterances give the best UAR on both datasets, although one- or two-second signals incur only a small accuracy loss.Short signals create more segments to merge, while long signals can potentially cause buffer overflow in memory-limited embedded systems.Source: §6.4 · p. 4, Figure 2 · p. 4

6.5. Classification Performance Analysis

  • The raw-speech architecture outperforms SVMs using MFCC, LogMel, GeMAPS, and eGeMAPS, and achieves results comparable to hand-engineered-feature CNNs on both datasets.Without data augmentation, it also outperforms two raw-speech studies; on IEMOCAP it reaches 60.23% UAR versus 61.9% and 61.7% for CNN models using spectrograms.Source: §6.5 · p. 4

7. Conclusions

On IEMOCAP and MSP-IMPROV, the proposed parallel multi-layer CNN stacked on an LSTM performs better than existing raw-waveform methods and comparably to methods using state-of-the-art hand-engineered features.

7. Conclusions

  • On IEMOCAP and MSP-IMPROV, the proposed network achieves better accuracy than existing raw-waveform methods and comparable accuracy to methods using state-of-the-art hand-engineered features.Source: §7 · p. 4
  • The authors attribute the performance to parallel CNN layers with multiple filter lengths capturing both long-term and short-term interactions.Source: §7 · p. 4
  • Future work will investigate improving raw-speech emotion-recognition accuracy and compare runtime and computational complexity against hand-engineered-feature methods.The planned comparison is intended to report the accuracy-complexity trade-off.Source: §7 · p. 4

Related papers

Loading 1904.03833v4…