PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Based Fully Convolutional Network for Speech Emotion Recognition

Yuanyuan Zhang, Jun Du, Zirui Wang, Jianshu Zhang

arXiv:1806.01506v2cs.SDcs.LGeess.AS

TL;DR

Speech emotion recognition is difficult in part because emotional labels are hard to collect. The paper combines a variable-length FCN with attention over spectrogram regions and reports absolute improvements over the best CNN+LSTM results on IEMOCAP.

  • Problem

    Emotional speech labels are difficult to collect, and this study focuses on voice characteristics for recognizing emotion.

    Source: §I · pp. 1–2

  • Method

    An FCN processes variable-size spectrograms without segmentation, while attention weights emotion-relevant time-frequency regions to form an utterance representation.

    Source: §II · p. 2, B. Attention layer · p. 3

  • Results

    The attention-based FCN improves on the best CNN+LSTM results by 1.6% absolute on WA and 1.9% absolute on UA.

    Source: C. Experiment results and analysis · pp. 3–4

  • Takeaways & Limitations

    The authors conclude that visual-recognition CNN architectures and natural-image pretraining can be adapted to speech emotion recognition.

    Source: §I · p. 2, §IV · p. 4

Abstract

from arXiv · show

Speech emotion recognition is a challenging task for three main reasons: 1) human emotion is abstract, which means it is hard to distinguish; 2) in general, human emotion can only be detected in some specific moments during a long utterance; 3) speech data with emotional labeling is usually limited. In this paper, we present a novel attention based fully convolutional network for speech emotion recognition. We employ fully convolutional network as it is able to handle variable-length speech, free of the demand of segmentation to keep critical information not lost. The proposed attention mechanism can make our model be aware of which time-frequency region of speech spectrogram is more emotion-relevant. Considering limited data, the transfer learning is also adapted to improve the accuracy. Especially, it's interesting to observe obvious improvement obtained with natural scene image based pre-trained model. Validated on the publicly available IEMOCAP corpus, the proposed model outperformed the state-of-the-art methods with a weighted accuracy of 70.4% and an unweighted accuracy of 63.9% respectively.

I. INTRODUCTION

The paper focuses on voice characteristics for speech emotion recognition, where emotional labels are hard to collect. It motivates an approach that avoids segmentation and directs attention to emotion-relevant spectrogram regions.

I. INTRODUCTION

  • Speech emotion affects voice characteristics, which this study uses to recognize underlying emotions.The introduction also notes that emotion affects linguistic content, but this study focuses on voice characteristics.Source: §I · p. 1
  • Traditional systems aggregate frame-level acoustic features into utterance vectors before classification.Features may include pitch, voicing probability, and energy; statistical functions such as mean and maximum summarize them.Source: §I · p. 1
  • CNN+LSTM systems segment or pad spectrograms, assigning each sub-utterance the whole-utterance label and averaging predictions at test time.The authors argue that this assumption is unreasonable and segmentation can disrupt speech continuity.Source: §I · p. 1
  • The proposed FCN handles variable-length speech without segmentation, while attention targets emotion-relevant time-frequency regions rather than time alone as in.The introduction also motivates transfer learning because emotional speech labels are difficult to collect and reports improvement with natural-scene-image pretraining.Source: §I · pp. 1–2

II. THE PROPOSED ARCHITECTURE

The proposed model combines an FCN with attention over spectrogram regions. It accepts variable-size spectrograms without segmentation or fixed-shape padding, and its components are jointly optimized.

II. THE PROPOSED ARCHITECTURE

  • Figure 2 presents the overall architecture of the attention-based FCN.Source: Figure 2 · p. 2
  • The FCN accepts variable-size spectrograms without segmenting them or padding them to a fixed shape.The authors state that this avoids losing critical information through segmentation.Source: §II · p. 2
  • The FCN encodes the spectrogram, and attention guides later layers toward specific time-frequency regions.All components of the system can be optimized jointly.Source: §II · p. 2

A. Fully convolutional network

The FCN encoder is built from CNN components and represents its output as a variable-length grid of region vectors. The authors adapt AlexNet by removing its fully connected layers.

A. Fully convolutional network

  • CNNs use convolution, pooling, and activation layers; shared local filters detect features across locations, while pooling removes noise and extracts robust features.The paper contrasts these learned representations with traditional handcrafted feature extraction.Source: A. Fully convolutional network · p. 2
  • The authors convert AlexNet into an FCN by removing its fully connected layers; VGGNet yields lower accuracy on this task with limited training data.AlexNet's first two convolutional layers use local response normalization, and all convolutional layers are followed by ReLU.Source: A. Fully convolutional network · p. 2
  • The encoder output is an F × T × C array, with F and T indexing frequency and time and C denoting channel size.It can be viewed as a variable-length grid of L = F × T elements, each a C-dimensional vector for a speech-spectrogram region.Source: A. Fully convolutional network · p. 2

B. Attention layer

The attention layer weights encoder regions according to their relevance to utterance emotion and aggregates them into an utterance emotion vector. A scale factor controls how uniform those weights are.

B. Attention layer

  • Attention weights encoder-region vectors unequally before aggregating them into an utterance emotion vector.The motivation is that not all time-frequency units contribute equally to the emotion of the utterance.Source: B. Attention layer · p. 3
  • An MLP with tanh transforms each annotation; its inner product with a learnable vector gives an importance score, which softmax normalizes into a weight.Source: B. Attention layer · p. 3
  • The scale factor λ controls weight uniformity: λ = 0 gives uniform weights, λ = 1 gives ordinary softmax, and the study uses λ = 0.3.The weighted sum of annotations forms the utterance emotion vector.Source: B. Attention layer · p. 3

A. Database and feature extraction

The experiments use IEMOCAP improvised speech in four emotion categories, with five-fold validation and raw spectrogram input. The spectrogram uses 40-msec windows, differing from the 20-msec setting in.

A. Database and feature extraction

B. Evaluation metric

Because IEMOCAP has imbalanced emotion classes, the evaluation reports both overall accuracy and the average accuracy across classes.

B. Evaluation metric

C. Experiment results and analysis

The attention-based FCN outperforms the best reported CNN+LSTM results on both accuracy measures. Its 2D attention maps assign different weights across time and frequency, with low weights on non-speech frames and high-frequency regions.

C. Experiment results and analysis

  • The experiments compare AlexNet and VGGNet-16 using fixed-length split-or-padded utterances and random or ImageNet-pretrained initialization.At test time, posterior probabilities are averaged across sub-utterances.Source: C. Experiment results and analysis · p. 3
  • 1.6% absolute improvement on WA and 1.9% on UA over the best results of CNN+LSTM Model1 and Model2.The comparison is with the best results across both models; the FCN is trained in one step.Source: C. Experiment results and analysis · pp. 3–4, Table II · p. 4
  • The 2D attention maps assign low weights to non-speech frames and different weights to time-frequency regions according to their emotion relevance.Figure 4 pairs each spectrogram location with its attention weight; brighter colors indicate larger weights.Source: C. Experiment results and analysis · p. 4, Figure 4 · p. 4
  • Attention weights are small in high-frequency areas and decrease from low to high frequency, while the maps scan both time and frequency.Source: C. Experiment results and analysis · p. 4

IV. CONCLUSIONS

The paper concludes that visual-recognition CNN architectures can be adapted to speech emotion recognition and that natural-image pretraining transfers to speech. Its attention-based FCN handles variable-length utterances and exceeds state-of-the-art accuracy on IEMOCAP.

IV. CONCLUSIONS

  • CNN architectures designed for visual recognition can be directly adapted for speech emotion recognition.Source: §IV · p. 4
  • The authors report that transfer learning builds a bridge between natural images and speech signals.Source: §IV · p. 4
  • The attention-based FCN handles variable-length utterances, focuses on emotionally salient spectrogram regions, and achieves beyond-state-of-the-art accuracy on IEMOCAP.Source: §IV · p. 4
Loading 1806.01506v2…