PaperlayerView exact revision on arXiv

Source-linked AI summary

Speech Emotion Recognition Using Multi-hop Attention Mechanism

Seunghyun Yoon, Seokhyun Byun, Subhadeep Dey, Kyomin Jung

arXiv:1904.10788v2eess.AScs.AIcs.CLcs.LGcs.SD

TL;DR

Because lexical content is often ignored in speech emotion recognition, this paper proposes multi-hop attention to use textual cues to attend to audio segments. On IEMOCAP, the approach outperforms the best baseline in weighted accuracy.

  • Problem

    Lexical content is an important source of emotion information that is usually ignored, and prior multimodal studies did not use attention over audio and text together.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    Multi-hop attention selects relevant textual segments and uses them to attend to audio segments for speech emotion classification.

    Source: §1 · p. 1, §3.2 · p. 2

  • Results

    6.5% (0.718 to 0.765 absolute) improvement over the best baseline in WA is reported on IEMOCAP.

    Source: §5 · p. 4

  • Takeaways & Limitations

    With ASR-processed transcripts, MHA-2-ASR achieves WA 0.73, which the authors say shows reliability when ground-truth transcripts are unavailable.

    Source: §5 · p. 4

Abstract

from arXiv · show

In this paper, we are interested in exploiting textual and acoustic data of an utterance for the speech emotion classification task. The baseline approach models the information from audio and text independently using two deep neural networks (DNNs). The outputs from both the DNNs are then fused for classification. As opposed to using knowledge from both the modalities separately, we propose a framework to exploit acoustic information in tandem with lexical data. The proposed framework uses two bi-directional long short-term memory (BLSTM) for obtaining hidden representations of the utterance. Furthermore, we propose an attention mechanism, referred to as the multi-hop, which is trained to automatically infer the correlation between the modalities. The multi-hop attention first computes the relevant segments of the textual data corresponding to the audio signal. The relevant textual data is then applied to attend parts of the audio signal. To evaluate the performance of the proposed system, experiments are performed in the IEMOCAP dataset. Experimental results show that the proposed technique outperforms the state-of-the-art system by 6.5% relative improvement in terms of weighted accuracy.

1. INTRODUCTION

The paper targets speech emotion recognition using acoustic and textual information together, addressing the usual neglect of lexical content. Its multi-hop attention approach selects relevant text and uses it to attend to audio, and outperforms the reported state-of-the-art system on IEMOCAP.

  • Emotion-recognition methods often use acoustic features, while lexical content is an important information source that is usually ignored.Words such as “gorgeous” and “stunning” can indicate happiness.Source: §1 · p. 1
  • Prior approaches used keywords or bag-of-words for textual information, while DNN approaches fused learned acoustic and textual representations.The paper describes its approach as improving on earlier dual-RNN work by adding attention.Source: §1 · p. 1
  • 6.5% relative improvement in weighted accuracy over the state-of-the-art system on IEMOCAP is reported for the proposed approach.Source: §1 · p. 1
  • Multi-hop attention selects relevant textual parts and applies them to attend to audio segments for classification.The method is trained to exploit textual and acoustic information in tandem.Source: §1 · p. 1

2. RELATED WORK

Prior work spans acoustic models, attention within speech, and multimodal systems combining acoustic features with text. The paper identifies tandem attention over audio and text as an unaddressed aspect of that work.

  • Speech emotion recognition research included classical models, CNNs using spectrograms or audio features, nonlinear-decision models, and attention models for prominent speech segments and temporal dependencies.Source: §2 · pp. 1–2
  • Earlier multimodal studies combined acoustic information with transcripts, but none used attention over audio and text together for contextual understanding.Source: §2 · p. 2
  • Multimodal approaches included identifying emotional phrases and verbal-cue salience, and neural models combining acoustic information with conversation transcripts.Source: §2 · p. 2

3. MODEL

The model first encodes audio and text separately with bidirectional recurrent encoders, then uses multi-hop attention to make each modality inform the selection of relevant segments in the other.

3.1. Bidirectional Recurrent Encoder

  • The audio-BRE encodes MFCC sequences with forward and backward LSTMs, concatenates their hidden states with prosodic features, and predicts emotion from the final representation.A residual connection is added to promote convergence during training.Source: §3.1 · p. 2
  • The text-BRE tokenizes the transcript and feeds it through a BRE to obtain textual hidden representations, using the audio-BRE training objective.Source: §3.1 · p. 2

3.2. Proposed Multi-Hop Attention

  • MHA-1 uses the final audio representation as context to attend to the textual sequence, producing a text representation informed by audio.Source: §3.2 · p. 2
  • MHA-2 uses the audio-informed text representation to attend to audio, and MHA-3 applies another attention hop to text before softmax predicts emotion.The updated textual representation incorporates audio information again.Source: §3.2 · pp. 2–3

4. EXPERIMENTS

Experiments compare audio, text, and multi-hop systems on IEMOCAP, including tests with ASR transcripts. MHA-2 leads the baseline comparison, while error analysis shows class-specific gains alongside persistent confusions.

4.1. Dataset and Experimental Setup

  • The IEMOCAP evaluation used 5,531 utterances across four emotion categories and 10-fold cross-validation, with excitement merged into happiness.Each utterance received the majority annotators’ agreed emotion label.Source: §4.1 · p. 3

4.2. Feature extraction and Implementation details

  • Audio inputs combined 120-dimensional MFCC features with prosodic features, while text inputs were tokenized transcripts, including ASR-derived transcripts for practical tests.Audio segments were capped at 750 and text sequences at 128; training used cross-entropy, Adam, gradient clipping, and dropout.Source: §4.2 · p. 3

4.3. Performance evaluation

  • Performance was measured using weighted and unweighted accuracy averaged over the 10-fold cross-validation experiments.The study used the same dataset and features as prior researchers.Source: §4.3 · p. 3
  • 6.5% relative improvement (0.718 to 0.765) in WA: MHA-2 outperformed the best baseline, MDRE.Table 1 presents the model performance comparisons.Source: §4.3 · p. 3, Table 1 · p. 4
  • 1.6% relative improvement (0.718 to 0.730) in WA: MHA-2-ASR outperformed MDRE despite ASR transcripts with 5.53% WER.Using ASR transcripts reduced WA for text-BRE-ASR and MHA-2-ASR relative to their transcript-based counterparts.Source: §4.3 · p. 4

4.4. Error analysis

  • MHA-2 gained 6%, 20%, 15%, and 13% relative accuracy over text-BRE for angry, happy, sad, and neutral, yet most remaining errors were neutral.Text-BRE improved angry and happy classification over audio-BRE, but sometimes labeled happy instances as sad.Source: §4.4 · p. 4

5. CONCLUSIONS

The paper proposes multi-hop attention to combine acoustic and textual data for speech emotion recognition. On IEMOCAP, MHA-2 outperforms the best baseline in weighted accuracy, and an ASR-transcript version is also evaluated.

5. CONCLUSIONS

  • Multi-hop attention selects relevant textual parts and uses them to attend to audio segments for speech emotion classification.The proposed MHA-2 outperformed the best baseline by 6.5% relative in WA on IEMOCAP; MHA-2-ASR achieved WA 0.73 with ASR-processed transcripts.Source: §5 · p. 4
Loading 1904.10788v2…