PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion Recognition From Speech With Recurrent Neural Networks

Vladimir Chernykh, Pavel Prikhodko

arXiv:1701.08071v2cs.CL

TL;DR

Speech conveys emotional cues that can affect intended meaning, and recognizing that emotional background is the paper’s central question. The authors propose a CTC-based approach that accommodates emotional and non-emotional portions of speech; experiments report performance comparable to recent work and a modest gap from human performance.

  • Problem

    Emotional and other conversational cues can influence a phrase’s intended meaning, motivating the question of how to recognize a voice’s emotional background.

    Source: §1.1 · p. 1

  • Method

    The authors propose an emotion-recognition algorithm based on CTC, which sums the probabilities of possible alignments for a labeling.

    Source: §3.2 · p. 7, §5 · p. 14

  • Results

    The proposed method’s results are comparable with state of the art, and the gap between human and model performance is not large.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The similar error structure for humans and the model provides another argument in favor of the model.

    Source: §5 · p. 14

    Two-channel recordings include overlapping voices for about 9% of utterance time, which may lead to undesired results.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

In this paper the task of emotion recognition from speech is considered. Proposed approach uses deep recurrent neural network trained on a sequence of acoustic features calculated over small speech intervals. At the same time special probabilistic-nature CTC loss function allows to consider long utterances containing both emotional and neutral parts. The effectiveness of such an approach is shown in two ways. Firstly, the comparison with recent advances in this field is carried out. Secondly, human performance on the same task is measured. Both criteria show the high quality of the proposed method.

1 Introduction

Speech understanding involves emotional and other conversational cues that can shape intended meaning, while emotion recognition remains difficult because labels are subjective and long speech raises questions about labeling granularity. Earlier systems used handcrafted features and aggregation, whereas later deep-learning approaches moved toward end-to-end processing; the proposed model is reported to match more recent work and support sequences of emotion labels.

1.1 Problem

  • Emotional and other conversational cues can alter a phrase’s intended meaning, making their extraction relevant to human-involved systems.The paper names speaker identity, intonation, emotion and loudness as examples of cues beyond text.Source: §1.1 · pp. 1–2
  • Emotion recognition is complicated by subjective labels, unresolved labeling granularity for long speech, and costly, biased or difficult-to-collect emotion databases.The paper discusses variation in how people understand emotion, how to label continuous speech, and challenges in assembling emotion corpora.Source: §1.1 · p. 2

1.2 Related works

  • Earlier approaches used handcrafted acoustic features with generative models or conventional classifiers, while later work introduced deep networks and recurrent models.The paper identifies simple aggregation and ELMs as drawbacks of two earlier deep-learning approaches.Source: §1.2 · p. 3
  • The first edition aimed to remove intermediate handcrafted components through a fully end-to-end pipeline.The paper also frames its model as addressing drawbacks of the earlier approaches.Source: §1.2 · p. 3
  • The proposed model is on par with more recent approaches and supports sequences of emotion labels, which the authors say other approaches do not support.The paper presents the sequence output as an extension beyond the approaches it compares against.Source: §1.2 · p. 3

2 Data

Experiments use the IEMOCAP database and convert speech into sequences of acoustic features for emotion recognition. The data section describes the recordings, emotion-labeling procedure, and preprocessing choices.

2 Data

  • IEMOCAP was selected for its elaborate acquisition methodology, free academic license, long recordings, and detailed markup.It contains approximately 12 hours of recordings from five male-female actor pairs in dialogue sessions, with scripted and improvised conversations.Source: §2 · p. 4, §2.1 · p. 4

2.1 Database structure

  • Two microphones capture the male and female voices, whose interruptions can overlap for about 9% of utterance time and cause both voices to enter a recording.Assessors evaluated utterances using audio and video; the analysis retains anger, excitement, neutral, and sadness from the ten available options.Source: §2.1 · p. 4
  • An emotion label is assigned only when at least half of the experts agree; about 25% of utterances receive no label.Significantly fewer than half of the remaining utterances have agreement from all experts.Source: §2.1 · p. 4

2.2 Preprocessing

  • Preprocessing divides utterances into 200-millisecond frames with 100-millisecond overlap and computes acoustic features to form the model’s input sequence.The 34 calculated features exclude prosodic and linguistic features; CNN-based feature extraction is proposed as future work.Source: §2.2 · pp. 5–6
  • Frame durations from 30 to 200 milliseconds showed no significant quality difference, while shorter frames increased computation time; the authors chose 200 milliseconds.Thirty milliseconds roughly corresponds to a phoneme and 200 milliseconds to a word in normal spoken English.Source: §2.2 · p. 5

3 Method

The method applies CTC to map frame-feature sequences to emotion-label sequences, accommodating unequal lengths and recordings with emotional and non-emotional spans.

3 Method

  • CTC adds a NULL label, maps each frame to an extended-label sequence, then collapses repeated labels and NULLs into the output.This accommodates recordings that contain stretches without emotion as well as multiple emotions.Source: §3 · p. 6

3.1 Notation

  • The dataset pairs each feature sequence Xi with a true label sequence zi, whose length may differ from the input but cannot exceed it.The model is selected by maximizing the likelihood of the true label sequences given their inputs.Source: §3.1 · pp. 6–7

3.2 CTC approach

  • CTC maps long frame-feature inputs to short emotion-label outputs by combining the probabilities of different alignments for each labeling.Input sequences may be about 100 frames while output sequences are usually no more than 2–4 labels.Source: §3 · p. 6, §3.2 · p. 7
  • An RNN supplies timestep probabilities, while forward-backward dynamic programming computes alignment sums and supports gradient-based training.The paper tests best-path search and beam search to approximate the most probable labeling.Source: §3.2 · pp. 7–8

4 Experiments

The experiments compare the proposed approach with other methods, addressing weak utterance-level supervision and the possibility that long utterances contain neutral as well as emotional spans.

4 Experiments

  • The experiments compare the proposed approach with different baselines, including two methods evaluated in the same setting.The paper directs readers to Sections 4.2 and 4.4 for the algorithms and results.Source: §4 · p. 8
  • A single emotional label for an utterance weakly supervises many frames, although long speech may include frames without emotion.CTC is presented as one way to address this mismatch.Source: §4 · p. 8
  • On IEMOCAP, each utterance has one emotion, so the CTC output sequence has length one and can be treated as an utterance-level label.Source: §4 · p. 8

4.1 Metrics

The evaluation uses overall accuracy for comparability and mean class accuracy to account for class imbalance.

4.1 Metrics

  • Overall accuracy is the fraction of correct answers among all examples and is easy to compare with other papers’ results.Source: §4.1 · p. 9
  • Mean class accuracy averages accuracy calculated within each class across all classes.Source: §4.1 · p. 9
  • Mean class accuracy is introduced to reduce class-imbalance influence, since IEMOCAP’s neutral class is approximately 1.7x times bigger than excitement.Overall accuracy does not account for class imbalance.Source: §4.1 · p. 9

4.2 Baselines

The baselines include framewise classification and a one-label sequence-to-label model, described alongside their assumptions and implementation choices.

4.2.1 Framewise

  • The framewise baseline trains on two loudest frames per utterance, assigns each the utterance emotion, then uses frame labels and majority voting to classify utterances.It uses a Random Forest and assumes two loudest frames represent the whole utterance.Source: §4.2.1 · p. 9
  • Framewise predictions look adequate on short utterances but become sawtooth and unstable on longer ones.Figure 4 plots frame-level emotion probabilities across frames for utterances with known emotions.Source: §4.2.1 · p. 9, Figure 4 · p. 10

4.2.2 One-label

  • The one-label baseline assigns one emotional label to each utterance regardless of length, using sequence-to-label rather than CTC’s sequence-to-sequence paradigm.Its architecture uses stacked Bidirectional LSTM units and dense classification layers with categorical cross-entropy.Source: §4.2.2 · pp. 9–10
  • Because utterance sequences share 34 dimensions but vary in duration, they cannot be efficiently batched directly; padding short sequences and truncating long ones to unified length 78, approximately the 90th percentile, enables efficient mini-batch training.Source: §4.2.2 · p. 10

4.3 CTC

The CTC setup limits output sequences to four valid forms because IEMOCAP assigns one emotion per utterance, then collapses each form to one emotion label. Inputs are padded or truncated to 78 frames, with original lengths retained for decoding.

4.3 CTC

  • Because IEMOCAP assigns one emotion per utterance, the network generates four valid label-sequence types, each collapsed to one “Emo” label during CTC decoding.“Emo” denotes exactly one of anger, excitement, neutral, or sadness.Source: §4.3 · p. 10, Figure 5 · p. 11
  • Input sequences are padded or truncated to 78 frames, while their original lengths are retained so decoding ignores padded positions.Source: §4.3 · pp. 10–11
  • The CTC model uses stacked Bidirectional LSTM units, dense classification layers, and CTC loss.Source: §4.3 · p. 11

4.4 Comparison

The comparison evaluates three approaches with five-fold grouped cross-validation. CTC performs slightly better than One-label and much better than Framewise and Dummy.

4.4 Comparison

  • Grouped cross-validation keeps samples from each speaker group out of either the training or test set, measuring quality independently of speaker.Source: §4.4 · p. 11
  • The 10 IEMOCAP speakers were grouped in recording pairs, yielding a 20% test split rather than a potentially unstable 10% speaker-only split.The reported results average across five folds.Source: §4.4 · p. 11
  • CTC performs slightly better than One-label and much better than Framewise and Dummy in the five-fold grouped cross-validation comparison.The comparison results are reported in Table 1.Source: Table 1 · p. 11, §4.4 · pp. 11–12

4.5 Error structure

The error analysis examines CTC’s predictions against expert labels and expert disagreement. It finds stronger recognition of sadness and substantial overlap between model errors and human divergence.

4.5 Error structure

  • The authors examine error structure after finding a gain of about 3–5% over classical models, smaller than the large gap reported by Graves et al..Source: §4.5 · p. 12
  • Sadness recognition is much higher than recognition of the other emotions in the confusion matrix comparing predictions with expert labels.The authors relate this pattern to prior discussion that audio plays a larger role in sadness recognition.Source: §4.5 · p. 12, Figure 6 · p. 12
  • Figure 6b relates model error percentage by emotion to the number of experts whose answer differed from the final assigned emotion.More red indicates greater error.Source: §4.5 · p. 12, Figure 6 · p. 12
  • When analysis is restricted to utterances on which experts agreed, accuracy is approximately 65%, compared with 54% overall.Source: §4.5 · p. 12
  • CTC’s model errors coincide with human disagreement more often than the approximately 33% expected under random errors.Table 2 reports how frequently model errors match inconsistent expert assessments.Source: §4.5 · p. 13, Table 2 · p. 13

4.6 Human performance

The authors measure human performance to contextualize model errors and the subjective nature of emotion labels. Human accuracy is about 70% on both reported metrics, and the authors argue that quality should be judged using error structure as well as accuracy.

4.6 Human performance

  • Five Russian-speaking lab colleagues assessed random utterance subsets, received feedback after answers, and each utterance was rated by at least two assessors.Eight preliminary examples were excluded from final statistics; the assessors were not professional actors or psychologists.Source: §4.6 · p. 13
  • Both overall accuracy and mean class accuracy for human assessors are about 70%.Source: §4.6 · p. 13
  • Human errors are nonrandom: anger and sadness are hard to confuse, while excitement and happiness are easy to confuse.Source: §4.6 · p. 14
  • The authors argue that model quality requires both sufficiently high accuracy and reasonable error structure resembling human errors; they conclude the CTC model meets both criteria.Source: §4.6 · p. 14

5 Conclusion

The paper proposes CTC-based emotion recognition that accounts for neutral parts within emotional utterances and can output emotion sequences. Experiments find results comparable to the state of the art and a relatively small gap from human performance.

5 Conclusion

  • The proposed CTC-based approach accounts for non-emotional parts within emotional utterances.The authors identify this as one of the method’s two main advantages.Source: §5 · p. 14
  • The approach can predict a sequence of emotions for one utterance.The authors identify sequence prediction as the second main advantage.Source: §5 · p. 14
  • The model is comparable with state-of-the-art results, and its performance gap from humans is not large, with a similar error structure.The authors present the similar human and model error structure as further support for the model.Source: §5 · p. 14
  • Future work proposed by the authors includes replacing handcrafted MFCC features with learnable methods and applying domain adaptation with speech-recognition pretraining and fine-tuning.Source: §5 · p. 14

Appendix A

Appendix A contrasts the one-label and CTC BLSTM architectures, describing their recurrent-layer directions and output processing. The one-label model uses a four-unit softmax for the selected IEMOCAP emotions, while the CTC model adds sequence-wise processing and decoding.

Appendix A

  • Both architectures process input with forward- and backward-order LSTM layers, reverse backward outputs into forward order, and stack the layer outputs.Source: Figure 8 · p. 17, Figure 9 · p. 18
  • In the one-label approach, the last LSTM layer outputs only its last state, followed by a dense layer and a four-unit softmax for the selected IEMOCAP emotions.Source: Appendix A · p. 17
  • The CTC approach applies a shared dense layer to each sequence element and uses a Lambda layer for CTC decoding.An additional data_len input supplies the original input length for more precise decoding.Source: Appendix A · p. 18
Loading 1701.08071v2…