Source-linked AI summary
Learning Alignment for Multimodal Emotion Recognition from Speech
Haiyang Xu, Hui Zhang, Kun Han, Yun Wang, Yiping Peng, Xiangang Li
TL;DR
Speech emotion recognition must integrate subtle acoustic and textual cues, yet existing multimodal approaches do not fully model their temporal interaction. The paper learns attention-based alignment between speech frames and text words, fuses the aligned features, and feeds them to a sequence model. On IEMOCAP, the proposed approach achieves state-of-the-art performance, while the study notes that its speech-encoder exploration is limited.
Problem
Speech emotion recognition needs multimodal speech and text information, but existing approaches do not adequately learn their fine-grained temporal interaction.
Method
The model uses attention to align speech frames with text words, fuses aligned multimodal features, and applies a recurrent sequence model for emotion classification.
Results
The proposed approach achieves state-of-the-art emotion recognition results on the IEMOCAP dataset.
Takeaways & Limitations
Learning alignment between speech and text is presented as an effective way to build multimodal features for speech emotion recognition.
Takeaways & Limitations
The paper does not focus on exploring speech encoders and reports comparable results across the tested architectures when combined with the multimodal model.
Abstract
from arXiv · showhide
Speech emotion recognition is a challenging problem because human convey emotions in subtle and complex ways. For emotion recognition on human speech, one can either extract emotion related features from audio signals or employ speech recognition techniques to generate text from speech and then apply natural language processing to analyze the sentiment. Further, emotion recognition will be beneficial from using audio-textual multimodal information, it is not trivial to build a system to learn from multimodality. One can build models for two input sources separately and combine them in a decision level, but this method ignores the interaction between speech and text in the temporal domain. In this paper, we propose to use an attention mechanism to learn the alignment between speech frames and text words, aiming to produce more accurate multimodal feature representations. The aligned multimodal features are fed into a sequential model for emotion recognition. We evaluate the approach on the IEMOCAP dataset and the experimental results show the proposed approach achieves the state-of-the-art performance on the dataset.
1. Introduction
Speech emotion recognition matters for natural human–computer interaction but remains difficult because emotion is expressed across modalities and speech-only settings are common. The paper addresses this challenge by aligning speech and text temporally before sequential emotion classification.
- The paper targets emotion recognition for human–computer interaction, where naturally understanding human emotions remains difficult.
- Because emotion is expressed through voice and content, combining speech and text can capture information unavailable to unimodal systems.
- Speech-only applications can use automatic speech recognition to generate text, enabling joint modeling of acoustic and textual emotion cues.
- Existing multimodal models often overlook fine-grained temporal relations between speech frames and text words.
- The proposed approach learns speech–text alignment with attention, fuses aligned word-level features, and models their sequence with a recurrent network.
2. Related Work
Prior speech emotion research progressed from engineered features and conventional classifiers toward neural sequence models, while multimodal work commonly fused separately modeled modalities. The paper distinguishes its approach by using attention to align speech and text sequences rather than only modeling or concatenating them.
- Earlier speech emotion systems used engineered features or statistical representations with classifiers such as GMMs, HMMs, SVMs, and neural networks.
- Deep-learning studies increasingly used recurrent architectures because speech has sequential structure.
- Multimodal emotion models typically extracted audio, visual, or textual features and fused them at feature or decision level.
- Prior speech–text methods modeled the two sequences separately and used direct concatenation, while tensor fusion modeled intra- and inter-modality dynamics.
- The paper claims to be the first to use attention specifically to align speech and text sequences for speech emotion recognition.
3. Algorithm Details
The model processes speech and ASR-generated text through separate encoders, aligns their temporal representations with attention, and fuses them for sequence-based emotion classification.
- 3. Algorithm Details: The architecture contains speech and text encoders followed by an attention-based multimodal fusion network with an LSTM classifier.The speech path directly encodes audio, while the text path uses ASR output and word embeddings.
- 3.1. Speech Encoder: A BiLSTM encodes speech frames, whose hidden states are used for alignment with the text sequence.Speech is framed using 20 ms windows shifted every 10 ms before sequential encoding.
- 3.1. Speech Encoder: Alternative speech encoders, including CNN-LSTM and attention-based LSTM architectures, produced comparable results when combined with the proposed multimodal model.The paper does not focus on systematically exploring speech encoder variants.
- 3.2. Text Encoder: The text encoder converts ASR-generated words into embeddings and uses a BiLSTM to represent each word for multimodal alignment.The approach uses the public Google Cloud Speech API to generate text rather than training an ASR system for the emotion dataset.
- 3.3. Attention Based Alignment: Attention learns soft alignment weights between speech frames and text words, producing aligned speech features for multimodal fusion.The alignment uses hidden states from both BiLSTM encoders; each word receives a weighted speech representation.
- 3.3. Attention Based Alignment: The aligned speech and text states are concatenated and passed through a multimodal BiLSTM before pooling, nonlinear transformation, and softmax classification.The loss uses cross entropy for C-class emotion classification.
4. Evaluations
Experiments on IEMOCAP compare unimodal, concatenation-based multimodal, and alignment-based multimodal emotion recognition models. The proposed attention-alignment approach achieves the strongest reported performance, while oracle text and alignment analyses examine recognition and temporal-alignment effects.
- Main results: The proposed approach achieves the best reported IEMOCAP results on both WA and UA, outperforming other multimodal approaches.Table 1 compares speech-only, text-only, and multimodal models using recognized text.
- Main results: Combining speech with recognized text significantly improves both WA and UA over the corresponding unimodal models.The unimodal baselines are attention-based LSTM models for speech and text.
- Multimodal comparisons: Learned alignment outperforms direct concatenation, supporting fine-grained temporal interaction between speech and text.The concatenation baseline separately encodes each modality, aggregates each sequence, and concatenates the resulting vectors.
- Oracle-text analysis: Oracle text improves the text-only and direct-concatenation methods by around 3%, while the proposed oracle-text model achieves the dataset’s best results.These experiments use provided transcripts and word-level alignment as additional analysis conditions.
- Alignment analysis: The attention-based alignment method outperforms hard alignment using ground-truth word boundaries, indicating emotion-task optimization benefits feature aggregation.Hard alignment averages speech-LSTM hidden states over frames corresponding to each word and concatenates them with text-LSTM states.
5. Conclusions
The paper addresses speech emotion recognition by aligning speech and recognized text with attention before sequential emotion classification. On IEMOCAP, the proposed approach is reported as superior to other approaches and achieves state-of-the-art results.
- 5. Conclusions: The proposed model learns attention-based alignment between speech and recognized text, fuses the modalities, and feeds the fused features into a sequence model.The sequence model performs emotion classification.
- 5. Conclusions: The proposed approach achieves superior emotion-recognition results and state-of-the-art performance on the IEMOCAP dataset.