Source-linked AI summary
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
Xu Lin, Ke Wang, Hui Kang, Xinying Wang
TL;DR
The paper addresses weaknesses of unimodal emotion recognition, including noise, information scarcity, and limited contextual cues. It combines multi-feature audio and ResNet50-BiLSTM video encoders with multi-head attention fusion, and reports competitive performance across major metrics on MELD and IEMOCAP. The authors position the framework as a practical approach for capturing complementary speech and facial-expression cues.
Problem
Unimodal emotion recognition is limited by noise contamination, information scarcity, and insufficient contextual relevance.
Method
The framework combines Wav2Vec2, MFCC, and statistical audio features; ResNet50-BiLSTM visual features; and multi-head attention-based feature-level fusion.
Results
Experiments on MELD and IEMOCAP report competitive performance across major metrics and outperformance of several state-of-the-art baselines.
Takeaways & Limitations
The framework captures diverse emotional cues from speech and facial expressions through complementary modality representations.
Takeaways & Limitations
The current work primarily focuses on audio and visual modalities, leaving physiological signals such as EEG, skin conductance, and heart rate for future research.
Abstract
from arXiv · showhide
Multimodal emotion recognition has attracted growing interest due to its importance in human-computer interaction, remote education, and healthcare. This paper proposes a novel multimodal emotion recognition framework that integrates rich audio and visual feature extraction with an attention-based fusion strategy. For audio, we extract three complementary feature types: semantic embeddings from Wav2Vec2, MFCC features, and statistical acoustic descriptors such as pitch, energy, and rhythm. These are aligned and fused via a BiLSTM to capture temporal dependencies. For video, we propose a ResNet50-BiLSTM architecture that combines deep residual learning and sequential modeling to extract expressive spatiotemporal features from facial sequences. To enhance multimodal synergy, we introduce a feature-level fusion mechanism based on multi-head attention, allowing the model to adaptively weigh contributions across modalities. Experiments conducted on the MELD and IEMOCAP datasets demonstrate that our model significantly outperforms baselines in both accuracy and robustness. Furthermore, ablation studies show that the attention-based fusion strategy significantly improves performance in unbalanced data settings. Our findings suggest that the proposed framework effectively captures diverse emotional cues from speech and visual expressions, and offers a practical and generalizable approach for real-world multimodal emotion recognition tasks.
1 Introduction
The paper motivates multimodal emotion recognition because unimodal systems are vulnerable to noise, information scarcity, and missing contextual cues. It proposes complementary audio and video encoders with cross-modal attention, reporting stronger overall and minority-class performance on MELD and IEMOCAP.
- Unimodal emotion recognition is vulnerable to noise contamination, information scarcity, and limited contextual relevance.
- Human emotion perception integrates auditory, visual, and contextual cues, which can correct misleading speech-only interpretations.
- The framework combines a three-feature audio BiLSTM, a ResNet-BiLSTM video encoder, and cross-modal attention fusion.
- Experiments on MELD and IEMOCAP show higher overall accuracy and minority-class performance than state-of-the-art methods.
2 Methods
The method uses complementary audio features and a ResNet50-BiLSTM video encoder, then fuses modality representations with multi-head attention. Audio processing combines semantic, spectral, and statistical cues before temporal modeling.
- The framework has BiLSTM audio, ResNet50-BiLSTM video, and multi-head attention fusion modules.
- Audio processing combines MFCCs, Wav2Vec2 semantic embeddings, and prosodic or statistical descriptors before BiLSTM modeling.
- Wav2Vec2 supplies semantic embeddings, MFCCs describe speech spectral envelopes, and statistical features provide explicit acoustic emotion cues.
BiLSTM-Based Audio Feature Fusion Model
The audio module aligns three feature types to a common representation, combines them using weighted summation, and reshapes the result for BiLSTM processing. Its final representation concatenates forward and backward recurrent outputs.
- The audio module uses Wav2Vec2 embeddings, MFCCs, and statistical audio features with dimensions 1 × 512, 1 × 25, and 1 × 25.
- MFCC and statistical features are projected to the Wav2Vec2 dimension before fusion.
- The three aligned features are fused by weighted summation with α + β + γ = 1.
- The fused vector is reshaped for BiLSTM input, and forward and backward outputs are concatenated into the final audio representation.
2.2 BiLSTM-based Video Emotion Recognition Module
The video module selects representative frames and processes them with a ResNet-50 backbone followed by BiLSTM temporal modeling. This design targets informative facial spatiotemporal features while reducing redundant frames.
- The video module combines a ResNet-50 backbone with a BiLSTM temporal modeling layer.
- An entropy-based algorithm selects a fixed number of representative frames using uniform intervals or content-aware criteria.
- Selected frames are preprocessed and passed through pretrained ResNet-50 for deep feature extraction.
Entropy-Based Frame Selection
The video pipeline extracts spatial features with ResNet-50 and models frame sequences with BiLSTM to produce temporally aggregated video embeddings.
- ResNet-50 feature extraction: ResNet-50 uses residual units and skip connections to facilitate gradient flow and support deeper networks without accuracy degradation.Residual blocks combine convolutional transformations with a skip pathway, while global average pooling aggregates spatial information without additional parameters.
- ResNet-50 feature extraction: The initial convolutional layer extracts low-level image features before max pooling reduces their spatial dimensions.The convolution uses a 7 × 7 kernel with stride 2 and 64 output channels; max pooling selects local maxima.
- Residual blocks: The residual transformation applies three convolutional layers with 1×1, 3×3, and 1×1 kernels, followed by a skip connection.The skip pathway uses a 1 × 1 convolution to align input dimensions before addition.
- Feature aggregation: Global average pooling produces a 1 × 1 × 2048 global feature vector from the resulting feature map.The operation aggregates global spatial information without introducing additional parameters and helps prevent overfitting.
- Temporal modeling: A BiLSTM hierarchically models short- and long-range temporal patterns across ResNet-50 frame features to yield a temporally aggregated video embedding.Sequential modeling establishes contextual relationships across the frame sequence.
2.3 Multimodal Audio-Visual Emotion Recognition
The multimodal model combines audio and video feature extraction modules with feature-level multi-head attention fusion, as illustrated in Fig. 3.
- Overall architecture: The architecture combines two unimodal feature extraction modules with a feature-level fusion strategy based on multi-head attention.The overall architecture is illustrated in Fig. 3.
Feature-Level Fusion Based on Multi-Head Attention Mechanism
The fusion module uses multi-head cross-attention to combine audio and video representations, adaptively weighting modalities and producing a final fused feature vector.
- Fusion strategy: Multi-head attention performs feature-level fusion for multimodal emotion recognition.The mechanism is designed to combine audio and video representations rather than process either modality independently.
- Cross-attention inputs: Video features provide the query, while normalized audio features provide the key and value inputs to cross-attention.The resulting attention output is computed from QV, KA, and VA after LayerNorm processing.
- Feature refinement: The attention output is added to the video query, normalized, processed by a feedforward layer, and combined into the final fused feature vector A∗.This sequence forms an intermediate representation before the final residual update.
- Adaptive fusion: The fusion strategy adaptively calibrates modality weights to address feature redundancy and complementarity.The authors also state that it captures long-range dependencies and contextual relevance while maintaining real-time performance.
- Robustness: The approach is reported to handle data imbalance and improve recognition accuracy for minority emotion classes.The passage characterizes this as enhancing overall system robustness and precision.
- Classification: An SVM serves as the final classifier for mapping high-dimensional fused features to emotion categories.It is selected for comparison with classic SVM-based baselines and for robustness to overfitting on smaller datasets.
Emotion Classification with SVM
The SVM classifier maps fused representations into a high-dimensional space and separates emotion categories with a decision hyperplane designed to generalize across such features.
- Decision boundary: SVM maps input data into a high-dimensional feature space and identifies a hyperplane that best separates emotion classes.The decision boundary is represented by f(x) = w⊤X + b = 0.
- Margin and support vectors: The margin is the distance between the closest samples on either side of the separating hyperplane, called support vectors.The weight vector is perpendicular to the hyperplane, while b is the bias.
- Class constraints: Support vectors satisfy f(x) = 1 for positive classes and f(x) = −1 for negative classes.These values define the class-side constraints around the decision boundary.
- Generalization: The formulation is intended to generalize well in high-dimensional spaces for distinguishing emotional categories from fused features.This supports using SVM as the final classifier for the multimodal representation.
3 Experiments
Experiments evaluate the proposed audio, video, and multimodal models on MELD and IEMOCAP using standard classification metrics. The framework outperforms comparison models, with strong multimodal performance and gains from multi-head attention fusion.
- Experimental Setup: The experiments use synchronized MELD and IEMOCAP data to assess unimodal and multimodal emotion recognition with Accuracy, Precision, Recall, and F1-Score.MELD contains over 13,000 utterances across seven emotion classes, while IEMOCAP provides approximately 12 hours of synchronized multimodal interactions.
- Audio Feature Extraction: 80.3% overall accuracy was achieved by the multi-feature audio model on MELD, with stronger recognition for anger, joy, and neutral than fear and sadness.The audio model was evaluated using an 8:2 training-test split.
- Video Feature Extraction: 71.7% average test accuracy was achieved by ResNet50-BiLSTM, exceeding AlexNet at 68.1% and GoogleNet at 67.3% on MELD.The comparison used the same emotion classifier after facial-frame preprocessing.
- Multimodal Evaluation: 93.7% average accuracy was obtained on the MELD multimodal test set across seven emotion categories, including over 90% accuracy for the large-scale neutral class.The reported performance remained strong despite limited training samples.
- Multimodal Evaluation: On IEMOCAP, the proposed model improved accuracy over EF-LSTM, BERT, and CNN-LSTM by 10.5%, 7.6%, and 7.4%, respectively.Its F1-Score improvements over those baselines were 9.4%, 4.9%, and 4.5%.
- Ablation Study: Compared with models without fusion, multi-head attention improved MELD accuracy by 10.8% and F1-Score by 9.4%.The ablation results attribute the gain to dynamic modality weighting and extraction of complementary information.
4 Conclusions
The framework combines multi-feature audio and visual encoding with attention-based fusion to improve multimodal emotion recognition. It reports enhanced feature integration while identifying physiological signals and newer pretrained or Transformer-based models as future directions.
- The framework integrates deep audio and video feature extraction with attention-based fusion to improve multimodal emotion classification.Its design targets higher-quality unimodal features and stronger synergy between modalities.
- Audio encoding combines Wav2Vec2 semantic embeddings, MFCCs, and statistical acoustic descriptors, while ResNet50-BiLSTM extracts spatiotemporal facial representations.
- Multi-head attention performs feature-level fusion by dynamically weighting modalities, preserving complementarity and reducing information loss.Experiments on MELD and IEMOCAP show competitive performance across major metrics and improvements over several state-of-the-art baselines.
- The current scope is limited to audio and visual modalities, with physiological signals, pretrained multimodal networks, and Transformer-based cross-modal modeling proposed for future improvement.