Source-linked AI summary
M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, Dinesh Manocha
TL;DR
Emotion recognition systems must combine co-occurring modalities despite sensor noise, missing signals, and uncertain modality reliability. M3ER uses CCA-based modality checking, proxy feature generation, and per-sample multiplicative fusion for face, speech, and text. It reports 82.7% accuracy on IEMOCAP and 89.0% on CMU-MOSEI, a collective 5% absolute improvement over prior methods.
Problem
Multimodal emotion recognition lacks an agreed efficient fusion mechanism, while real-world modalities can be noisy, corrupted, or missing.
Method
M3ER uses CCA to distinguish effective from ineffectual modalities, multiplicative fusion to weight modality reliability per sample, and proxy features for ineffective modalities.
Results
82.7% accuracy on IEMOCAP and 89.0% on CMU-MOSEI yielded a collective 5% absolute accuracy improvement over prior methods.
Takeaways & Limitations
M3ER is reported as robust to sensor noise and supports emotion recognition when some modalities are corrupted or missing.
Takeaways & Limitations
The method is applied to databases with three input modalities and can confuse certain class labels.
Abstract
from arXiv · showhide
We present M3ER, a learning-based method for emotion recognition from multiple input modalities. Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities. M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a per-sample basis. By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise. M3ER also generates proxy features in place of the ineffectual modalities. We demonstrate the efficiency of our network through experimentation on two benchmark datasets, IEMOCAP and CMU-MOSEI. We report a mean accuracy of 82.7% on IEMOCAP and 89.0% on CMU-MOSEI, which, collectively, is an improvement of about 5% over prior work.
1 Introduction
M3ER addresses perceived emotion recognition by combining face, speech, and text while handling modality noise and missing signals. It uses multiplicative fusion, modality checking, and proxy features, with reported gains on two benchmarks.
- Perceived emotion recognition infers perceived rather than actual emotional states from sensor data for applications including human-computer interaction, surveillance, robotics, games, and entertainment.
- Face, speech, and text provide complementary cues, but individual modalities may be corrupted, missing, occluded, or affected by sensor artifacts.
- Additive fusion assumes every modality is potentially useful, whereas multiplicative fusion models each modality’s relative reliability per sample.
- M3ER uses deep neural networks with data-driven multiplicative fusion across face, speech, and text feature vectors.
- Canonical Correlational Analysis distinguishes effective from ineffectual modality signals, while proxy feature vectors replace corrupted or missing modalities.
- 82.7% accuracy on IEMOCAP and 89.0% on CMU-MOSEI represent a collective 5% absolute accuracy improvement over prior methods.
2 Related Work
Prior multimodal emotion-recognition work progressed from classical classifiers to deep architectures and commonly used additive early or late fusion. M3ER applies CCA to identify effective and ineffective modalities before combining them.
- Multimodal Emotion Recognition: Earlier multimodal emotion-recognition studies used Support Vector Machines, linear regression, and logistic regression on smaller datasets.
- Multimodal Emotion Recognition: As datasets grew, researchers developed deep architectures including deep belief networks, convolutional neural networks, hierarchical networks, and relational tensor networks.
- Modality Combination: Prior early- and late-fusion approaches generally relied on additive combinations whose performance depends on assigning relative emphasis across modalities.
- Canonical Correlational Analysis: CCA projects input vectors into a common space by maximizing component-wise correlation and has been used to obtain correlated multimodal features.
- Canonical Correlational Analysis: M3ER uses CCA to check correlations among modalities and distinguish effective from ineffective inputs.
3 M3ER: Our Approach
M3ER checks modality effectiveness, regenerates proxy features for ineffectual inputs, and combines modalities with multiplicative fusion. Its modular design incorporates these components into existing multimodal classification networks.
- Overview: M3ER represents face, text, and speech features and uses a modality check to distinguish effective from ineffectual signals.The check filters ineffectual data before subsequent processing.
- Modality Check Step: Canonical Correlation Analysis computes pairwise modality correlations, which are compared with a threshold to identify ineffectual features.Features are projected into a shared lower-dimensional space before correlation scores are computed.
- Regenerating Proxy Feature Vectors: Proxy feature vectors approximate missing modalities through a learned linear transformation from effective feature vectors.The method can combine proxy vectors generated from multiple effective modalities by taking their mean.
- Multiplicative Modality Fusion: M3ER’s multiplicative fusion explicitly boosts stronger modalities rather than suppressing weaker ones through the original loss formulation.The modified loss avoids the original product-based dependence of each modality’s loss on other modalities.
- Multiplicative Modality Fusion: M3ER is modular: its fusion, modality check, and proxy-vector regeneration can be added to existing multimodal classification networks.The paper demonstrates this integration on IEMOCAP and CMU-MOSEI.
4 Implementation Details
M3ER is evaluated on CMU-MOSEI and compared with prior state-of-the-art methods using multimodal features and a memory-fusion network. The reported setup includes feature extraction, recurrent processing, and fixed training splits and hyperparameters.
- Feature Extraction: M3ER on CMU-MOSEI uses 300-dimensional pre-trained GloVe word embeddings for textual features.
- Evaluation: M3ER is compared with three prior state-of-the-art methods on IEMOCAP and CMU-MOSEI using F1 scores and mean classification accuracies.The reported comparison summarizes increases of around 5–10% in mean accuracy and 1–23% in F1 score.
- Feature Extraction: CMU-MOSEI speech features include Mel-frequency cepstral coefficients, pitch, voiced/unvoiced segmentation, and glottal source parameters.
- Feature Extraction: Facial features combine face embeddings from facial-recognition models, facial action units, and facial landmarks.
- Network and Training: For CMU-MOSEI, M3ER integrates its multiplicative fusion layer into a memory fusion network with single-hidden-layer LSTMs of output dimension 32.The network also uses a 128-dimensional memory variable initialized to zeros.
- Network and Training: 70% of CMU-MOSEI is used for training, 10% for validation, and 20% for testing, with batch size 256 and 500 training epochs.Training uses Adam with learning rate 0.01 on an NVIDIA GeForce GTX 1080 Ti GPU.
5 Experiments and Results
M3ER is evaluated on IEMOCAP and CMU-MOSEI using benchmark comparisons, confusion analyses, modality-weight studies, qualitative examples, and ablations. Results show improved performance and benefits from its modified loss, modality checking, proxy regeneration, and multiplicative fusion.
- Ablation Experiments: The modified multiplicative fusion loss improves both F1 and mean accuracy by 6–7% compared with the original loss.The ablation study evaluates these changes on IEMOCAP and CMU-MOSEI.
- Comparison with SOTA: M3ER improves prior methods by 1–23% in F1 scores and 5–10% in mean accuracies across the reported benchmark comparisons.The evaluation summarizes F1 scores and mean accuracies for IEMOCAP and CMU-MOSEI.
- Confusion Matrix: More than 73% of samples in each emotion class were correctly classified across the two confusion matrices.Some labels had no observed confusions, while happy and angry showed limited confusion in both datasets.
- Interpretability of Multiplicative Layer: Corrupting the face modality reduced its average fusion weight by 12%, with weight redistributed to text and speech.This sample-dependent adjustment is consistent with the multiplicative layer responding to input quality.
- Ablation Experiments: Adding modality checking improves F1 by 2–5% and mean accuracy by 4–5% on datasets with ineffectual modalities.Without proxy feature regeneration, the resulting reduced-modality version remains 2–7% lower in F1 and 5–7% lower in mean accuracy than non-ablated M3ER.
- Ablation Experiments: With all components enabled, M3ER achieves overall F1 scores of 0.761 on IEMOCAP and 0.856 on CMU-MOSEI.The corresponding overall mean accuracies are 78.2% and 85.0%; regenerated proxy vectors are around 0.01% of the respective original-vector L2 norms away.
6 Conclusion, Limitations, and Future Work
M3ER combines multiplicative fusion with modality checking and proxy feature regeneration for three-modality emotion recognition. The authors report limitations in class confusion and binary per-class classification, while proposing broader modalities and more elaborate fusion as future work.
- Conclusion: M3ER uses multiplicative fusion, a modality check step, and regenerated proxy features to handle bad signals from face, speech, and text.The model currently targets databases with three input modalities.
- Limitations: The model often confuses certain class labels and currently performs binary classification per class rather than modeling emotion perception as a probability distribution.The authors identify multi-class classification as future work.
- Future Work: Future work includes more elaborate fusion, more than three modalities, walking styles, and contextual information.These extensions are proposed alongside the move toward multi-class classification.