Source-linked AI summary
CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
Hao Sun, Hongyi Wang, Jiaqing Liu, Yen-Wei Chen, Lanfen Lin
TL;DR
Multimodal sentiment and depression estimation require effective fusion of textual, acoustic, and visual information, while prior attention-based approaches can be computationally costly. CubeMLP addresses this with three MLP units that mix features across sequence, modality, and channel axes. Experiments on sentiment and depression datasets report competitive or state-of-the-art performance with low computational burden.
Problem
The paper addresses multimodal mental-state estimation and the need for effective feature fusion across modalities amid the computational cost of attention-based approaches.
Method
CubeMLP processes multimodal features with three MLP units that mix them across sequence, modality, and channel axes before classification.
Results
CubeMLP achieves competitive sentiment-analysis results and strong depression-detection progress, with reported MAEs of 0.770 on CMU-MOSI, 0.529 on CMU-MOSEI, and 4.37 on AVEC2019, plus a CCC of 0.583 on AVEC2019.
Takeaways & Limitations
The results support CubeMLP as an efficient MLP-based framework for multimodal sentiment analysis and depression detection.
Abstract
from arXiv · showhide
Multimodal sentiment analysis and depression estimation are two important research topics that aim to predict human mental states using multimodal data. Previous research has focused on developing effective fusion strategies for exchanging and integrating mind-related information from different modalities. Some MLP-based techniques have recently achieved considerable success in a variety of computer vision tasks. Inspired by this, we explore multimodal approaches with a feature-mixing perspective in this study. To this end, we introduce CubeMLP, a multimodal feature processing framework based entirely on MLP. CubeMLP consists of three independent MLP units, each of which has two affine transformations. CubeMLP accepts all relevant modality features as input and mixes them across three axes. After extracting the characteristics using CubeMLP, the mixed multimodal features are flattened for task predictions. Our experiments are conducted on sentiment analysis datasets: CMU-MOSI and CMU-MOSEI, and depression estimation dataset: AVEC2019. The results show that CubeMLP can achieve state-of-the-art performance with a much lower computing cost.
1 INTRODUCTION
The paper frames multimodal mental-state estimation as a feature-mixing problem and proposes CubeMLP to mix multimodal features across sequence, modality, and channel axes with reduced computational cost.
- Motivation: Multimodal data combines textual, acoustic, and visual information to estimate sentiment tendencies and depression levels.Features from each modality can be represented as two-dimensional matrices indexed by sequence length and feature-channel size.
- Prior Work: Prior fusion methods exchange information mainly through sequence-wise, channel-wise, or attention-based interactions between modalities.TFN and coattention methods mix features along selected axes, while Transformer approaches use self-attention for modality interactions.
- Motivation: Transformers have substantial self-attention memory requirements, motivating entirely MLP-based structures that can reduce computational costs while maintaining high performance.MLP-Mixer and ResMLP replace self-attention with MLPs in computer vision tasks.
- Proposed Method: CubeMLP integrates modality features into a tensor and applies three independent MLP units for sequential-mixing, modality-mixing, and channel-mixing.The mixed features are flattened and fed to a classifier, while each MLP unit contains two fully connected affine transformations.
- Contributions: Experiments on sentiment analysis and depression detection show competitive or favorable performance against state-of-the-art approaches while reducing computational burden.The study evaluates CMU-MOSI, CMU-MOSEI, and AVEC2019 across the two mental-state estimation tasks.
2 RELATED WORKS
Related work covers multimodal sentiment and depression detection methods, MLP-based vision models, and the transfer of axis-wise MLP mixing to multimodal features.
- Multimodal Sentiment Analysis: Multimodal sentiment analysis predicts sentiment tendencies from facial expressions, acoustic tone, and spoken words in each utterance.Earlier methods fused modalities sequentially or calculated cross-modal correlations and coattention.
- Multimodal Depression Detection: Depression detection requires longer time sequences because depression is treated as a persistent long-term human characteristic.Prior approaches used PCA and SVM, CNNs, attention networks, Transformers, and hybrid self-attention and convolutional structures.
- MLP-based Models: MLP-based vision models replace self-attention with MLPs because Transformer-based processing retains a substantial computational burden.Examples include MLP-Mixer, ResMLP, and Hire-MLP.
- MLP-based Models: CubeMLP transfers MLP-based processing to multimodal features by mixing across sequential, modality, and channel axes.The framework adds an MLP to comprehensively mix the three axes of multimodal features.
- MLP-based Models: The CubeMLP overview feeds extracted features into stacked blocks, mixes them, and flattens the resulting features for prediction.The MLP-unit structure includes layer normalization and a residual shortcut.
3 PROPOSED METHODS
CubeMLP processes multimodal utterance features by mixing them independently across sequence, modality, and channel axes with three MLP units, then flattens the result for prediction. Its units use affine transformations, nonlinear activation, layer normalization, and shared axis-specific parameters.
- 3 PROPOSED METHODS: CubeMLP receives textual, acoustic, and visual features and mixes them separately along sequential, modality, and channel axes.The features are combined into a multimodal tensor before processing, and the mixed representation is later used for sentiment or depression prediction.
- 3 PROPOSED METHODS: The three MLP units transform the tensor from L×M×D to L′×M′×D′ through sequential-, modality-, and channel-mixing operations.The first unit acts on L, the second on M, and the third on D; each unit contains two fully connected layers and a nonlinear activation.
- 3 PROPOSED METHODS: Each axis-specific MLP applies learned affine transformations with parameters shared across the vectors processed on that axis.For sequential mixing, W_L and B_L are shared by all modality-channel vectors; layer normalization and nonlinear activation are also used.
- 3 PROPOSED METHODS: After three-axis mixing, the resulting multimodal tensor is flattened and passed to a classifier that predicts sentiment tendency or depression level.The classifier maps the flattened representation to a scalar prediction.
- 3 PROPOSED METHODS: Training uses MAE for sentiment regression and CCC loss for depression regression.MAE emphasizes small errors in subtle sentiment predictions, while CCC is used for depression estimation because it accounts for correlation and accuracy and is unbiased by scale and location changes.
4.1 Datasets
The experiments evaluate CubeMLP on two multimodal sentiment datasets and one multimodal depression dataset. These datasets provide utterance-level audiovisual or multimodal recordings with task-specific annotations and splits.
- 4.1 Datasets: The study evaluates sentiment analysis on CMU-MOSI and CMU-MOSEI and depression detection on AVEC2019 DDS.The selected tasks estimate related mind states from multimodal data.
- 4.1 Datasets: CMU-MOSI contains Internet-collected utterance videos annotated with sentiment tendencies from −3 to 3.It provides 1283 training, 229 validation, and 686 test utterances.
- 4.1 Datasets: CMU-MOSEI enlarges CMU-MOSI while retaining the same annotations.Its splits contain 16315 training, 1817 validation, and 4654 testing utterances.
- 4.1 Datasets: AVEC2019 DDS consists of audiovisual recordings from patients’ clinical interviews conducted by a virtual agent.Its modalities include multiple feature types, such as MFCC, eGeMaps, and deep visual or acoustic features.
4.2 Experimental Setup
The experimental setup standardizes multimodal feature dimensions and sequence lengths while using different sequence lengths for sentiment analysis and depression estimation.
- 4.2 Experimental Setup: Sequence length L is set to 100 for sentiment analysis and 1000 for depression estimation.Variable-length samples are zero-padded or truncated to match the selected length.
- 4.2 Experimental Setup: Each modality feature uses channel dimension D=128, and the number of modalities is fixed at M=3.The three modalities are textual, acoustic, and visual; reduced dimensions L′, M′, and D′ are explored separately.
4.3 Evaluation Metrics
The study evaluates sentiment regression and classification with task-specific standard metrics, while depression detection uses CCC and MAE.
- 4.3 Evaluation Metrics: Sentiment regression is evaluated using MAE and Pearson correlation, Corr.The sentiment tendency can also be converted into binary or 7-class classification, evaluated with accuracy and F1-score.
- 4.3 Evaluation Metrics: AVEC2019 depression detection is evaluated with CCC and MAE.These metrics have been used in previous depression detection studies.
- 4.3 Evaluation Metrics: CCC ranges from −1 to 1, where −1 denotes total negative correlation and 1 denotes perfect positive correlation.The metric’s stated range provides the interpretation of its correlation values.
5 RESULTS & ANALYSIS
CubeMLP achieves competitive multimodal sentiment and depression-estimation results while using an all-MLP feature-mixing design. Analyses show that mixing across sequence, modality, and channel axes improves performance, maps modalities toward similar representations, and reduces computational cost.
- Experimental Results: CubeMLP achieves a MAE of 0.770 on CMU-MOSI and 0.529 on CMU-MOSEI, competitive with state-of-the-art approaches.
- Experimental Results: On AVEC2019 DDS, CubeMLP achieves a CCC of 0.583 and an MAE of 4.37.
- Ablation Study: Sequential-mixing and channel-mixing contribute more than modality-mixing, while combining all three MLP units produces the best CMU-MOSI performance.
- Dimension Exploration: Performance decreases when any output dimension is less than 3, whereas dimensions >= 3 yield smaller fluctuations and lower-cost tensor conversion with little information loss.The paper attributes this behavior to three modalities and describes the tensor rank as 3.
- Feature Visualization: Across CubeMLP blocks, modality features become more similar and semantically condensed as feature sizes shrink and deeper processing proceeds.The first block retains noticeable modality differences, while later blocks produce increasingly similar representations.
- Computational Complexity Comparison: CubeMLP improves performance while using less computational memory than other state-of-the-art multimodal fusion techniques.
6 CONCLUSION
CubeMLP treats multimodal fusion as feature mixing across all axes, achieving state-of-the-art performance for sentiment analysis and depression detection with low computational burden.
- CubeMLP unifies multimodal feature processing by mixing features across all axes.The framework reframes multimodal fusion as feature mixing.
- CubeMLP achieves state-of-the-art performance for sentiment analysis and depression detection.
- CubeMLP maintains a low computational burden while processing multimodal features.
- The paper uses component comparisons, ablation studies, and visual analysis to examine CubeMLP and its multimodal processing capabilities.