Source-linked AI summary
Mixture of Disentangled Experts with Missing Modalities for Robust Multimodal Sentiment Analysis
Xiang Li, Xiaoming Zhang, Dezhuang Miao, Xianfu Cheng, Dawei Li, Honggui Han, Zhoujun Li
TL;DR
Real-world missing or corrupted modalities make robust multimodal sentiment representation learning difficult, especially for existing feature-disentangled methods. DERL uses hybrid experts, multi-level collaborative reconstruction, and modality-aware routing to address this problem. Across two benchmarks, it outperforms strong baselines under varied missing-modality conditions, including gains on MOSI under intra-modal missingness.
Problem
Existing feature-disentangled MSA methods struggle to handle heterogeneous information variations under uncertain missingness.
Method
DERL adaptively disentangles private and shared representations, reconstructs degraded features from multiple levels, and fuses modality experts according to their learned contributions.
Results
DERL outperforms feature-disentangled and other strong baselines on two MSA benchmarks under various missing-modality scenarios.
Takeaways & Limitations
DERL provides robust multimodal sentiment representations and importance-aware fusion for incomplete inputs.
Abstract
from arXiv · showhide
Multimodal Sentiment Analysis (MSA) integrates multiple modalities to infer human sentiment, but real-world noise often leads to missing or corrupted data. However, existing feature-disentangled methods struggle to handle the internal variations of heterogeneous information under uncertain missingness, making it difficult to learn effective multimodal representations from degraded modalities. To address this issue, we propose DERL, a Disentangled Expert Representation Learning framework for robust MSA. Specifically, DERL employs hybrid experts to adaptively disentangle multimodal inputs into orthogonal private and shared representation spaces. A multi-level reconstruction strategy is further developed to provide collaborative supervision, enhancing both the expressiveness and robustness of the learned representations. Finally, the disentangled features act as modality experts with distinct roles to generate importance-aware fusion results. Extensive experiments on two MSA benchmarks demonstrate that DERL outperforms state-of-the-art methods under various missing-modality conditions. For instance, our method achieves improvements of 2.47% in Acc-2 and 2.25% in MAE on MOSI under intra-modal missingness.
1. Introduction
DERL addresses robust multimodal sentiment analysis when modalities are missing or degraded by adaptively disentangling representations, reconstructing features collaboratively, and routing modality experts for fusion. Experiments on two benchmarks report stronger performance than existing baselines under varied missing-modality conditions.
- Random missingness disrupts the balance between modality-private and shared information, while static disentanglement and single-level reconstruction struggle with degraded inputs.
- HED adaptively decouples modality-private and shared sentiment features through a mixture of disentangled experts.
- MLCR reconstructs each modality from multiple views, providing collaborative supervision that improves representation quality, completeness, and robustness.
- MRF treats disentangled features as modality experts and learns their contributions for importance-aware multimodal fusion.
- DERL outperforms feature-disentangled and other strong baselines across two MSA benchmarks under various missing-modality scenarios.
2. Related Work
Related MSA methods include complete-input context-based models and robust approaches for incomplete or degraded data. Disentangled learning and mixture-of-experts provide relevant foundations, but uncertain missingness complicates adaptive separation of private and shared information.
- Context-based MSA models assume complete multimodal inputs and learn unified representations through cross-modal interactions.
- Robust noise-aware MSA methods use reconstruction or joint representation learning to address incomplete and degraded multimodal data.
- Disentangled multimodal learning separates modality-specific and shared information using space-based or mutual-information-based approaches.
- Randomly missing modalities disrupt the proportion between private and shared information, making adaptive disentanglement difficult.
- Mixture-of-Experts models specialize through multiple experts, while sparse variants reduce computation by activating only a subset.
3. Methodology
DERL addresses incomplete multimodal sentiment analysis by combining unified encoding with adaptive disentanglement, multi-level reconstruction, and importance-aware fusion. Its components separately model modality-specific and shared information, recover degraded representations, and weight modalities during fusion.
- Multimodal Input and Encoding: DERL first simulates missing modalities and applies unified Transformer encoders to pre-align incomplete and complete multimodal features.Text tokens may be replaced with [UNK], while missing visual and audio frames are replaced with zero vectors; each modality can lose 0%–100% of features independently.
- Hybrid Expert Disentanglement: Hybrid Expert Disentanglement adaptively routes each modality to private and shared experts, separating modality-specific cues from modality-invariant sentiment information.Soft routing networks produce token-wise gating scores, while shared experts use parameters across modalities and private and shared features are encouraged to reduce redundancy.
- Multi-Level Collaborative Reconstruction: Multi-Level Collaborative Reconstruction recovers degraded semantics at input, disentanglement, and joint levels through complementary reconstruction supervision.The mechanism reconstructs complete features, aligns private and shared representations with complete counterparts, and jointly reconstructs encoded complete features.
- Modality Routing and Fusion: Modality Routing and Fusion treats modality features as experts, assigns token-wise weights, and emphasizes stronger modalities while retaining complementary cues.A routing network computes weights before Transformer-based fusion layers capture deeper cross-modal interactions.
- Task Learning: The fused representation is mean-pooled for sentiment prediction, with training combining task regression, disentanglement regularization, and reconstruction losses.The final fusion result is described as robust to imbalanced modalities under uncertain missingness.
4. Experiments
DERL is evaluated on MOSI and MOSEI under intra-modal and inter-modal missingness, with additional ablations, missing-rate, expert-count, confusion-matrix, and efficiency analyses. Across these settings, it shows strong robustness, while results also identify the roles of its components and moderate expert configurations.
- Results on Inter-Modal Missingness: 21.6 percentage points higher average F1 than DMD on MOSI under inter-modal missingness, while MOSEI gains 0.82% over CorrKD.On complete-modality MOSEI evaluation, DERL reaches 85.57% of the best-reported F1.
- Ablation Study: Removing HED or MLCR consistently degrades performance, showing that adaptive expert disentanglement and feature reconstruction contribute under missing modalities.The ablation study covers both intra-modal and inter-modal missing scenarios on MOSI.
- Effect of Missing Rates: Performance first improves and then declines as the predefined missing rate increases, with moderate missingness providing the best trade-off.Low missingness limits learning from incomplete inputs, whereas excessive missingness degrades input quality and meaningful reconstruction.
- Sensitivity of Expert Numbers: MOSI performance is best with balanced moderate expert counts, exemplified by one private and three shared experts.More than four private experts can reduce performance through overfitting, while excessive shared experts introduce redundancy and reduce robustness.
- Efficiency Analysis: DERL maintains a smaller parameter footprint and the fastest training speed among compared methods.Without the text encoder, DERL retains 2.89M core parameters and reduces redundant cross-modal interactions relative to TFR-Net, LNLN, and DMD.
5. Conclusion
The paper concludes that DERL handles heterogeneous information variations under uncertain missingness through adaptive disentanglement, collaborative reconstruction, and importance-aware fusion. Experiments under varied missing conditions support its effectiveness, while predefined missing cases remain a limitation.
- 5. Conclusion: DERL adaptively separates private and shared representations, reconstructs missing semantic cues collaboratively, and performs importance-aware multimodal fusion.Its hybrid expert strategy, multi-level reconstruction, and modality routing and fusion address incomplete inputs.
- 5. Conclusion: DERL’s experiments under various missing conditions demonstrate the effectiveness of the proposed framework.
- 5. Conclusion: DERL focuses on predefined missing cases, leaving more complex real-world missing patterns as a challenging extension.
Appendix A. Feature Extraction
The appendix describes the feature extraction pipelines for text, audio, and visual modalities used with MOSI and MOSEI. It also points to dataset-split statistics reported in Table A.7.
- Text Modality: Text utterances are encoded with bert-base-uncased into 768-dimensional features using sequence length 50 for both datasets.
- Audio Modality: Audio features are extracted with COVAREP, using 5 dimensions for MOSI and 74 for MOSEI.The sequence lengths are 375 for MOSI and 500 for MOSEI.
- Visual Modality: Visual facial action-unit and head-pose features are obtained with Facet, with 20 dimensions for MOSI and 35 for MOSEI.The visual sequence length is 500 for both datasets.
- Dataset Statistics: Table A.7 reports, for each data split, counts of negative, neutral, and positive samples.
Appendix B. Baseline Models
The appendix compares baseline models spanning robust missing-modality methods, disentangled representation learning, language-centered fusion, interaction-efficient architectures, and mixture-of-experts approaches. These baselines use varied mechanisms including transformations, reconstruction, distillation, correlation learning, and modality routing.
- Representation-Learning Baselines: Disentangled and self-supervised baselines separate or preserve shared and modality-specific information through latent decomposition, self-supervision, mutual information, or distillation.The listed methods include MISA, Self-MM, MMIM, and DMD.
- Fusion Baselines: Language-centered and efficient fusion baselines reinforce text or regulate multimodal interactions using visual and acoustic cues or MLP-based communication blocks.CENet, ALMT, and CubeMLP represent these approaches.
- Mixture-of-Experts Baselines: EMOE dynamically adjusts modality contributions with a mixture-of-experts mechanism while maintaining unimodal discriminability through knowledge distillation.
- Missing-Modality Baselines: Missing-modality baselines address incomplete inputs through transformations, reconstruction, Bayesian perturbation, graph modeling, correlation distillation, or language dominance.The listed methods include MCTN, TransM, TFR-Net, SMIL, GCNet, CorrKD, and LNLN.