Source-linked AI summary
Multimodal Adaptive Expert Selection with Text Routing and Ordinal Prototype Optimization for Sentiment Analysis
Xiaode Chen, Jiakang Yu, Hongtao Deng, Huina Qu, Xun Zhu, Yinxia Lou
TL;DR
Existing MSA methods are limited by static computation and contrastive objectives that overlook sentiment ordinality. MAESTRO uses text-guided expert routing and ordinal-aware prototype learning, achieving state-of-the-art benchmark performance with interpretable dynamic routing.
Problem
Existing disentanglement methods use static computation graphs and generic contrastive objectives that do not adequately adapt to semantic complexity or preserve sentiment-intensity order.
Method
MAESTRO dynamically routes audio-visual experts using textual context and applies distance-based penalties in O-PCL to structure representations by ordinal sentiment intensity.
Results
MAESTRO achieves state-of-the-art performance on CMU-MOSI and CMU-MOSEI, outperforming strong baselines across most key metrics.
Takeaways & Limitations
Qualitative analysis indicates that dynamic routing shifts attention to specific non-verbal signals during semantic conflicts such as sarcasm.
Abstract
from arXiv · showhide
Multimodal Sentiment Analysis (MSA) is a fundamental component of affective computing that aims to decipher complex emotional states by integrating verbal content with non-verbal cues including vocal intonation and facial micro-expressions. While recent disentanglement-based approaches have advanced the field, their potential is hindered by two methodological challenges. First, static computation graphs process all samples indiscriminately regardless of semantic complexity, which leads to suboptimal representation for diverse emotional expressions and contextual scenarios. Second, generic contrastive objectives often neglect the intrinsic ordinal hierarchy of sentiment intensities. To systematically address these limitations, we introduce Multimodal Adaptive Expert Selection with Text Routing and Ordinal prototype optimization (MAESTRO), a novel framework designed to dynamically orchestrate and refine multimodal representations. Drawing inspiration from an orchestra conductor, we design a Text-Guided Hybrid Mixture-of-Experts (MoE) mechanism. Unlike static fusion, this module utilizes linguistic context as a routing signal to dynamically activate specific audio-visual experts, thereby resolving cross-modal ambiguity through adaptive feature enhancement. Furthermore, to capture fine-grained sentiment gradations, we propose an Ordinal-aware Prototype Contrastive Learning (O-PCL). By incorporating distance-based penalties into the prototype learning objective, O-PCL enforces a structured latent space that preserves the natural order of emotion. Extensive experiments on the CMU-MOSI and CMU-MOSEI benchmarks demonstrate that MAESTRO achieves state-of-the-art performance, and qualitative analysis further confirms the interpretability of our dynamic routing paradigm.
1 Introduction
MAESTRO addresses static multimodal processing and ordinally unaware sentiment objectives with text-guided expert routing and distance-aware prototype learning.
- Limitations: Existing disentanglement frameworks process every sample through identical fixed computation graphs, limiting adaptation to semantic complexity and cross-modal variation.Explicit sentiment may need little verification, whereas ambiguous sarcasm may require specialized acoustic or visual modules.
- Limitations: Generic contrastive objectives treat sentiment negatives uniformly, ignoring that sentiment intensities have an intrinsic order and that distant label errors are more severe.The latent space therefore fails to represent semantic distances between sentiment levels.
- Proposed Framework: MAESTRO uses language as a conductor to dynamically gate and weight audio-visual experts, replacing static fusion with sample-specific computation.The Text-Guided Hybrid MoE is designed to adapt non-verbal processing to textual context.
- Proposed Framework: O-PCL adds distance-based penalties that push samples farther from emotionally distant prototypes, aligning embeddings with the ordinal hierarchy of sentiment labels.This objective targets fine-grained sentiment gradations rather than treating all negative prototypes identically.
- Results: MAESTRO achieves state-of-the-art performance across most key metrics on CMU-MOSI and CMU-MOSEI.The authors report substantial improvements over strong baselines.
2 Related Work
Related work progresses from fixed multimodal fusion and disentanglement toward dynamic architectures, while contrastive learning still often overlooks sentiment ordinality.
- Multimodal Representation Learning: Early multimodal methods combine unimodal features through tensor fusion or cross-modal Transformers, whereas later methods disentangle modality-invariant and modality-specific representations.TFN and MulT model interactions directly; MISA and DLF separate representation subspaces.
- Dynamic Multimodal Modeling: Static computation graphs apply identical encoders and fusion layers to every sample, making context-dependent cues such as sarcasm difficult to resolve.The limitation is especially relevant when noise levels and semantic ambiguities vary across instances.
- Dynamic Multimodal Modeling: Mixture-of-Experts architectures offer instance-adaptive computation by activating only a sparse subset of experts for each input.MoE has gained traction in natural language processing and computer vision, but remains underexplored in MSA.
- Dynamic Multimodal Modeling: Existing multimodal MoE-style methods generally condition expert selection on multimodal representations rather than using linguistic context as a global routing signal.MAESTRO distinguishes itself through a Text-Guided Router that orchestrates visual and acoustic experts.
- Ordinal Contrastive Learning: Standard contrastive learning treats sentiment categorically, overlooking ordinal differences such as the semantic gap between +1 and +2.Ordinal regression exists, but few approaches integrate it effectively with contrastive learning.
3.1 Overall Architecture
MAESTRO forms a five-stage multimodal pipeline that disentangles features, enhances audio-visual representations through text-conditioned routing, fuses them, and predicts sentiment.
- Overall Pipeline: The pipeline extracts high-level representations, disentangles modality-specific and common subspaces, enhances non-verbal features, performs multimodal fusion, and predicts sentiment.Text-guided enhancement conditions routing on the text-specific feature.
- Feature Extraction: Pre-extracted text, audio, and visual inputs are projected into high-level representations using modality-specific encoders.Text uses BERT, while acoustic and visual sequences use convolutional projections followed by Transformer-based networks.
- Latent Representation: Feature disentanglement separates each modality into modality-specific and modality-invariant common representations, regularized by orthogonality, reconstruction, and O-PCL.These constraints target purity and complementarity of the latent spaces.
- Text-Guided Enhancement: The Maestro block uses a Text-Guided Hybrid MoE to orchestrate a shared expert and expert pools for audio and video enhancement.The resulting refined representations are sensitive to linguistic context before final fusion and sentiment prediction.
3.2 Feature Disentanglement
MAESTRO disentangles shared and modality-specific information with geometric constraints, then uses O-PCL to organize common representations according to sentiment intensity.
- 3.2 Feature Disentanglement: The disentanglement paradigm separates unique modal dynamics from shared sentiment semantics into independent specific and common subspaces.This decomposition addresses redundancy and noise caused by directly fusing heterogeneous features.
- 3.2 Feature Disentanglement: Reconstruction regularization keeps the decoded concatenated representation close to the input, preserving information during disentanglement.The decoder maps the concatenated representation back to feature space and minimizes squared ℓ2 reconstruction error.
- 3.2 Feature Disentanglement: Orthogonality regularization minimizes correlation between subspaces, encouraging linear independence and isolating modality-specific noise.The constraint is expressed through the Frobenius norm of the cross-subspace correlation.
- 3.2.2 Latent Space Regularization: O-PCL treats adjacent sentiment levels as closer than distant ones and injects an explicit distance-dependent penalty into prototype contrast.This addresses conventional prototype contrastive learning's uniform treatment of negative classes.
- 3.2.2 Latent Space Regularization: Ordinal labels are obtained by quantizing continuous sentiment scores into K ordered levels, with larger labels indicating more positive sentiment.The maximum ordinal gap is K−1, consistent with Acc-5 and Acc-7 evaluation.
- 3.2.2 Latent Space Regularization: Increasing logits for ordinal-distant negative prototypes raises confusion loss and pushes representations away from emotionally distant prototypes.The resulting latent space is structured to respect sentiment hierarchy.
3.3 Text-Guided Hybrid MoE
The Text-Guided Hybrid MoE uses linguistic context to route and modulate audio-visual experts dynamically, replacing modality-isolated self-routing with context-aware computation. Sparse Top-K selection, dual gating, hybrid aggregation, and load balancing support specialized yet stable expert processing.
- Dynamic routing: Textual semantics act as a global conductor signal that dynamically orchestrates audio and visual experts for context-dependent non-verbal interpretation.The router conditions non-text expert selection on global linguistic context and local modality dynamics.
- Dynamic routing: The cross-modal router replaces self-routing by selecting non-text experts using both linguistic context and local modality dynamics.Standard self-routing conditions selection only on the input modality itself.
- Sparse expert selection: Top-K routing activates only the K experts with the largest gating scores, producing sparse sample-specific computation.The sparse routing weight controls each expert’s contribution.
- Sparse expert selection: Sparse selection encourages expert specialization and filters irrelevant signals, while auxiliary load balancing mitigates expert collapse.The hybrid design balances dynamic adaptability with static robustness.
- Context-aware dual gating: Dual gating combines sparse expert routing with an internal channel gate that modulates expert outputs using text context.The sparse routing weight controls expert contribution, while the channel gate provides finer-grained modulation.
- Hybrid aggregation: Hybrid aggregation combines a shared expert for modality-invariant patterns with routed experts that provide text-aligned refinements.Non-selected experts contribute zero through sparse routing weights.
3.4 Multimodal Fusion and Prediction
After text-guided enhancement, MAESTRO retains the text representation, pools common modality-invariant features, concatenates them with specificity-enhanced representations, and predicts sentiment with a regularized regression head.
- Feature preparation: The Maestro Block produces refined modality-specific features that incorporate linguistically aligned non-verbal cues while retaining the original text-specific feature.These representations are used in the subsequent fusion and prediction formulas.
- Multimodal fusion: Average pooling condenses shared semantic information across the three modalities into a representative common feature.The operation aggregates modality-invariant features across modalities.
- Multimodal fusion: MAESTRO constructs Z by concatenating the consolidated common feature with specificity-enhanced representations.Z contains consistent sentiment semantics and context-sensitive unimodal dynamics.
- Prediction: A two-layer multilayer perceptron with nonlinear activations and dropout maps the fused vector Z to predicted sentiment intensity.Dropout regularization is used to prevent overfitting.
3.5 Optimization Objectives
MAESTRO is trained end to end with a weighted combination of task and regularization terms. Sentiment regression uses mean absolute error, while an auxiliary load-balancing objective discourages skewed expert utilization.
- Overall objective: The total objective is a weighted combination of the task loss and multiple regularization terms optimized end to end.The coefficients λ1, λ2, λ3, and λ4 balance the components.
- Task objective: Sentiment prediction on benchmarks such as CMU-MOSI is formulated as regression using mean absolute error as the task objective.The loss compares ground-truth and predicted sentiment scores over a batch.
- Load balancing: Expert collapse occurs when routing repeatedly activates a small subset of experts and leaves others underutilized.The paper identifies this as a key challenge in sparse MoE training.
- Load balancing: An auxiliary load-balancing loss encourages balanced expert utilization and is optimized with the main objective to discourage skewed routing.For each expert, the formulation uses routed-sample fractions and average gating probabilities.
- Load balancing: The auxiliary term complements Top-K sparse routing by preventing a few experts from dominating traffic while preserving informative per-sample computation.It balances expert utilization without removing the router’s selective behavior.
4 EXPERIMENTS
Experiments on CMU-MOSI and CMU-MOSEI show that MAESTRO outperforms strong baselines, while ablations and routing visualizations examine its dynamic selection, text guidance, disentanglement, and ordinal optimization.
- Results: CMU-MOSI results show MAESTRO achieving Acc-2 of 87.20%, MAE of 0.689, and Corr of 0.807 against strong baselines.The comparison highlights ConFEDE as the strongest competitor and attributes improved regression precision to ordinal-aware optimization.
- Results: On CMU-MOSEI, MAESTRO surpasses all baselines on classification tasks with Acc-2 of 85.91% and F1 of 85.88%.The reported advantage is associated with text-guided dynamic routing on the larger, more diverse dataset.
- Ablation Study: Removing the Maestro Block produces the lowest ablation performance, with Acc-7 dropping to 45.32% compared with dynamic selection.The ablation contrasts static feature concatenation with routing that filters irrelevant information for each sample.
- Ablation Study: Replacing the Text-Guided Router lowers Acc-2 to 85.42% from 87.20%, supporting linguistic guidance for resolving ambiguous non-verbal cues.The variant retains MoE but selects experts from audio and visual features without textual context.
- Ablation Study: Removing disentanglement lowers Acc-2 to 85.15%, while removing O-PCL increases MAE from 0.689 to 0.724.The results associate purified modality representations with routing reliability and ordinal constraints with regression precision.
- Further Analysis: In a sarcastic CMU-MOSI case, the router assigns weights of 0.78 and 0.65 to Visual Expert 2 and Acoustic Expert 4, respectively.The sparse activation redirects attention toward facial expressions and vocal tonality when literal text conflicts with negative non-verbal signals.
5 Conclusion
MAESTRO replaces static multimodal processing with text-guided routing that dynamically orchestrates audio-visual experts, while O-PCL structures representations around sentiment intensity order. Experiments on CMU-MOSI and CMU-MOSEI report state-of-the-art performance and interpretable routing under semantic conflicts such as sarcasm.
- MAESTRO shifts Multimodal Sentiment Analysis from static computation graphs to dynamic, context-aware routing.
- Its Text-Guided Router uses linguistic semantics to activate only the most discriminative audio-visual cues for each sample.
- O-PCL incorporates distance-based penalties into prototype learning to align the latent space with the intrinsic order of sentiment intensities.
- MAESTRO establishes state-of-the-art performance on CMU-MOSI and CMU-MOSEI.
- Qualitative analysis finds that the dynamic mechanism shifts focus to specific non-verbal signals when semantic conflicts such as sarcasm arise.