DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis

Introduction

• The open challenge is learning essential multimodal representations without redundancy or conflict, amid differences in modality contributions and concerns about reciprocal information transfer.The introduction notes that language, vision, and audio contribute differently, while reciprocal transfer may introduce redundancy or conflict.Source: Introduction · pp. 1–2
• DLF addresses redundant and conflicting cross-modal information through a pipeline of feature disentanglement, enhancement, fusion, and prediction.It is designed for language-dominant MSA.Source: Introduction · p. 2
• Four geometric measures regularize the shared and specific spaces separately and jointly to refine disentanglement.Source: Introduction · p. 2
• The LFA uses language-guided multimodal cross-attention to attract complementary modality-specific information toward language.The targeted enhancement includes Video→Language, Audio→Language, and Language→Language pathways.Source: Introduction · p. 2
• Hierarchical predictions leverage pre-fused and post-fused features to improve total MSA accuracy.Source: Introduction · p. 2

Multimodal Sentiment Analysis

• MSA research includes representation-learning and fusion-oriented methods, with distillation approaches also used to bridge modality gaps.Prior approaches may treat modalities equally, which the passage associates with redundant and conflicting information.Source: Multimodal Sentiment Analysis · pp. 2–3
• The DLF overview presents a pipeline of feature extraction, disentanglement, enhancement, fusion, and prediction, with three core components.Those components are feature disentanglement, the LFA, and hierarchical predictions.Source: Figure 2 · p. 3
• DLF's LFA directs knowledge transfer toward the dominant language modality to mitigate redundancy and conflict.Source: Multimodal Sentiment Analysis · p. 3

Disentangled Multimodal Representation Learning

• Disentangled multimodal methods separate shared and modality-specific subspaces, but DLF uses disentanglement to facilitate language-targeted enhancement.Four measures regularize shared and specific spaces jointly and independently.Source: Disentangled Multimodal Representation… · p. 3

Overview

• The framework takes language, vision, and audio inputs represented as modality-specific two-dimensional tensors and follows a pipeline from feature extraction through prediction.Its core components are feature disentanglement, the LFA, and hierarchical predictions.Source: Proposed Approach · p. 3, Overview · p. 3

Feature Disentanglement Module

• DLF separates modality-shared and modality-specific features, using reconstruction and geometric regularization to reduce redundancy and conflicts.Its four regularization measures use Euclidean distances and cosine similarity; the losses include a modified triplet loss and soft orthogonality loss.Source: Feature Disentanglement Module · pp. 3–4

Language-Focused Attractor (LFA)

• The LFA applies language-query cross-attention and self-attention to decoupled modality-specific features, drawing complementary information from other modalities to strengthen language representation.Cascaded Multimodal Transformer layers perform the enhancement.Source: Language-Focused Attractor (LFA) · pp. 4–5, Figure 3 · p. 5
• DLF combines higher-level shared features with enhanced modality-specific features to form the final multimodal representation.Shared features pass through a unified Transformer layer and two fully connected layers before projection.Source: Language-Focused Attractor (LFA) · p. 5

Hierarchical Predictions

• Hierarchical predictions jointly use modality-shared, modality-specific, and final output losses rather than relying only on a single output loss.The final output is predicted after multimodal fusion.Source: Hierarchical Predictions · p. 5

Datasets and Evaluation Metrics

• MOSI (2,199 clips) and MOSEI (22,856 clips) are evaluated with Acc-7, Acc-5, Acc-2, F1, Corr, and MAE.Source: Datasets and Evaluation Metrics · p. 5

Implementation Details

• BERT-base-uncased yields 768-dimensional linguistic features; visual features use Facet facial action units, and audio uses 74-dimensional COVAREP features.Source: Implementation Details · pp. 5–6

Main Results

• DLF outperforms eleven leading MSA methods on almost all metrics across MOSI and MOSEI, with Table 1 reporting the comparison.Source: Table 1 · p. 6, Main Results · p. 6

Ablation Study

• On MOSI, language is dominant; language-plus-audio or language-plus-vision improves especially fine-grained classification, while trimodal DLF beats bimodal DLF on all metrics.The authors attribute the trimodal result to each modality contributing uniquely when its information is reasonably exploited.Source: Ablation Study · p. 6, Table 2 · p. 7
• Removing soft orthogonality yields the worst performance; modified triplet, reconstruction, and specific losses also contribute through shared-feature learning and disentanglement consistency.Source: Ablation Study · p. 6, Table 2 · p. 7
• Removing LFA, FDM, or hierarchical predictions lowers overall performance; the HP ablation retains only the final-output loss.Replacing LFA with three separate query components also lowers performance; the authors say LFA mitigates potential redundancy or conflict in cross-attention.Source: Table 2 · p. 7, Ablation Study · p. 7

Further Analysis

• On MOSI, most sentiment classes exceed 40% accuracy, but HN and HP—especially HN—perform worst, and their samples are relatively scarce.The authors identify the long-tailed distribution as limiting overall MSA performance.Source: Figure 4 · p. 7, Further Analysis · p. 7
• Figure 5 shows DLF separating sentiment representations better than DMD; the authors attribute this mainly to LFA mitigating redundant and conflicting multimodal information.Source: Figure 5 · p. 7, Further Analysis · p. 7

• DLF follows feature extraction, disentanglement, enhancement, fusion, and prediction, with disentanglement, LFA, and hierarchical predictions identified as key contributors.Source: Conclusion · p. 7
• Comparisons with eleven baselines and comprehensive ablation studies are reported as verifying DLF’s superiority.Source: Conclusion · p. 7
• The authors say DLF may exploit imbalanced modality capabilities, and LFA may generalize by changing the dominant modality.Source: Conclusion · p. 7
• The method considers only complete-modality scenarios; missing modalities may limit its feature disentanglement and enhancement modules.Source: Conclusion · p. 7