Source-linked AI summary
Decoupled Multimodal Distilling for Emotion Recognition
Yong Li, Yuanzhi Wang, Zhen Cui
TL;DR
MER faces modality heterogeneities, varying modality contributions, and sub-optimal direct crossmodal distillation. DMD decouples modality features and applies adaptive graph-based distillation, with experiments showing superior or comparable performance to state-of-the-art MER methods.
Problem
MER is challenged by intrinsic modality heterogeneities, differing modality contributions, and feature distribution mismatches that make direct crossmodal distillation sub-optimal.
Method
DMD decouples each modality representation into modality-irrelevant and modality-exclusive spaces and uses GD-Units to learn adaptive crossmodal distillation directions and weights.
Results
DMD achieves superior or comparable results to state-of-the-art MER methods, with quantitative and qualitative experiments consistently demonstrating its effectiveness.
Takeaways & Limitations
Decoupled feature spaces and graph distillation provide flexible knowledge transfer across modalities for robust MER.
Takeaways & Limitations
The reconstruction losses do not guarantee complete feature decoupling because information can leak between representations.
Abstract
from arXiv · showhide
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decoupled into two parts, i.e., modality-irrelevant/-exclusive spaces, in a self-regression manner. DMD utilizes a graph distillation unit (GD-Unit) for each decoupled part so that each GD can be performed in a more specialized and effective manner. A GD-Unit consists of a dynamic graph where each vertice represents a modality and each edge indicates a dynamic knowledge distillation. Such GD paradigm provides a flexible knowledge transfer manner where the distillation weights can be automatically learned, thus enabling diverse crossmodal knowledge transfer patterns. Experimental results show DMD consistently obtains superior performance than state-of-the-art MER methods. Visualization results show the graph edges in DMD exhibit meaningful distributional patterns w.r.t. the modality-irrelevant/-exclusive feature spaces. Codes are released at \url{https://github.com/mdswyz/DMD}.
1. Introduction
MER combines language, visual, and acoustic signals, but modality heterogeneity creates unequal recognition performance and makes fixed crossmodal distillation inadequate. DMD addresses this by decoupling modality features and applying specialized graph distillation with learned directions and weights.
- MER integrates language, visual, and acoustic time-series data to infer human sentiment from video clips.
- Different modalities convey semantic information differently, producing significant performance discrepancies and complicating robust multimodal representation learning.Language is described as more abstract and typically more important for MER than nonverbal behaviors.
- Fixed distillation directions or weights are cumbersome, while direct crossmodal distillation is sub-optimal because modality feature distributions mismatch.The preferred distillation source can vary by emotion and example.
- DMD decouples each modality into modality-irrelevant and modality-exclusive spaces using shared and private encoders plus self-regression.A margin loss further regularizes relationships between the decoupled representations.
- DMD applies homogeneous and heterogeneous graph distillation units to the decoupled spaces, automatically learning distillation directions and weights.The graph vertices represent modality representations or logits, while edges encode knowledge-distillation directions and weights.
- Experiments on public MER datasets report superior or comparable results to state-of-the-art methods, while visualizations show meaningful graph-edge distributions for both GD units.
2. Related Works
Related MER work uses fusion or crossmodal attention to combine modalities, while feature decoupling addresses heterogeneity and redundancy. Knowledge distillation has expanded from teacher–student transfer toward graph-based message passing among multiple knowledge sources.
- MER methods commonly use fusion strategies or crossmodal attention to generate reinforced multimodal representations.
- Fusion approaches design multimodal integration mechanisms, but heterogeneity and information redundancy hinder effective feature fusion.Feature-decoupling methods explore multimodal characteristics and commonalities to facilitate fusion.
- Crossmodal attention methods learn adaptations and correlations between modalities to achieve semantic alignment.MulT is presented as a representative multimodal-transformer approach.
- Knowledge distillation transfers knowledge by minimizing divergence between teacher and student logits and has extended to intermediate-feature distillation.
- Graph-based distillation methods explore message passing among multiple teachers, students, and instances of knowledge.
3. The Proposed Method
DMD decouples multimodal representations into modality-irrelevant and modality-exclusive spaces, then applies specialized graph distillation and adaptive fusion for MER.
- Multimodal feature decoupling: DMD separates multimodal representations into homogeneous modality-irrelevant and heterogeneous modality-exclusive features using shared and private encoders.Self-regression reconstructs coupled features, while margin and orthogonality constraints support the decoupling.
- Multimodal feature decoupling: Reconstruction losses alone may permit information leakage between decoupled representations, leaving homogeneous features meaningless.The method addresses this limitation with additional decoupling constraints.
- Multimodal feature decoupling: A margin loss makes same-emotion crossmodal homogeneous features closer than same-modality features from different emotions.The loss uses positive pairs from different modalities with the same class and negative pairs from the same modality with different classes, separated by margin α.
- Graph distillation: Each decoupled feature space receives a GD-Unit whose directed graph represents modality nodes and learned distillation directions and strengths.Edge weights are computed from modality logits and representations, then normalized with softmax to enable flexible knowledge transfer.
- Graph distillation: HomoGD distills across sufficiently aligned homogeneous features, whereas HeteroGD uses multimodal transformers and crossmodal attention to bridge heterogeneous distribution gaps.The resulting reinforced heterogeneous features and homogeneous features are adaptively fused for multimodal emotion recognition.
- Objective optimization: The full objective combines the emotion-task loss with HomoGD and HeteroGD distillation losses, weighted by λ1 and λ2.The task loss is mean absolute error.
4. Experiments
Experiments on CMU-MOSI and CMU-MOSEI compare DMD with prior MER methods and analyze its components, unimodal behavior, feature spaces, and graph edges. DMD shows superior accuracy, improved modality features, better separability, and meaningful adaptive distillation patterns.
- Comparison with the state-of-the-art: DMD obtains superior MER accuracy on CMU-MOSI and CMU-MOSEI under both unaligned and aligned settings.It also consistently improves over feature-disentangling methods and multimodal-transformer approaches.
- Ablation study: Feature decoupling reduces information redundancy, provides discriminative multimodal features, and reduces the performance gap among modalities.With feature decoupling, standard deviations of ACC2 and F1 decrease across the three modalities.
- Ablation study: DMD outperforms MulT with graph distillation and vanilla MulT, supporting the combination of feature decoupling and graph distillation.The comparison specifically identifies decoupling multimodal features before distillation as feasible and reasonable.
- Visualization of the decoupled features: DMD’s homogeneous features show emotion-category separability, while its heterogeneous features show the best modality separability.Without graph distillation, homogeneous features lose distinguishability under the 7-class setting, and heterogeneous features become less separable.
- Visualization of graph edges in the GD-Units: HomoGD is dominated by L →A and L →V, whereas HeteroGD additionally shows V →A after visual feature discriminability is enhanced.The three unimodal accuracies reported for the graph-edge analysis are 84.5%, 83.8%, and 71.0%, respectively.
5. Conclusion and discussion
DMD addresses varying modality contributions and multimodal heterogeneities through decoupled feature spaces and adaptive cross-modal distillation. Experiments support its effectiveness, while intra-modal interactions remain unmodeled.
- DMD decouples modal features into modality-irrelevant and modality-exclusive spaces using self-regression.
- Two GD-Units enable adaptive cross-modal distillation across the decoupled feature spaces.
- Quantitative and qualitative experiments consistently demonstrate DMD's effectiveness for multimodal emotion recognition.
- DMD does not explicitly consider intra-modal interactions, which the authors identify as future work.