Source-linked AI summary
Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry
Oriol Marín, Roger Marí, Gloria Haro, Rafael Redondo
TL;DR
Multimodal ERC must combine heterogeneous cues while accounting for conversational context and emotional dynamics. The paper extends SDT with appearance-plus-geometry visual features, class-wise adaptive fusion, and a valence-arousal prior, yielding complementary gains on MELD and IEMOCAP. The results support structured facial cues, emotion-dependent modality weighting, and affective geometry as useful components for multimodal ERC.
Problem
Multimodal ERC must integrate heterogeneous text, audio, and visual cues while modeling conversational context and affective dynamics.
Method
The paper extends SDT with appearance-and-geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions.
Results
Experiments on MELD and IEMOCAP show complementary gains from geometry-enhanced visual representations, class-wise fusion, and targeted affective-transition correction.
Takeaways & Limitations
Structured facial descriptors complement appearance features, modality relevance depends on the predicted emotion category, and affective geometry provides targeted transition gains while largely preserving stable-turn performance.
Takeaways & Limitations
Evaluation is restricted to two benchmarks, while visual processing can fail under occlusion, profile views, or overlapping speech and the affective prior requires dataset-specific annotations.
Abstract
from arXiv · showhide
Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational context and emotional dynamics. We extend the Self-Distillation Transformer architecture for ERC with appearance+geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions. On the MELD and IEMOCAP datasets, geometry-enhanced visual representations improve weighted F1 by 0.27 and 4.36 points over appearance-only features, respectively, while class-wise adaptive fusion provides further gains of 0.17 and 0.25 points over the original softmax gate. The valence-arousal prior yields targeted improvements of 0.30 and 0.74 accuracy points on emotionally shifted utterances while preserving performance on stable turns. These results indicate that structured facial cues, emotion-dependent modality weighting, and affective geometry provide complementary benefits for multimodal ERC.
1 Introduction
Multimodal ERC must integrate heterogeneous cues while tracking conversational and affective dynamics. This work extends SDT with stronger facial representations, emotion-dependent fusion, and affective geometry for emotion shifts.
- Emotion Recognition in Conversations must model linguistic, vocal, facial, contextual, and speaker-related cues as affect evolves across dialogue turns.
- Standard fusion can be dominated by text, while modality informativeness varies across utterances and emotion categories.
- The method extends SDT to test stronger facial representations, class-wise modality adaptation, and affective geometry for emotionally shifted utterances.
- Experiments on MELD and IEMOCAP show consistent multimodal-fusion gains and dataset-dependent improvements in emotion-shift recognition.
2 Related Work
Prior ERC work models conversational context, speaker relations, cross-modal interactions, facial appearance, and emotion transitions through recurrent, graph, attention-based, and affective-space approaches. This work builds on SDT while replacing its original fusion gate with class-wise adaptive fusion.
- Contextual and multimodal ERC: ERC research has progressed from recurrent and memory-based models to graph and Transformer architectures for conversational context and speaker relations.
- Contextual and multimodal ERC: Attention-based methods model long-range context and cross-modal interactions, but standard mechanisms do not explicitly estimate modality reliability for each utterance.
- SDT framework: SDT combines intra-modal and inter-modal Transformers with gated multimodal fusion and unimodal self-distillation.
- SDT framework: This work retains SDT’s encoding and self-distillation while replacing its original softmax gate with class-wise adaptive modality fusion.
- Visual representations: Appearance features capture holistic facial information, whereas geometry-based representations emphasize facial structure and deformation as complementary cues.
- Emotion transitions: Emotion-shift research includes discrete change detection and continuous valence-arousal distances for representing relationships among emotion categories.
3 Method
The method extends SDT with stronger appearance-and-geometry visual features, class-wise modality fusion, and a valence-arousal prior for affective transitions.
- 3 Method: The pipeline extends SDT with updated unimodal encoders, intra- and inter-modal Transformers, class-wise adaptive fusion, and a valence-arousal prior.The prior is applied after multimodal fusion to bias predictions toward plausible affective transitions.
- 3 Method: The training objective combines task and modality-specific cross-entropy losses with KL self-distillation between softened unimodal and fused predictions.The loss follows the original SDT formulation.
- 3.1 Feature Extraction and Modality Encoding: The visual stream samples 16 frames per utterance, detects faces and active speakers, and substitutes zero embeddings when no valid face is found.This processing precedes visual modality encoding.
- 3.1 Feature Extraction and Modality Encoding: Appearance and geometry facial descriptors are combined after separate projection and normalization to strengthen the visual representation.Appearance features use ViT, while geometry uses landmarks, expression parameters, and action units mapped by a lightweight MLP.
- 3.2 Class-Wise Adaptive Modality Fusion: Class-wise adaptive fusion estimates each modality’s contribution separately for every emotion category instead of using SDT’s shared dimension-wise softmax gate.The method combines modality logits using explicit class- and modality-specific contribution scores.
- 3.3 Valence-Arousal Prior for Emotion Shifts: The valence-arousal prior favors emotion classes close to the preceding affective state, providing a transition-aware correction to final predictions.Russell’s circumplex represents each emotion with valence-arousal coordinates, and smaller transitions receive preference.
4 Experiments
Experiments on MELD and IEMOCAP evaluate progressive improvements from visual geometry, class-wise fusion, and an affective transition prior. Results show complementary gains, with effects varying by dataset and emotional stability.
- 4.2 Overall Comparison and Ablation: 75.02 mean weighted F1 is achieved after adding appearance+geometry visual representations and class-wise adaptive fusion, up from 70.34 with updated encoders.The geometry-enhanced stage reaches 74.81 mean weighted F1, followed by 75.02 with class-wise fusion; the updated encoders affect MELD and IEMOCAP differently.
- 4.3 Visual Representation Analysis: 4.36 weighted-F1 points are gained by geometry-enhanced visual representations over appearance-only features on IEMOCAP, while MELD gains are smaller.All geometry-only representations outperform ViT appearance features on IEMOCAP, whereas geometry-only results are similar to the ViT baseline on MELD.
- 4.4 Class-Wise Adaptive Fusion Analysis: 0.17 and 0.25 weighted-F1 points are added by class-wise adaptive fusion on MELD and IEMOCAP, respectively, relative to the softmax gate.The modality-wise class-agnostic variant performs worse, suggesting that a single score per modality is too coarse for emotion-dependent informativeness.
- 4.4 Class-Wise Adaptive Fusion Analysis: Class-wise gains concentrate on less-represented emotions, although softmax fusion remains stronger for a small subset of classes.No clear relationship is observed between the gains and positive versus negative emotion valence.
- 4.4 Class-Wise Adaptive Fusion Analysis: Degradation training recovers most of IEMOCAP’s loss under audio and visual corruption, but does not benefit MELD, which is already dominated by text.The degraded partitions include audio, visual, or joint degradation; the datasets differ in recording conditions and natural noise.
- 4.5 Valence-Arousal Prior: The valence-arousal prior produces modest overall weighted-F1 changes but more apparent improvements on emotionally shifted utterances while reducing corrections for stable predictions.Its stronger effect on IEMOCAP is consistent with dataset-specific valence-arousal coordinates that better match the empirical affective distribution.
5 Conclusion
The framework combines geometry-enhanced visual representations, class-wise adaptive fusion, and a valence-arousal prior, yielding complementary benefits for multimodal ERC. Future work targets dataset- and speaker-specific affective geometries, degraded modalities, and more spontaneous conversations.
- Geometry-enhanced visual representations, class-wise fusion, and the valence-arousal prior provide complementary benefits for multimodal ERC on MELD and IEMOCAP.The prior specifically improves emotionally shifted utterances while largely preserving stable-turn performance.
- Class-wise fusion improves over softmax-gated and class-agnostic alternatives, indicating that modality relevance depends on the predicted emotion category.
- Future research should address dataset- and speaker-specific affective geometries, degraded modality inputs, and more spontaneous conversational data.Analyzing learned class-wise weights could also clarify which modality drives each emotion prediction.