PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects

Chengyan Wu, Yiqiang Cai, Yang Liu, Pengxu Zhu, Yun Xue, Ziwei Gong, Julia Hirschberg, Bolei Ma

arXiv:2505.20511v2cs.CL

TL;DR

Emotion in dialogue is conveyed through multiple modalities, while utterance-level recognition also depends on conversational context. This survey systematically reviews MERC datasets, methods, and evaluation, compares representative approaches, and identifies open challenges; it aims to serve as a practical reference for developing robust and inclusive systems.

  • Problem

    Emotional states in conversation are conveyed through linguistic, acoustic, and visual signals, and recognizing them involves conversational context.

    Source: §1 · p. 1

  • Method

    The survey reviews recent MERC research, including datasets, evaluation strategies, representative methods, and open challenges.

    Source: §1 · p. 2

  • Results

    The survey presents a structured review of MERC, compares representative approaches, and highlights open research problems.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The survey aims to serve as a practical reference supporting development of robust and inclusive emotion recognition systems.

    Source: §7 · p. 9

    The discussion of open challenges and underexplored directions is not exhaustive and does not provide definitive answers.

    Source: Limitations · p. 10

Abstract

from arXiv · show

While text-based emotion recognition methods have achieved notable success, real-world dialogue systems often demand a more nuanced emotional understanding than any single modality can offer. Multimodal Emotion Recognition in Conversations (MERC) has thus emerged as a crucial direction for enhancing the naturalness and emotional understanding of human-computer interaction. Its goal is to accurately recognize emotions by integrating information from various modalities such as text, speech, and visual signals. This survey offers a systematic overview of MERC, including its motivations, core tasks, representative methods, and evaluation strategies. We further examine recent trends, highlight key challenges, and outline future directions. As interest in emotionally intelligent systems grows, this survey provides timely guidance for advancing MERC research.

1 Introduction

MERC combines linguistic, acoustic, and visual signals to recognize emotions in dialogue, where utterance meaning depends on conversational and speaker context. This survey reviews recent MERC research and methods, identifies gaps in existing reviews, and outlines open challenges and future directions.

1 Introduction

  • Unlike isolated-sentence classification, ERC tracks conversational interplay and speaker-specific context while labeling each dialogue utterance.The survey positions MERC as adding information from multiple modalities to conversational emotion recognition.Source: §1 · p. 1
  • MERC integrates linguistic, acoustic, and visual signals to identify an utterance’s emotional state.The survey’s example combines text, audio, and visual inputs in a multimodal model.Source: §1 · p. 1, Figure 1 · p. 1
  • Earlier surveys focus on non-conversational MER or feature fusion, while key MERC topics and recent MLLM advances remain undercovered.The survey identifies cross-modal alignment, reasoning, modality missingness, and conflicts as areas needing fuller treatment.Source: §1 · pp. 1–2
  • The survey reviews recent MERC developments across diverse datasets and methodologies.It describes its review as systematic and comprehensive.Source: §1 · p. 2
  • It compares MERC approaches’ strengths and limitations to offer theoretical insights and practical guidance for method selection..Source: §1 · p. 2
  • The survey identifies open MERC challenges and proposes future research directions for researchers and practitioners.It aims for these directions to guide ongoing and future investigations.Source: §1 · p. 2

2 Task Settings and Review Methodology

The survey defines MERC as predicting an emotion label for each utterance using multimodal information. Its literature review searches scholarly sources and selects conversational, multimodal benchmark studies, prioritizing recent work.

2 Task Settings and Review Methodology

  • The survey defines modalities as distinct information channels conveying emotional or communicative signals, commonly text, acoustics, and vision.Text represents transcribed language; acoustic features include tone, pitch, and energy; visual cues include facial expressions, gaze, and gestures.Source: §2 · p. 2
  • MERC predicts an emotion label for each utterance in a multi-speaker dialogue using its associated multimodal representation.The survey describes textual, acoustic, and visual signals as common modalities, with the representation combining modalities.Source: §2 · p. 2
  • The literature search draws on the ACL Anthology, Google Scholar, and general search engines, focusing on major NLP venues and workshops..Source: §2 · p. 2
  • Selection emphasizes papers using at least two modalities, conversational context, and benchmark datasets, prioritizing work from 2020 onward while retaining foundational studies.Selection includes reviewing each paper’s abstract, introduction, conclusion, and limitations.Source: §2 · p. 2

3 Datasets and Evaluation

The section surveys multimodal conversational datasets and commonly used evaluation metrics. It describes diversity across dataset domains and languages, including low-resource languages.

3 Datasets and Evaluation

  • MERC datasets are grouped into English-centered and non-English categories, including resources for French, Spanish, Greek, Polish, African languages, and Mandarin.The listed English-centered datasets include IEMOCAP, MELD, CMU-MOSEI, AVEC, EmoryNLP, and MEmoR.Source: §3 · p. 3
  • Dataset domains have diversified across TV series, videos, and movies, while language coverage has expanded to include low-resource African languages.The section also names French, Spanish, Greek, Polish, and Mandarin among the languages represented.Source: §3 · p. 3, Table 1 · p. 3
  • MERC studies commonly report Accuracy, Weighted-F1, Macro-F1, and Micro-F1, as well as per-emotion scores for finer-grained analysis.The survey describes its evaluation resources as covering multiple languages and points readers to Appendix §A for benchmark details.Source: §3 · pp. 2–3

4 Feature Processing

Feature processing extracts features from text, audio, and visual streams and then models contextual dependencies. The survey distinguishes sequential situation-level context from speaker-level structure and interactions.

4 Feature Processing

  • Feature processing in MERC comprises feature extraction followed by context modeling, with preprocessing tailored to conversational settings.The survey frames this as a typical preprocessing pipeline for extracting meaningful information from dataset features.Source: §4 · p. 3

4.1 Feature Extraction

  • Mainstream multimodal approaches initially process text, audio, and visual streams separately, with extracted features used differently than in unimodal emotion classification.For unimodal ERC, extraction aims to capture enough information in one modality for emotion classification; Table 2 surveys extraction models used in multimodal studies.Source: §4.1 · p. 3, Table 2 · p. 3

4.2 Context Modeling

  • Situation-level modeling uses sequential dependencies among utterances to represent contextual semantics and a speaker’s temporal emotional dynamics.The section describes a contextual sequence representation derived from each utterance’s textual feature.Source: §4.2 · p. 3
  • Speaker-level modeling adds speaker identity or interaction structure, using speaker embeddings or dialogue graphs to represent speaker- and context-aware features and inter-utterance dependencies.Graph edges encode interaction links, with weights for interaction strength and types for speaker-related or structural relationships; GNN aggregation incorporates connected-node information.Source: §4.2 · p. 4

5 Methodology

The survey organizes MERC methods by their core modeling paradigm: graph-based, fusion-based, and generation-based. It reviews how these approaches model conversational structure, interact across modalities, or use LLMs, along with their respective limitations.

5 Methodology

  • The taxonomy groups methods by their core innovation—graph modeling, cross-modal interaction, or LLM-based generation—even when methods combine components.The categories are not strictly disjoint; the survey uses them to organize the literature by its primary modeling paradigm.Source: §5 · pp. 4–5

5.1 Graph-based Methods

  • Graph-based methods model utterances and their relations to capture conversational dependencies, extending from traditional GNNs to hypergraphs and Fourier graph methods.Examples address missing modalities through joint classification and reconstruction, higher-order interactions through hypergraphs, and local and global emotional signals through Fourier graph operators.Source: §5.1 · p. 5, Figure 3 · p. 5
  • Graph models capture long-range dependencies and speaker interactions, but heterogeneous multimodal signals remain difficult to integrate without alignment that avoids noisy connections.Utterances are modeled as nodes and relations as edges.Source: §5.1 · pp. 5–6

5.2 Fusion-based Methods

  • Fusion-based methods use tailored cross-modal interactions, ranging from equal modality interaction to text-dominant schemes that treat other modalities as complementary signals.Transformer strategies include hierarchical interaction and gating, cross-modal attention, and cue extraction; these methods can overlook speaker dependencies and dialogue-level structure.Source: §5.2 · p. 6, Figure 4 · p. 6

5.3 Generation-based Methods

  • Generation-based methods reformulate ERC as text generation with LLMs, using retrieval-based prompting, dialogue-history prompts, multimodal instruction tuning, or lightweight adaptation.Some approaches incorporate acoustic and textual inputs, image and text instruction data, or video-derived behavior; MSE-Adapter uses a frozen LLM with feature alignment and multi-scale fusion, while SpeechCueLLM converts speech features into prompts.Source: §5.3 · p. 7
  • LLM-based generation offers flexible adaptation across datasets and domains, but its end-to-end design limits fine-grained multimodal control and explicit modeling of multimodal dependencies.The survey also notes substantial computational demands and describes lightweight plug-ins as one approach to efficient multimodal ERC.Source: §5.3 · p. 7

6 Challenges and Prospects

The survey identifies obstacles spanning data resources, multimodal modeling, and deployment. It highlights research directions including more inclusive datasets, stronger alignment and fusion, efficient adaptation, and additional modalities.

FAIR-related Issues Pose Challenges in MERC

  • MERC datasets face limited diversity and FAIR-related gaps, including restricted access, inconsistent labels, and weak metadata that constrain reuse and generalization.The survey proposes multilingual benchmarks with standardized metadata and open licensing.Source: §6 · pp. 7–8

Low-Resource, Multilingual, and Multicultural Settings

  • Low-resource, multilingual, and multicultural MERC remains constrained by scarce expert-annotated data and cultural differences in emotional expression and labeling.The survey points to culture-adaptive pretraining and cross-lingual transfer as future directions.Source: §6 · p. 8

Complexities of Fusion Strategies across Modalities

  • Asynchronous signals, differing time scales, and turn-dependent emotion make utterance-level alignment and long-range conversational modeling difficult.Balancing contextual sentiment cues with multimodal fusion, and choosing effective fusion strategies, remain open questions.Source: §6 · p. 8

Cross-Modal Alignment, Noise Modality, Missing Modality, and Modality Conflicts

  • Misaligned, noisy, missing, imbalanced, or conflicting modalities can weaken fusion, motivating research into uncertainty-aware methods and selective use of trustworthy signals.The survey cites cross-modal attention, graph-based fusion, mutual learning, modality dropout, and reinforcement learning among approaches or directions.Source: §6 · p. 8

Effective Modality Selection

  • Choosing which modalities to retain remains difficult because some add noise while others matter only for particular tasks.Learnable modality gates and sparsity-inducing regularization are proposed for suppressing uninformative modalities during training.Source: §6 · pp. 8–9

Efficient Fine-tuning Approach Using Multimodal LLMs

  • Efficiently adapting multimodal LLMs to nuanced emotion understanding remains an open challenge, especially in low-resource and culturally diverse settings.Adapter modules and low-rank fine-tuning are proposed as ways to adapt large models with minimal data.Source: §6 · p. 9

MERC Application

  • MERC deployment calls for robust, adaptable systems in naturalistic settings, with incremental learning and user feedback identified as possible directions.The survey points to human-computer interaction, healthcare, education, and virtual collaboration as application areas.Source: §6 · p. 9

Expanding the Modality Space in Emotion Recognition

  • Gaze, gestures, posture, and physiological signals are underrepresented in MER, in part because synchronized high-quality data and evolving annotation standards are difficult to obtain.The survey calls for integrating these modalities into benchmarks and developing models that can use them robustly in real-world scenarios.Source: §6 · p. 9

7 Conclusion

MERC integrates linguistic, acoustic, visual, and other signals to understand emotion in dialogue. The survey reviews methods and open problems, including data scarcity, modality alignment, and cross-cultural generalization.

7 Conclusion

  • MERC seeks to recognize emotion in dialogue by integrating linguistic, acoustic, visual, and other modalities.Source: §7 · p. 9
  • Data scarcity, modality alignment, and generalization across languages and cultures remain significant challenges for MERC.Source: §7 · p. 9
  • The survey structures the MERC landscape, compares representative approaches, and highlights open research problems as a reference for future work.Source: §7 · p. 9

Limitations

The survey’s scope prioritizes recent MERC work and offers high-level coverage rather than exhaustive technical or benchmarking detail. Its review and discussion also leave some contributions and open challenges outside its coverage.

Limitations

  • The survey summarizes representative methods at a high level, prioritizing advanced recent technologies over full technical detail.Approaches developed before 2020 receive limited coverage as the survey focuses on recent trends aligned with rapidly evolving large-scale multimodal systems.Source: Limitations · p. 10
  • The review draws largely on English-language publications from major venues and repositories, so contributions from other regions, languages, or domains may be underrepresented.Source: Limitations · p. 10
  • The benchmark section highlights widely used datasets but does not aim for an exhaustive comparison.The survey points readers to complementary surveys for more in-depth benchmarking.Source: Limitations · p. 10
  • The discussion identifies open challenges and underexplored directions but is not exhaustive and does not offer definitive answers.It aims to surface issues and foster further inquiry.Source: Limitations · p. 10

A Datasets Details

The section describes conversational and multimodal datasets spanning dyadic and multi-party dialogue, monologues, human-agent interaction, and multilingual resources, with varied emotion labels and annotation formats.

IEMOCAP

  • IEMOCAP contains 151 dyadic video dialogues and 7,433 utterances from ten speakers, with six emotion labels and sessions 1–4 for training and the last session for testing.Source: A Datasets Details · p. 17

MELD

  • MELD extends EmotionLines with 1,433 Friends-derived multi-party conversations and 13,708 utterances labeled across seven emotion categories.Source: A Datasets Details · p. 17

CMU-MOSEI

  • CMU-MOSEI comprises 23,453 YouTube monologue video segments, with 7-point sentiment ratings and intensity labels for six emotions across aligned language, visual, and acoustic modalities.Source: A Datasets Details · p. 17

M3ED

  • M3ED provides 24,449 utterances from 990 Chinese TV-series dialogues, with text, audio, visual, speaker metadata, and blended emotion labels.The survey describes it as the first large-scale multimodal emotional dialogue corpus in Chinese.Source: A Datasets Details · p. 17

ACE

  • ACE contains 6,162 Akan-language utterances from 385 dialogues, with multimodal data, seven emotion categories, word-level prosodic prominence, and a gender-balanced set of 308 speakers.Source: A Datasets Details · p. 17

MEmoR

  • MEmoR includes 8,536 person-level samples with 14 fine-grained emotions, covering speakers and non-speakers even when modalities are partially or fully missing.Its samples combine target person, emotion moment, and multimodal inputs including personality features.Source: A Datasets Details · p. 18

AVEC

  • AVEC provides utterance-level annotations aggregated from 0.2-second labels for valence, arousal, expectancy, and power in human-agent interactions.Source: A Datasets Details · p. 18

M-MELD and EmoryNLP

  • M-MELD adds human-translated French, Spanish, Greek, and Polish utterances to MELD’s multimodal structure, while EmoryNLP offers Friends-derived textual labels across multi-party dialogue.M-MELD retains MELD’s emotion labels; EmoryNLP contains over 12,000 utterances annotated with seven emotions.Source: A Datasets Details · p. 18
Loading 2505.20511v2…