PaperlayerView exact revision on arXiv

Source-linked AI summary

Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies

Sicheng Zhao, Guoli Jia, Jufeng Yang, Guiguang Ding, Kurt Keutzer

arXiv:2108.10152v1eess.SPcs.AIcs.LGcs.MM

TL;DR

MER addresses how machines can recognize and interpret emotions expressed through multiple human and digital modalities. This tutorial synthesizes emotion models, modalities, annotations, tasks, challenges, methods, applications, and future directions. Across reviewed methods, deep models generally outperform non-deep models, although performance varies by dataset and task.

  • Problem

    The central problem is enabling machines to recognize, interpret, process, and simulate emotions from multimodal affective information.

  • Method

    The tutorial systematically covers emotion representations, affective modalities, data and annotations, computational tasks, challenges, methodologies, applications, and future directions.

  • Results

    Deep models generally outperform non-deep models, but the best-performing method differs across datasets and multiclass classification is harder than binary classification.

  • Takeaways & Limitations

    MER research must continue addressing discriminative emotion features, personalized reactions, and the relative importance of modalities in real-world settings.

  • Takeaways & Limitations

    Emotion labels are expensive, time-consuming, inconsistent, and sometimes incomplete or noisy, motivating unsupervised, weakly supervised, few-shot, and zero-shot learning.

Abstract

from arXiv · show

Humans are emotional creatures. Multiple modalities are often involved when we express emotions, whether we do so explicitly (e.g., facial expression, speech) or implicitly (e.g., text, image). Enabling machines to have emotional intelligence, i.e., recognizing, interpreting, processing, and simulating emotions, is becoming increasingly important. In this tutorial, we discuss several key aspects of multi-modal emotion recognition (MER). We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks, followed by the description of main challenges in MER. Furthermore, we present some representative approaches on representation learning of each affective modality, feature fusion of different affective modalities, classifier optimization for MER, and domain adaptation for MER. Finally, we outline several real-world applications and discuss some future directions.

I. INTRODUCTION

Emotion recognition matters because emotions influence human decisions and behavior, while emotionally intelligent machines could support more harmonious services and real-time behavioral monitoring. The tutorial frames MER around multiple modalities, emotion models, annotations, computational tasks, challenges, methods, applications, and future directions.

  • Emotions influence judgment, decision making, consumer preferences, accident risk, and students’ learning experiences.
  • Emotionally intelligent machines are intended to recognize, interpret, process, and simulate emotions for applications including personalized services and abnormal-behavior monitoring.
  • MER combines explicit affective cues, such as facial expression, eye movement, speech, action, and physiological signals, with implicit affective stimuli.
  • Multiple modalities provide complementary cues that can improve interpretation compared with relying on a single modality.
  • The tutorial surveys emotion models, affective modalities, data and annotations, computational tasks, challenges, methodologies, applications, and future directions.
  • Categorical and dimensional emotion models offer complementary representations, while mood, affect, and sentiment are related concepts that the tutorial does not distinguish in detail.

III. AFFECTIVE MODALITIES

MER uses explicit bodily or behavioral cues and implicit stimuli from digital media. These modalities include human signals such as facial expression and speech, alongside text, audio, and images that can convey or evoke emotion.

  • Explicit Affective Cues: Explicit affective cues include facial expression, eye movement, speech, action, gait, and electroencephalogram signals.
  • Explicit Affective Cues: Facial expressions and eye movements provide observable nonverbal and physiological-adjacent indicators of emotional states and intentions.
  • Explicit Affective Cues: Speech conveys emotion through vocal properties including intonation, loudness, and tempo, while actions and body movement also carry emotional information.
  • Implicit Affective Stimuli: Implicit affective stimuli include text, general audio, and images, which can carry informative emotions or evoke emotions in people.

IV. DATA COLLECTIONS AND EMOTION ANNOTATIONS

MER datasets are built through data collection followed by emotion annotation, using sources ranging from existing media to recordings in controlled environments. The resulting data support classification and related computational tasks with categorical, multilabel, or distributed emotion targets.

  • Data Collection: MER datasets are constructed through data collection and emotion annotation, using existing media or new recordings in specific environments.
  • Emotion Annotation: Annotation strategies include target emotions, automatic labels, worker judgments, participant self-reports, major voting, and dimensional tools such as FeelTrace and SAM.
  • Computational Tasks: MER tasks include classification, regression, detection, and retrieval.
  • Emotion Classification: Emotion classification can assign one dominant label, multiple labels, or a label distribution reflecting differing importance across emotions.
  • Datasets: Dataset summaries organize released MER resources for comparing their modalities, collection settings, and annotation characteristics.

B. Emotion Regression

Emotion regression learns mappings from multimodal data to continuous emotion values, commonly in dimensional spaces. MER also includes detection and retrieval tasks, but affective retrieval remains difficult to evaluate because emotions are abstract and subjective.

  • Emotion Regression: Emotion regression learns a mapping from an instance to continuous emotion values in a Cartesian space.
  • Emotion Regression: Regression methods commonly predict average dimensional values, while some approaches model continuous emotion distributions in valence-arousal space.
  • Emotion Detection: Emotion detection identifies which emotion occurs where in source data, such as assigning different emotions to separate sentences in one review.
  • Emotion Retrieval: Affective retrieval searches for emotionally matched content by comparing local patches or sequences against a fixed distance threshold.
  • Emotion Retrieval: Affective retrieval is difficult to evaluate because its target characteristics are abstract and subjective.

VI. CHALLENGES

MER faces challenges from the gap between extracted features and perceived emotion, subjective human responses, and incomplete modality data. These issues complicate feature evaluation and reliable multimodal learning.

  • A. Affective Gap: The affective gap measures inconsistency between extracted features and perceived high-level emotions, making emotion-related feature evaluation difficult.Different visual appearances or voice intonations can convey different emotions despite similar semantic content.
  • B. Perception Subjectivity: People can respond differently to the same stimulus because of cultural, personal, contextual, and psychological factors.In ASCERTAIN, all 36 videos received labels from at least four of seven valence and arousal scales by 58 subjects.
  • C. Data Incompleteness: Data incompleteness arises when collection problems, such as sensor failure, corrupt or remove information from specific modalities.Examples include contaminated EEG signals and unclear facial images at night.

D. Cross-modality Inconsistency

Cross-modality inconsistency arises when modalities conflict, contribute unequally, or are unavailable across domains. MER therefore must address disagreement, modality imbalance, limited labels, and domain shift.

  • D. Cross-modality Inconsistency: Different modalities may express conflicting emotions, such as masked facial or speech cues versus unconscious EEG responses.Images and text in social-media posts may also be semantically unrelated.
  • D. Cross-modality Inconsistency: Modalities can contribute unequally to an emotion, as illustrated by online news containing lengthy text alongside images.This creates imbalanced multimodal inputs for applications such as predicting readers’ emotional reactions.
  • F. Label Noise And Absence: Deep-learning MER methods require large labeled datasets, but emotion annotation is expensive, time-consuming, inconsistent, incomplete, and noisy.The paper identifies unsupervised, weakly supervised, few-shot, and zero-shot learning as potential alternatives.
  • F. Label Noise And Absence: A widely used MER framework combines representation learning, feature fusion, and classifier optimization for tasks including classification, regression, detection, and retrieval.When transferring from a labeled source domain to an unlabeled target domain, domain shift causes performance decay under direct transfer.

VII. COMPUTATIONAL METHODOLOGIES

MER methodologies build modality-specific representations before integrating them into a framework of representation learning, feature fusion, and classifier optimization. The reviewed representations span text, audio, images, video, gait, and EEG.

  • VII. COMPUTATIONAL METHODOLOGIES: MER frameworks generally comprise representation learning, feature fusion, and classifier optimization when sufficient target-domain labels are available.Domain adaptation is additionally considered when target labels are unavailable and related source-domain labels exist.
  • Text Representation: Text representation requires mapping words to real-valued vectors, modeling semantic relationships, and constructing a unified representation for the whole text.Early methods represented each vocabulary word with a one-hot vector.
  • Audio Representation: Audio emotion recognition uses acoustic features including pitch, log energy, zero-crossing rate, spectral features, voice quality, and jitter.The ComParE acoustic feature set is commonly used as a baseline in the Computation Paralinguistics Challenge series.
  • Image Representation: Image emotion representations range from hand-crafted color, texture, shape, and composition features to facial-expression cues based on detected and aligned faces.For facial expression images, face landmarks are encoded for recognition.
  • Video and Gait Representation: Video representations model successive frames and their temporal information, while gait representations explicitly encode skeletal posture and movement.Gait modeling can use joint coordinates and derived posture or motion quantities.
  • EEG Representation: EEG emotion recognition commonly extracts differential-entropy features from frequency bands or electrode-position relationships, alongside end-to-end deep learning.EEG supports analysis of frequency-band, electrode-position, and temporal information in brain responses.

B. Feature Fusion of Different Affective Modalities

Feature fusion integrates representations from multiple modalities to predict categorical or continuous emotions. MER uses model-free strategies such as early and late fusion, alongside model-based kernel, graph, neural, attention, and tensor methods.

  • Fusion Overview: Feature fusion integrates multiple modality representations to predict either a specific emotion category or a continuous emotion value.The two broad strategies are model-free fusion and model-based fusion.
  • Model-free Fusion: Model-free fusion includes early, late, and hybrid fusion; early fusion concatenates modality features but requires synchronization and fails when modalities are missing.Late fusion combines unimodal prediction results through mechanisms such as averaging or voting.
  • Model-based Fusion: Model-based fusion performs fusion during learning-model construction and includes kernel-based, graph-based, neural-network, attention-based, and tensor-based methods.These methods are designed more explicitly for multimodal data than simple model-free techniques.
  • Model-based Fusion: Kernel-based fusion offers flexible kernel selection and convex losses but can require substantial memory and inefficient inference because testing relies on training support vectors.Graph-based fusion constructs separate graphs for different modalities.
  • Model-based Fusion: Neural fusion combines modality features or predictions, whereas attention fusion learns scalar weights to form weighted sums of feature representations.Attention mechanisms can target different components, such as spatial image features.
  • Temporal Modeling: Early fusion supports temporal modeling, while late and hybrid fusion do not because their unimodal predictions are already available before fusion.Except for kernel-based fusion, model-based methods can support temporal modeling, including HMMs and CRFs for graph-based fusion.

C. Classifier Optimization for Multi-modal Emotion Recognition

Classifier optimization in MER spans modality-specific sequence, spatial, and temporal models, alongside traditional and deep learning approaches. The section also situates these methods within broader multimodal classification pipelines.

  • Text emotion recognition: Text emotion classifiers use RNNs and CNNs to model sequential and local contextual semantics from word embeddings.LSTM updates hidden states across words, while CNN extracts features from consecutive words.
  • Audio emotion recognition: Audio emotion recognition combines traditional HMM and SVM methods with deep sequence-to-sequence, attention, and hybrid convolutional-recurrent models.Deep methods avoid classical HMM independence assumptions and support end-to-end learning.
  • Image emotion recognition: Image emotion recognition evolved from hand-crafted features with SVM toward end-to-end deep optimization using cross-entropy and metric losses.Triplet and N-pair losses are used to obtain more discriminative features.
  • Video emotion recognition: Video emotion recognition commonly uses a two-stage pipeline that extracts visual or audio features before training classifiers for discrete emotion categories.Investigated classifiers include SVM, GMM, HMM, DBNs, and CRFs.
  • Other modalities: Gait and EEG emotion recognition use recurrent, convolutional, and hybrid architectures to capture temporal, spatial, or joint-coordinate information.EEG systems include cascade convolutional-recurrent and LSTM-based structures, while gait systems use LSTM representations of joint sequences.

1) Quantitative Comparison of Representative MER Methods:

The comparison evaluates representative non-deep and deep MER methods across five datasets and examines domain adaptation for unlabeled target domains. Results favor deep models overall, while performance varies by dataset, task, and language representation.

  • Experimental setup: The evaluation compares SVM, RF, THMM, MT-LSTM, BC-LSTM, TFN, MARN, MFN, FT, and MAG on five datasets with face, speech, and transcript modalities.Features come from Facet, COVAREP, and GLOVE, BERT, or XLNet embeddings; accuracy, F1, and mean absolute error are used.
  • Quantitative comparison: Deep models generally outperform non-deep models, but the best-performing method differs across datasets.RF performs best among non-deep models except on CMU-MOSI, where SVM performs better than RF and THMM.
  • Quantitative comparison: 77.1 vs. 34.7 for MARN on CMU-MOSI shows that multiclass classification is more difficult than binary classification.The comparison is between the corresponding multiclass and binary results.
  • Quantitative comparison: BERT and XLNet outperform GLOVE on CMU-MOSI, with XLNet generally better than BERT.XLNet-based MAG reaches near-human performance but remains below human performance.
  • Domain adaptation: Multi-modal domain adaptation trains on labeled source domains while aligning them with unlabeled target domains, with fused-feature discrepancy or discriminator strategies providing extensions to multimodal settings.The main difficulty is aligning multiple modalities between source and target domains simultaneously.

VIII. APPLICATIONS

MER supports applications that infer emotions from user-generated content, consumer behavior, communication, health-related signals, and cross-modal entertainment data. These applications connect affective analysis with social understanding, service improvement, and personalized experiences.

  • Social networks: Social-network MER can mine opinions and sentiments about real-world topics and events from users’ texts, images, and videos.The paper gives epidemic-related attitudes across countries and regions as an example.
  • Business intelligence: Business intelligence uses MER for marketing, consumer behavior analysis, and call-center monitoring.Applications include analyzing how facial expressions affect approach behavior and detecting caller and operator emotional states.
  • Health and psychological care: Medical and psychological-health applications monitor repeatedly shared negative emotions to help track mental status.The paper links persistent negative information sharing with the need to monitor psychological health.
  • Entertainment: Entertainment systems use MER to connect EEG characteristics with musical structures and match images with emotionally compatible music.The examples include mapping brainwave features to notes, intensity, and pitch.

IX. FUTURE DIRECTIONS

Future MER research should address unresolved modeling challenges, realistic deployment conditions, personalization, and real-world applications. The tutorial emphasizes context-aware methods, adaptation beyond laboratory settings, and feedback from deployed systems.

  • Open challenges: Open challenges include extracting emotion-discriminative features, balancing common and personalized reactions, and weighting important modalities.These challenges remain despite promising performance across several multimodal settings.
  • New methodologies: Contextual and prior-knowledge modeling should incorporate conversational or social environments together with user traits such as personality and age.Different personality types may perceive the same stimulus differently.
  • More practical MER settings: MER in the wild must handle noise, cultural and linguistic variation, and diverse ways of expressing emotion.Laboratory-focused methods do not fully represent the complexity of real-world data.
  • More practical MER settings: Personalized and group emotion recognition are proposed because dominant-emotion prediction is insufficient for subjective individual reactions.Adapting a general model with a small amount of labeled individual data is presented as one possible approach.
  • Real applications: Deploying MER in marketing, education, healthcare, and service applications can provide feedback that promotes further development.The tutorial identifies recent progress as making real-world incorporation more feasible than before.
Loading 2108.10152v1…