Source-linked AI summary
EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle
Trisha Mittal, Pooja Guhan, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, Dinesh Manocha
TL;DR
The paper targets perceived human emotion recognition from images and videos, where contextual cues are important beyond isolated signals. EmotiCon combines multimodal, semantic, and sociodynamic context, achieving improved performance on EMOTIC and GroupWalk while remaining limited on datasets with little context.
Problem
Perceived human emotion recognition requires contextual information, but available datasets and methods provide limited coverage of multimodal, scene, and inter-agent context.
Method
EmotiCon combines faces and gaits, semantic scene context, and depth-based modeling of proximity and sociodynamic interactions for multi-label emotion classification.
Results
EmotiCon outperforms prior methods on EMOTIC and GroupWalk, including 35.48 AP across 26 EMOTIC classes.
Takeaways & Limitations
The model and GroupWalk dataset extend context-aware emotion recognition toward naturalistic settings using modalities capturable with commodity hardware.
Takeaways & Limitations
The approach is not expected to improve substantially on datasets with limited context, such as IEMOCAP.
Abstract
from arXiv · showhide
We present EmotiCon, a learning-based algorithm for context-aware perceived human emotion recognition from videos and images. Motivated by Frege's Context Principle from psychology, our approach combines three interpretations of context for emotion recognition. Our first interpretation is based on using multiple modalities(e.g. faces and gaits) for emotion recognition. For the second interpretation, we gather semantic context from the input image and use a self-attention-based CNN to encode this information. Finally, we use depth maps to model the third interpretation related to socio-dynamic interactions and proximity among agents. We demonstrate the efficiency of our network through experiments on EMOTIC, a benchmark dataset. We report an Average Precision (AP) score of 35.48 across 26 classes, which is an improvement of 7-8 over prior methods. We also introduce a new dataset, GroupWalk, which is a collection of videos captured in multiple real-world settings of people walking. We report an AP of 65.83 across 4 categories on GroupWalk, which is also an improvement over prior methods.
1. Introduction
EmotiCon addresses perceived human emotion recognition by combining multiple modalities, scene context, and inter-agent interactions. It introduces a context-aware model and evaluates it on EMOTIC and the new GroupWalk dataset.
- Perceived emotion recognition matters for applications including human-computer interaction, surveillance, robotics, games, and entertainment.
- Context is treated as essential because perceived emotion depends on more than isolated facial or bodily cues.The paper motivates context-aware recognition using Frege’s Context Principle and psychology research.
- EmotiCon combines faces and gaits, background visual information, and sociodynamic interactions as three interpretations of context.
- The model introduces depth-based CNN modeling of agent proximity and releases GroupWalk, containing 45 videos and about 3544 emotion-annotated agents.
- 35.48 AP on EMOTIC improves by 7 −8 over prior methods across 26 discrete emotion classes.
2. Related Work
Prior emotion-recognition research progressed from unimodal systems to multimodal and context-aware models, while datasets increasingly incorporate contextual information. The paper situates EmotiCon within this progression and reviews the limitations of existing datasets and architectures.
- Uni/Multimodal Emotion Recognition: Earlier emotion-recognition systems primarily used single modalities such as facial expressions, speech, body gestures, gaits, or physiological signals.
- Uni/Multimodal Emotion Recognition: Multimodal emotion recognition combines modalities through early, late, or hybrid fusion methods.
- Context-Aware Emotion Recognition in Psychology Research: Psychology research frames emotion as context-dependent, and context-aware recognition work studies background information and other agents.
- Context-Aware Emotion Recognition Datasets: Existing emotion datasets often focus on one modality or controlled settings, although EMOTIC provides contextual images with 26 discrete emotion classes.
3. Our Approach: EmotiCon
EmotiCon interprets context through multimodal cues, semantic scene information, and proximity-based interactions among agents. It processes these context streams from images to produce emotion-recognition features.
- Approach overview: EmotiCon combines multiple modalities, semantic context, and socio-dynamic proximity to recognize perceived emotions.The framework uses modality-specific features, masked scene information, and agent-interaction representations.
- Context 1: Multiple modalities: Context 1 uses modality streams whose feature vectors are combined through multiplicative fusion.The number of streams can vary with the available modalities, and each modality produces a feature vector before fusion.
- Context 2: Semantic context: Context 2 masks the primary agent to extract semantic components such as objects, spatial extents, keywords, and activities from the scene.An attention mechanism focuses on relevant image regions after masking the agent bounding box.
- Context 3: Socio-dynamic interactions: Context 3 models socio-dynamic interactions using depth-derived proximity features and graph-based agent relationships.Depth maps encode pixel distance from the camera, while the graph representation uses agent coordinates and adjacency information.
4. Network Architecture and Implementation Details
EmotiCon implements three context streams for multimodal, semantic, and interaction-based emotion recognition, then fuses their representations for multi-label classification. Training combines multiplicative and classification losses.
- Architecture overview: The implementation preprocesses inputs for three context streams and fuses their learned representations before classification.The architecture includes modality, semantic-context, and interaction-context components.
- Context 1: Context 1 extracts facial landmarks and 2D gait coordinates, processes them with separate networks, and multiplicatively fuses the resulting features.OpenFace provides a 144-dimensional face vector, while OpenPose provides 25 two-dimensional body coordinates.
- Context 2: Context 2 applies an Attention Branch Network to the masked image to identify important semantic regions.The attention branch produces a semantic-context representation from potentially important image locations.
- Context 3: Context 3 uses CNN processing of depth maps and experiments with graph convolutional modeling of agent interactions.The depth network uses alternating convolution and pooling layers, while the graph-based network operates on agent relationships.
- Training objective: The classification objective is multi-label, using a one-versus-all soft-margin loss combined with the multiplicative-fusion loss.The two losses are combined to train EmotiCon.
5. Datasets
The paper evaluates context-aware emotion recognition on EMOTIC and introduces GroupWalk, a real-world video dataset with visible faces and gaits. It also notes that datasets without context provide limited scope for this approach.
- Dataset scope: Many other emotion-recognition datasets lack contextual information, limiting the expected benefit of this context-aware approach on them.The authors report that they do not expect significant improvement over state of the art on datasets without context.
- EMOTIC: EMOTIC contains 23,571 unconstrained-environment images with 34,320 annotated people and 26 multi-label emotion categories.Each person can receive multiple emotion labels.
- GroupWalk: GroupWalk contains 45 stationary-camera videos from 8 real-world settings, with 3,544 agents annotated across four emotion labels.Ten annotators labeled agents with clearly visible faces and gaits; the dataset uses 36 training and 9 testing videos.
- GroupWalk: GroupWalk additionally labels dominance and friendliness because these affects matter for joint and group tasks.The paper provides further annotation details in its appendix.
6. Experiments and Results
Experiments evaluate EmotiCon against prior methods on EMOTIC and GroupWalk, using AP and ablations to assess its context components. Qualitative results show how semantic context and depth maps support predictions, while also revealing failures when context is absent or confusing.
- Comparison with SOTA: EmotiCon outperforms prior methods on both EMOTIC and GroupWalk datasets, with AP evaluated for its GCN-based and depth-map implementations of Context 3.The experiments compare overall and class-level emotion-classification performance using Average Precision.
- Qualitative Results: Qualitative results display input agents, extracted faces and gaits, learned attention maps, and depth maps across examples from both datasets.These visualizations connect each model component to the corresponding input and prediction context.
- Failure Cases: EmotiCon is confused both when context is absent and when abundant context becomes confusing.These two failure cases are illustrated in Figure 4.
- Qualitative Results: Semantic context identifies a coffin and a child’s kite to convey sadness and pleasure, respectively, while depth maps capture proximity and inter-agent interactions.Depth maps also mark a tennis player about to swing and a woman coming from a hospital, conveying anticipation and sadness.
- Ablation Experiments: Context 2 is more expressive on EMOTIC, whereas Context 3 is more representative on GroupWalk.The authors relate this difference to EMOTIC’s richer backgrounds and fewer people per image, versus GroupWalk’s higher agent density.
7. Conclusion, Limitations, and Future Work
EmotiCon combines multimodal, situational, and sociodynamic context using readily available modalities, and the paper releases GroupWalk for naturalistic emotion-recognition research. The authors note label confusion and restrict current classification to discrete emotions, with continuous emotion modeling left for future work.
- EmotiCon combines faces and gaits, situational context, and sociodynamic context using modalities capturable or extractable with commodity hardware.
- The authors release GroupWalk, a dataset intended to support research on emotion recognition with naturalistic modalities.
- The model often confuses certain class labels and currently performs multiclass classification over discrete emotion labels.Future work includes moving toward continuous valence, arousal, and dominance modeling and exploring additional context interpretations.
A.1. Annotation Procedure
GroupWalk uses human annotations of perceived emotions for agents observed in videos from multiple real-world settings. Annotation targets represent four major emotion labels, with extreme and somewhat variants handled within those categories.
- GroupWalk contains 45 stationary-camera videos from eight real-world settings, with 10 annotators labeling 3,544 agents with clearly visible faces and gaits.
- Annotators could replay videos freely and categorized the emotion they perceived for each agent.
- Per-annotator friendliness and dominance label distributions are reported separately in Figures 7 and 8.
- The annotation scheme uses Angry, Happy, Neutral, and Sad as four major labels, with Extreme and Somewhat variants except for Neutral.
A.3. Analysis
GroupWalk annotations are examined across annotators to characterize emotion-label distributions and disagreement. Happy and Sad are frequently confused with Neutral.
- Annotator distributions are shown for every emotion class, with majority labels used to study common confusions.
- Happy and Sad labels are often confused with Neutral across annotators.
- 10 annotators labeled 3544 agents from 45 videos in GroupWalk.
B. EmotiCon on IEMOCAP Dataset
On IEMOCAP, EmotiCon is evaluated across speech, text, and face modalities in controlled conversational recordings. It reaches 84.5% accuracy versus 82.7% for state-of-the-art works, with limited improvement attributed to minimal context.
- IEMOCAP contains speech, text, and face modalities from 10 actors in spontaneous and scripted conversations, with four emotion labels.
- Controlled data collection with minimal context results in limited improvement, while EmotiCon’s performance mainly reflects incorporating modalities.
- Friendliness and dominance label distributions are also reported for GroupWalk annotators.