Source-linked AI summary
Context Based Emotion Recognition using EMOTIC Dataset
Ronak Kosti, Jose M. Alvarez, Adria Recasens, Agata Lapedriza
TL;DR
Automatic emotion recognition in natural images has focused mainly on faces and sometimes body pose, while scene context has been less explored partly because suitable data were lacking. The paper introduces EMOTIC and CNN models that combine person and whole-scene information, finding that contextual information is relevant for recognizing emotions in the wild. The dataset and trained models are publicly available.
Problem
Scene context provides important information for perceiving emotion, but its use in automatic emotion recognition has been limited partly by a lack of suitable data.
Method
The paper creates the EMOTIC dataset and trains a CNN that jointly processes a person's bounding box and the whole image for apparent-emotion recognition.
Results
Scene context is relevant for recognizing emotions in the wild, based on comparisons of contextual feature types and a person-plus-scene CNN baseline.
Takeaways & Limitations
EMOTIC and the accompanying baseline motivate further research on context-aware emotion recognition and are publicly available to the research community.
Takeaways & Limitations
More than 25% of annotated people have partially occluded or very low-resolution faces, limiting reliance on facial-expression analysis.
Abstract
from arXiv · showhide
In our everyday lives and social interactions we often try to perceive the emotional states of people. There has been a lot of research in providing machines with a similar capacity of recognizing emotions. From a computer vision perspective, most of the previous efforts have been focusing in analyzing the facial expressions and, in some cases, also the body pose. Some of these methods work remarkably well in specific settings. However, their performance is limited in natural, unconstrained environments. Psychological studies show that the scene context, in addition to facial expression and body pose, provides important information to our perception of people's emotions. However, the processing of the context for automatic emotion recognition has not been explored in depth, partly due to the lack of proper data. In this paper we present EMOTIC, a dataset of images of people in a diverse set of natural situations, annotated with their apparent emotion. The EMOTIC dataset combines two different types of emotion representation: (1) a set of 26 discrete categories, and (2) the continuous dimensions Valence, Arousal, and Dominance. We also present a detailed statistical and algorithmic analysis of the dataset along with annotators' agreement analysis. Using the EMOTIC dataset we train different CNN models for emotion recognition, combining the information of the bounding box containing the person with the contextual information extracted from the scene. Our results show how scene context provides important information to automatically recognize emotional states and motivate further research in this direction. Dataset and code is open-sourced and available at: https://github.com/rkosti/emotic and link for the peer-reviewed published article: https://ieeexplore.ieee.org/document/8713881
1 INTRODUCTION
The paper argues that recognizing apparent emotion in natural images requires scene context alongside facial expression and body pose. It introduces the EMOTIC dataset and a CNN baseline that jointly uses person and scene information.
- Motivation: Scene context includes surroundings such as places, attributes, objects, and actions, which can clarify a person's apparent emotional state beyond face and posture.The paper illustrates this with a child whose birthday context supports interpreting him as engaged, happy, and excited.
- Contributions: EMOTIC contains spontaneous, unconstrained images annotated with 26 discrete emotion categories and Valence, Arousal, and Dominance dimensions.The dataset is publicly available, with construction and statistics described in the paper.
- Contributions: The baseline CNN jointly processes the person's bounding-box window and the whole image to predict apparent emotional state.The trained models are also publicly available.
- Contributions: The extended paper adds statistical dataset analysis, scene-centric algorithm analysis, annotator-consistency analysis, and 44.4% more annotated people than the earlier version.The additional analyses accompany retraining of the proposed baseline CNN models.
- Results: Comparative analysis of two scene-context features shows how context contributes to recognizing emotions in the wild.
2 RELATED WORK
Earlier computer-vision emotion-recognition work largely centers on facial expressions, with additional work using body cues, multimodal data, and specialized datasets. The paper distinguishes person emotion recognition in scene context from image sentiment analysis and reviews the limitations of existing datasets.
- Facial and body cues: Facial-expression methods commonly use facial geometry or appearance to recognize Action Units and six basic emotions.Current state-of-the-art facial-expression systems use CNNs for emotion or Action Unit recognition.
- Emotion representation: The VAD Emotional State Model represents emotion using Valence, Arousal, and Dominance, covering pleasantness, activity level, and control.
- Facial and body cues: Some prior work supplements facial information with shoulder location or body pose, but reported settings include small or non-spontaneous datasets.
- Related problems: Emotion recognition identifies emotions of a depicted person, whereas image sentiment analysis predicts what an observer will feel when viewing an image.The emotions of the depicted person and the observer can differ, including for sports or unfair-event images.
- Datasets: Existing emotion datasets include controlled facial-expression collections, acted or restrictive body-affect data, multimodal actor recordings, and broader human-centered challenge datasets.Examples include GENKI, ICML Face-Expression Recognition, UCDSEE, GEMEP, LAP, and EmotiW resources.
- Datasets: COCO includes some emotion-related person attributes, but they are not exhaustive for emotion recognition and do not annotate all people with affect attributes.
3 EMOTIC DATASET
EMOTIC is a large, unconstrained dataset for apparent emotion recognition, combining 26 discrete categories with continuous Valence, Arousal, and Dominance annotations. Its analyses characterize annotation agreement, dataset imbalance, emotion co-occurrence, and relationships between emotions and scene context.
- Dataset construction: EMOTIC contains 23,571 images and 34,320 annotated people collected from diverse, unconstrained environments, including Internet images, COCO, and Ade20k.Annotations describe people in varied places, social environments, and activities.
- Annotation collection: Annotations were collected through two Amazon Mechanical Turk tasks, one for discrete categories and one for continuous dimensions, using person-in-context images marked by red bounding boxes.Annotators received examples and summarized instructions for completing the tasks.
- Emotion representation: The dataset represents apparent emotion using 26 discrete categories and three continuous dimensions: Valence, Arousal, and Dominance.Each continuous dimension takes an integer value in the range [1−10].
- Agreement analysis: More than 50% of validation images achieved κ > 0.30, indicating significant categorical agreement despite the subjectivity of emotion recognition.Agreement was computed using Fleiss’ Kappa across the 26 categories.
- Agreement analysis: Average annotation dispersion was 1.04 for Valence, 1.57 for Arousal, and 1.84 for Dominance, with Dominance showing the greatest disagreement.The paper relates this higher dispersion to the greater difficulty of interpreting Dominance.
- Dataset analysis: Emotion categories are unbalanced and co-occur systematically, while scene-centric analyses reveal associations between emotional labels, places, attributes, and detected adjective-noun phrases.Examples include Annoyance co-occurring with Anger at 46.05% and context patterns involving sports, working environments, couples, and weddings.
4 CNN MODEL FOR EMOTION RECOGNITION IN SCENE CONTEXT
The baseline CNN jointly processes a person’s visible body and whole-image scene context, fusing their features to predict 26 discrete categories and three continuous emotion dimensions.
- Architecture: The model has body, image-context, and fusion modules for jointly estimating discrete categories and continuous dimensions.The body branch extracts person-related cues, while the image branch extracts scene-related features before fusion.
- Feature extraction: The body module uses the visible target body to capture facial, head, pose, and body-appearance cues.It is initialized through ImageNet pre-training.
- Feature extraction: The image module encodes scene category, attributes, objects, and interactions using scene-centric Places pre-training.These features represent contextual information from the whole image.
- Fusion and outputs: The fusion network reduces features to 256 dimensions and branches into 26 category outputs and 3 continuous-dimension outputs.The branches learn independent task representations after the shared fusion layers.
- Loss and training: Training combines discrete-category and continuous-dimension losses with empirically selected weights.The discrete task uses weighted Euclidean loss, while continuous dimensions use robust margin Euclidean or Smooth L1 losses.
- Loss and training: The continuous margin loss excludes errors within a defined margin from weight updates, while Smooth L1 uses squared error below a threshold of 1.The margin formulation uses binary weights for whether each prediction lies inside the error margin.
5 EXPERIMENTS
Experiments compare body-only and body-plus-image models, loss functions, and contextual feature types. Scene context consistently improves discrete-category recognition, whereas continuous-dimension differences are generally insignificant.
- Baseline comparison: B+I outperforms B in every discrete category except one, with B+I(SL1) better than the body-only model.Average Precision is the evaluation metric, with higher values indicating better performance.
- Baseline comparison: For continuous dimensions, all models produce similar Average Absolute Error values, with differences that are not significant.AAE is lower for better performance.
- Per-sample results: The per-sample Jaccard analysis also supports better discrete-category performance for B+I than B.Samples are sorted by decreasing Jaccard Coefficient, whose maximum is 1.
- Per-sample results: Per-sample continuous-dimension AAE results show no significant difference among the models.Samples are sorted with best performances on the left.
- Qualitative results: Qualitative examples generally show B+I outperforming B, although exceptions occur.Incorrect category recognition is marked in red, including an exception in Fig. 15.c.
- Context features: Context-feature logistic regressors obtain mean AP values of 23.00, 27.70, and 29.45 for Bf, Bf+If, and Bf+IS, respectively.The corresponding AAE values are 0.0704, 0.0643, and 0.0713.
- Context features: Both learned Places features and sentiment features contribute relevant information for discrete-category recognition, with IS outperforming If.IS was not trained on EMOTIC but was designed for sentiment analysis and trained on a large image dataset.
6 CONCLUSIONS
The paper presents EMOTIC as a dataset and baseline for recognizing apparent emotion in natural scenes by combining person and scene information. Its findings support contextual information as relevant to emotion recognition and motivate further research.
- Conclusions: EMOTIC contains 23,571 natural, unconstrained images with 34,320 people annotated using 26 categories and three VAD dimensions.The paper also analyzes annotation consistency and dataset statistics.
- Conclusions: The baseline CNN combines the person’s body bounding box with whole-image scene context for emotion recognition.The paper also compares two feature types for encoding contextual information.
- Conclusions: The reported results show the relevance of contextual information for recognizing emotions and motivate further research.The dataset and trained models are publicly available for research.