Source-linked AI summary
ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges
Dimitrios Kollias, Panagiotis Tzirakis, Alice Baird, Alan Cowen, Stefanos Zafeiriou
TL;DR
Affective behavior analysis in-the-wild requires benchmarks that cover diverse affective signals and real-world recording conditions. This paper presents the fifth ABAW Competition, defining four challenges over Aff-Wild2 and Hume-Reaction, with evaluation metrics and baselines. The competition therefore provides a shared framework for comparing systems across continuous affect, expressions, action units, and emotional reaction intensities.
Problem
The competition addresses the need to automatically analyze affective behavior in-the-wild for real-life human-computer interaction systems.
Method
The paper presents four competition challenges using extended Aff-Wild2 and Hume-Reaction corpora, evaluation metrics, and baseline systems.
Results
The paper reports baseline performance for valence-arousal estimation, expression classification, action unit detection, and emotional reaction intensity estimation.
Takeaways & Limitations
The competition establishes a shared evaluation framework spanning multiple affective behavior analysis tasks and multimodal corpora.
Takeaways & Limitations
Aff-Wild2 evaluation uses subject-independent partitions, so each person appears in only one of the training, validation, or testing sets.
Abstract
from arXiv · showhide
The fifth Affective Behavior Analysis in-the-wild (ABAW) Competition is part of the respective ABAW Workshop which will be held in conjunction with IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2023. The 5th ABAW Competition is a continuation of the Competitions held at ECCV 2022, IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 and CVPR 2017 Conferences, and is dedicated at automatically analyzing affect. For this year's Competition, we feature two corpora: i) an extended version of the Aff-Wild2 database and ii) the Hume-Reaction dataset. The former database is an audiovisual one of around 600 videos of around 3M frames and is annotated with respect to:a) two continuous affect dimensions -valence (how positive/negative a person is) and arousal (how active/passive a person is)-; b) basic expressions (e.g. happiness, sadness, neutral state); and c) atomic facial muscle actions (i.e., action units). The latter dataset is an audiovisual one in which reactions of individuals to emotional stimuli have been annotated with respect to seven emotional expression intensities. Thus the 5th ABAW Competition encompasses four Challenges: i) uni-task Valence-Arousal Estimation, ii) uni-task Expression Classification, iii) uni-task Action Unit Detection, and iv) Emotional Reaction Intensity Estimation. In this paper, we present these Challenges, along with their corpora, we outline the evaluation metrics, we present the baseline systems and illustrate their obtained performance.
1. Introduction
The fifth ABAW Competition targets affective behavior analysis in-the-wild through four challenges spanning continuous affect, expressions, action units, and emotional reaction intensities. It uses extended Aff-Wild2 and Hume-Reaction corpora to connect affective computing with related disciplines and real-world human-centered interaction.
- Motivation: The competition addresses affective behavior analysis in-the-wild, a targeted capability for HCI systems used in real-life applications.The broader goal is machines and robots that understand people’s feelings, emotions, and behaviors for more human-centered interaction.
- Corpora: Extended Aff-Wild2 contains around 600 audiovisual videos and around 3M frames annotated for dimensional affect, categorical affect, and facial action units.Dimensional affect includes valence and arousal; categorical affect includes six basic expressions plus neutral.
- Corpora: Hume-Reaction provides audiovisual reactions to emotional stimuli annotated with seven emotional expression intensities.The seven intensities include adoration, amusement, anxiety, disgust, empathic pain, fear, and surprise.
- Competition scope: Its four challenges cover valence-arousal estimation, expression classification, action unit detection, and emotional reaction intensity estimation.The first three challenges use Aff-Wild2, while emotional reaction intensity estimation uses Hume-Reaction.
- Context: The competition brings together affective computing, machine learning, multimodal signal processing, and other disciplines through a continuing ABAW series.The fifth competition was held with CVPR 2023 and continued competitions associated with ECCV 2022, CVPR 2022, ICCV 2021, FG 2020, and CVPR 2017.
2. Competition Corpora
The competition draws on multiple annotated, audiovisual corpora covering continuous affect, expressions, action units, and nuanced emotional reactions. Its data preparation includes subject-independent splits, face detection, landmark-based alignment, cropping, resizing, and normalization.
- Valence-Arousal Estimation Challenge: The valence-arousal corpus contains 594 videos, around 3 million annotated frames, and 584 subjects, with values ranging continuously in [−1, 1].Four experts provided the annotations, and sixteen videos contain two annotated subjects.
- Valence-Arousal Estimation Challenge: Aff-Wild2 valence-arousal data are split into training, validation, and testing sets subject-independently.A person appears strictly in only one of these sets.
- Expression Classification Challenge: The expression corpus contains 546 videos and 2,624,160 annotated frames from 437 subjects across six basic expressions, neutral, and other.Seven experts annotated the data frame by frame.
- Action Unit Detection Challenge: The action-unit corpus contains 541 videos and 2,627,632 annotated frames from 438 subjects covering 12 action units.Annotations were produced semi-automatically and performed frame by frame.
- Emotional Reaction Intensity Estimation Challenge: Hume-Reaction supports multi-output regression over seven self-annotated emotion classes in uncontrolled multimodal recording conditions.The dataset includes 2,222 participants from South Africa and the United States, aged 18.5–49.0 years.
- Preprocessing: Preprocessing detects faces and five landmarks with RetinaFace, crops and similarity-aligns images, then resizes them to 112 × 112 × 3 and normalizes intensities to [−1, 1].The cropped and aligned images were provided to participating teams and used in baseline experiments.
3. Evaluation Metrics Per Challenge
The challenges use task-specific metrics: average CCC for valence-arousal estimation, macro F1 for expression classification and action unit detection, and average Pearson correlation for emotional reactions.
- Valence-Arousal Estimation Challenge: Average CCC measures agreement between valence and arousal annotations and predictions, accounting for correlation and mean square difference.CCC ranges from −1 to 1, with higher values preferred.
- Expression Classification Challenge: Macro F1 averages F1 scores across all 8 expression categories.F1 combines precision and recall and ranges from 0 to 1, with higher values preferred.
- Action Unit Detection Challenge: Macro F1 averages F1 scores across all 12 action units for action unit detection.
- Emotional Reaction Intensity Estimation Challenge: Emotional reaction intensity estimation uses the average Pearson’s Correlation Coefficient across the 7 emotional reactions.
4. Baseline Networks Results
The paper presents reproducible baseline systems for the four ABAW Challenges and reports validation or sub-challenge evaluation results using task-specific metrics. Results indicate stronger video than audio performance for emotional reaction intensity, while adding audio generally does not improve video models.
- All baseline systems use existing open-source machine-learning toolkits and are implemented in TensorFlow for reproducibility.
- Valence-Arousal Estimation: The valence-arousal baseline uses an ImageNet-pretrained ResNet50 with a linear output layer producing valence and arousal estimates.
- The validation tables evaluate valence-arousal estimation by average CCC, expression classification by average F1 Score, and action-unit detection by average F1 Score.
- Emotional Reaction Intensity Estimation: The audio modality produces low correlation, with ρc = .0741, while the video baseline using Facial Action Units reaches ρc = .2801.
- Emotional Reaction Intensity Estimation: Combining audio and visual modalities generally does not outperform video models, although ViPER gains .047 when audio is added.
- Emotional Reaction Intensity Estimation: The emotional reaction intensity results report mean Pearson’s Correlation Coefficient across seven emotional reaction classes, with confidence intervals where available.
5. Conclusion
The fifth ABAW Competition comprises four affective-computing Challenges based on extended Aff-Wild2 and Hume-Reaction corpora. It covers valence-arousal estimation, expression classification, action-unit detection, and emotional reaction intensity estimation.
- The Competition includes four Challenges: valence-arousal estimation, expression classification with 8 categories, action-unit detection with 12 action units, and emotional reaction intensity estimation.
- The Competition uses an extended version of Aff-Wild2 and the Hume-Reaction dataset.