Source-linked AI summary
Analysing Affective Behavior in the second ABAW2 Competition
Dimitrios Kollias, Irene Kotsia, Elnar Hajiyev, Stefanos Zafeiriou
TL;DR
ABAW2 addresses automated affect analysis in-the-wild for human-centered interaction by organizing three behavior-recognition challenges around Aff-Wild2. The paper presents the shared corpora, evaluation framework, and baseline systems, with Aff-Wild2 serving as the first database annotated across all three tasks. Its contribution is a common benchmark and baseline infrastructure for valence-arousal, expression, and action-unit analysis.
Problem
Affective behavior analysis must operate in diverse real-world environments, while existing emotion representations cover different aspects of affect and require a shared in-the-wild benchmark.
Method
The paper organizes ABAW2 into three challenges using Aff-Wild2 and describes their corpora, evaluation metrics, preprocessing, and baseline networks.
Results
Aff-Wild2 provides a common large-scale benchmark annotated for valence-arousal estimation, seven basic expressions, and twelve action units, alongside baseline systems and results.
Takeaways & Limitations
ABAW2 establishes a shared competition framework for evaluating affective behavior analysis across three complementary in-the-wild tasks.
Abstract
from arXiv · showhide
The Affective Behavior Analysis in-the-wild (ABAW2) 2021 Competition is the second -- following the first very successful ABAW Competition held in conjunction with IEEE FG 2020- Competition that aims at automatically analyzing affect. ABAW2 is split into three Challenges, each one addressing one of the three main behavior tasks of valence-arousal estimation, basic expression classification and action unit detection. All three Challenges are based on a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the first one to be annotated for all these three tasks. In this paper, we describe this Competition, to be held in conjunction with ICCV 2021. We present the three Challenges, with the utilized Competition corpora. We outline the evaluation metrics and present the baseline system with its results. More information regarding the Competition is provided in the Competition site: https://ibug.doc.ic.ac.uk/resources/iccv-2021-2nd-abaw.
1. Introduction
The paper frames in-the-wild affective behavior analysis as a foundation for human-centered HCI and describes ABAW2’s three challenges on Aff-Wild2. These challenges cover dimensional affect, basic expressions, and facial action units.
- Motivation: In-the-wild affect analysis is presented as important for HCI systems intended for real-life applications.The stated goal is to enable machines and robots to understand feelings, emotions, and behaviors for human-centered interaction.
- Motivation: Human affect recognition could support natural machine assistance and contribute to trust, understanding, and closeness in real-world environments.The paper also states that interaction should not depend on context or demographic and social characteristics.
- Emotion Representation: Human emotion is represented categorically, through facial action units, or dimensionally using continuous valence and arousal.Valence indicates how positive or negative an emotional state is, while arousal indicates how passive or active it is.
- Competition Scope: ABAW2 comprises three in-the-wild challenges: valence-arousal estimation, seven basic expression classification, and 12 facial action unit detection.All three challenges use Aff-Wild2, described as the first comprehensive benchmark annotated for these tasks.
- Paper Organization: The paper introduces the competition corpora, evaluation metrics, challenge-specific baselines, and baseline results before concluding.The paper is organized across corpus, evaluation, baseline, and conclusion sections.
2. Competition Corpora
The ABAW2 competition uses Aff-Wild2 as a shared, large-scale in-the-wild corpus for valence-arousal estimation, expression classification, and action unit detection. The corpus provides task-specific annotations, subject-independent splits, and preprocessing resources for participants and baseline experiments.
- Corpus Overview: Aff-Wild2 is the first database annotated for valence-arousal estimation, action unit detection, and basic expression classification.These three tasks define the three ABAW2 challenges.
- Corpus Overview: Aff-Wild2 contains 548 YouTube videos and 2,813,201 frames, including 458 subjects and real-world behaviors ranging from subtle to extreme.Sixteen videos contain two annotated subjects; the corpus extends Aff-Wild with 260 additional videos and 1,413,000 frames.
- Preprocessing: The organizers provide face bounding boxes, five facial landmarks, cropped faces, and similarity-transformation-aligned images for participant development and baseline experiments.Faces are detected with an SSH detector based on ResNet and trained on WiderFace.
- Valence-Arousal Annotation: Valence-arousal annotations cover 545 videos, 2,786,201 frames, and 455 subjects, with continuous values ranging from −1 to 1.Four experts produced frame-by-frame joystick annotations, averaged into time-continuous labels.
- Dataset Scale: Aff-Wild2 is described as the largest audiovisual in-the-wild database annotated for valence and arousal, seven basic expressions, and action units.The paper states this separately for each annotation type.
- Data Splits: Each challenge uses training, validation, and test subsets partitioned subject-independently so each person appears in only one subset.The resulting video counts differ by task: valence-arousal 346/68/131, expressions 250/70/222, and action units 302/105/127.
- Expression Annotation: Expression annotations cover 539 videos, 2,595,572 frames, and 431 subjects, retaining labels supported by agreement from at least six of seven experts.Experts annotated each video frame as one of seven expressions or none.
- Action Unit Annotation: Action-unit annotations cover 534 videos and 2,565,169 frames across 426 subjects using a semiautomatic frame-by-frame procedure for twelve action units.The associated action units and annotation distributions are provided in Table 2.
3. Evaluation Metrics Per Challenge
ABAW2 evaluates three affect-analysis Challenges with task-specific metrics: CCC for valence-arousal estimation, weighted F1 and total accuracy for expression classification, and averaged AU F1 with total accuracy for AU detection.
- Valence-Arousal Estimation Challenge: The mean CCC for valence and arousal estimation is the main evaluation criterion.CCC measures agreement between annotation and prediction time series and penalizes correlated predictions shifted in value.
- Seven Basic Expression Classification Challenge: Total accuracy is defined as the fraction of predictions that are correct across all test samples.Its range runs from 0 to 1, with 1 as the best value.
- Seven Basic Expression Classification Challenge: Expression classification uses per-frame emotion predictions and a weighted average of F1 score and total accuracy.F1 combines precision and recall, while total accuracy is the fraction of correct predictions across test samples.
- Twelve Action Unit Detection Challenge: AU detection computes an unweighted mean F1 score across all 12 action units and averages it with total accuracy.AU F1 is calculated from per-frame presence-or-absence detections.
4. Baseline & Participating Teams’ Systems and Results
The baseline uses a shared VGG-FACE architecture across all three Challenges, changing only the task-specific output activation. It is evaluated on Aff-Wild2 validation data using task-specific metrics.
- Baseline Systems: Input images are cropped and aligned Aff-Wild2 faces with dimensions 112 × 112 × 3 and pixel intensities normalized to [-1,1], without data augmentation.The preprocessing uses the provided cropped and aligned images.
- Baseline Systems: All three baseline systems use VGG-FACE convolutional and pooling layers followed by two fully connected layers with 4096 hidden units each.The fully connected layers of VGG-FACE are discarded, while its 13 convolutional and pooling layers are retained.
- Baseline Systems: Valence-arousal estimation uses a linear output layer, seven-expression classification uses softmax, and 12-action-unit detection uses sigmoid.These task-specific output layers produce estimates, expression predictions, and action-unit predictions, respectively.
- Baseline Systems: The baselines are pretrained on VGG-Face, freeze their convolutional layers, and train only the three fully connected layers on Aff-Wild2.Training was implemented in TensorFlow with learning rate 10^-4 and batch size 256.
- Results: Validation performance is reported separately for valence-arousal estimation, seven-expression classification, and 12-action-unit detection using their corresponding evaluation metrics.The VA metric is CCC, expression classification uses a weighted F1 and total-accuracy measure, and AU detection uses the average of F1 and total accuracy.
5. Conclusion
The paper presents ABAW2 as a competition with three affective behavior challenges based on Aff-Wild2. It also presents baseline networks and their results for these tasks.
- 5. Conclusion: ABAW2 comprises challenges for valence-arousal estimation, seven basic expression classification, and action unit detection.The conclusion describes the competition as based on the large-scale Aff-Wild2 database.
- 5. Conclusion: Aff-Wild2 is described as the first database annotated for all three behavior tasks used in the competition.The three tasks are valence-arousal estimation, basic expression classification, and action unit detection.
- 5. Conclusion: The paper presents baseline networks and their results for the ABAW2 challenges.