Source-linked AI summary
ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges
Dimitrios Kollias
TL;DR
The paper addresses automatic affect analysis in unconstrained, real-world settings across categorical and dimensional facial behavior representations. It presents the third ABAW Competition, defining four challenges on the Aff-Wild2 benchmark and reporting evaluation metrics, baseline systems, and results.
Problem
Automatic affect analysis remains important for behavioral modeling, human-computer interaction, and affective computing, including recognition in unconstrained environments.
Method
The paper presents four ABAW challenges using the common Aff-Wild2 database: valence-arousal estimation, expression classification, action unit detection, and multi-task learning, with corpora, metrics, and baseline systems.
Results
The paper reports baseline evaluation results for all four challenges on Aff-Wild2 validation data.
Takeaways & Limitations
Aff-Wild2 provides a common benchmark for valence-arousal estimation, expression classification, and action unit detection, supporting the four ABAW competition challenges.
Abstract
from arXiv · showhide
This paper describes the third Affective Behavior Analysis in-the-wild (ABAW) Competition, held in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. The 3rd ABAW Competition is a continuation of the Competitions held at ICCV 2021, IEEE FG 2020 and IEEE CVPR 2017 Conferences, and aims at automatically analyzing affect. This year the Competition encompasses four Challenges: i) uni-task Valence-Arousal Estimation, ii) uni-task Expression Classification, iii) uni-task Action Unit Detection, and iv) Multi-Task-Learning. All the Challenges are based on a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the first one to be annotated in terms of valence-arousal, expressions and action units. In this paper, we present the four Challenges, with the utilized Competition corpora, we outline the evaluation metrics and present the baseline systems along with their obtained results.
1. Introduction
The paper situates affect recognition as important for behavioral modelling and human–machine interaction, while highlighting the shift toward in-the-wild analysis enabled by larger datasets and deep learning. The third ABAW Competition organizes this effort into four challenges using the Aff-Wild2 benchmark.
- Affect recognition supports behavioral modelling, human–machine interaction, and applications including medicine, driver monitoring, e-learning, and entertainment.
- Large real-life datasets and deep learning have expanded affect analysis from controlled environments to unconstrained, in-the-wild settings.
- Dimensional emotion representations describe affect continuously through valence and arousal, corresponding to positive–negative and active–passive behavior.
- The third ABAW Competition includes uni-task Valence-Arousal Estimation, Expression Classification, Action Unit Detection, and Multi-Task Learning Challenges.
- The challenges use Aff-Wild2, the first comprehensive in-the-wild benchmark covering valence-arousal, expressions, and action units.
- The paper introduces the competition corpora, evaluation metrics, baseline systems, obtained results, and concluding discussion.
2. Competition Corpora
Aff-Wild2 provides shared in-the-wild corpora for valence-arousal estimation, expression recognition, and action-unit detection, with annotations, face preprocessing, and a static image version supporting the competition challenges.
- Aff-Wild2 is annotated for valence-arousal estimation, expression classification, and action-unit detection, forming the basis of four competition challenges.
- The database is divided into training, validation, and test sets, with public training and validation annotations and supplied face bounding boxes and landmarks.
- The competition overview describes preprocessing that crops and aligns faces before using the resulting images in baseline experiments.
- Valence-Arousal Estimation Challenge: 567 videos provide valence-arousal annotations across 2,816,832 frames from 455 subjects, with values continuously ranging from −1 to 1.
- Expression Recognition Challenge: 548 videos provide frame-level annotations for six basic expressions, neutral, and other across 2,603,921 frames and 431 subjects.
- Action Unit Detection Challenge: 547 videos provide frame-level annotations for 12 action units across 2,603,921 frames and 431 subjects using semi-automatic annotation.
- Multi-Task Learning Challenge: The static s-Aff-Wild2 version contains 172,360 selected images annotated for valence-arousal, expressions, and 12 action units.
- Cropped and aligned images are resized to 112 × 112 × 3 pixels and normalized to [−1, 1].
3. Evaluation Metrics Per Challenge
The competition evaluates four affect-analysis challenges with task-specific metrics: mean CCC for valence-arousal, macro F1 for expressions and action units, and a summed multi-task score.
- Valence-Arousal Estimation: Valence-arousal estimation uses the average CCC of valence and arousal predictions.CCC measures agreement between annotation and prediction time series and ranges from -1 to 1, with higher values preferred.
- Expression Recognition: Expression recognition is evaluated by the average F1 Score across all 8 categories, called macro F1 Score.F1 combines precision and recall and ranges from 0 to 1, with higher values preferred.
- Action Unit Detection: Action unit detection uses the average F1 Score across all 12 AUs as its performance measure.The resulting macro F1 criterion evaluates performance across the twelve action units.
- Multi-Task Learning: Multi-task learning is scored by summing average valence-arousal CCC, average expression F1 across 8 categories, and average action-unit F1 across 12 AUs.This criterion combines the three task-level measures into one score.
4. Baseline Networks and Results
The paper establishes reproducible TensorFlow baselines for the four challenges and reports their validation results on Aff-Wild2 using challenge-specific network designs and metrics.
- Baseline Setup: All baseline systems use existing open-source machine-learning toolkits and TensorFlow to support reproducibility.Training used approximately six hours on a Titan X GPU, a learning rate of 10^-4, and batch size 256.
- Valence-Arousal Estimation: The valence-arousal baseline is an ImageNet-pretrained 50-layer ResNet with a linear output layer for valence and arousal.Its validation results are reported using mean CCC of valence-arousal predictions.
- Action Unit Detection: The action-unit baseline is a VGG16 model with fixed convolutional weights and sigmoid outputs for 12 action-unit predictions.Only the three fully connected layers were trained, and validation performance is reported in the corresponding challenge table.
- Multi-Task Learning: The multi-task baseline is a VGG16 model with fixed convolutional weights and 22 outputs for valence, arousal, expressions, and action units.Its output layer contains 2 linear, 8 softmax, and 12 sigmoid units.
- Reported Results: Validation results are presented for the valence-arousal, expression, action-unit, and multi-task challenges in separate tables.The expression table uses average F1 Score of 8 expressions as its evaluation criterion.
5. Conclusion
The paper presents the third Affective Behavior Analysis in-the-wild Competition, held with IEEE CVPR 2022, as a continuation of earlier ABAW Competitions.
- The third ABAW Competition is presented as the paper’s central contribution.
- The Competition is scheduled in conjunction with IEEE CVPR 2022.
- The third Competition follows the first and second ABAW Competitions.
http://arxiv.org/ps/2202.10659v2
The referenced figures are available as JPG files, including a whole-database Aff-Wild2 histogram and a valence-arousal-space visualization.
- The whole-Aff-Wild2 histogram figure is available in JPG format.
- The two referenced visualizations are separately identified by their figure filenames.
- The valence-arousal-space figure is available in JPG format.