Source-linked AI summary
ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges
Dimitrios Kollias
TL;DR
The paper presents the fourth ABAW Competition to evaluate affect analysis across multiple facial-behavior tasks and synthetic-to-real expression recognition. It defines two Challenges using Aff-Wild2-derived corpora: joint multi-task learning and learning basic expressions from synthetic data, with real data used for LSD validation and testing. The Competition therefore organizes a common benchmark around complementary task and data settings.
Problem
Affect analysis requires evaluating valence-arousal estimation, expression recognition, and action-unit detection, but the Competition’s new settings require these tasks to be learned jointly and expressions to transfer from synthetic to real data.
Method
The paper presents two Challenges based on Aff-Wild2: s-Aff-Wild2 for joint learning and manipulated Aff-Wild2 frames for synthetic-expression training.
Results
The paper reports the Competition corpora, evaluation metrics, baseline systems, and baseline results for the Multi-Task-Learning and Learning from Synthetic Data Challenges.
Takeaways & Limitations
The fourth ABAW Competition provides a shared Aff-Wild2-based benchmark spanning joint affect-task learning and synthetic-to-real expression recognition.
Takeaways & Limitations
LSD model refinement and methodology development must use only synthetic data, while MTL participants may not use the audiovisual Aff-Wild2 database.
Abstract
from arXiv · showhide
This paper describes the fourth Affective Behavior Analysis in-the-wild (ABAW) Competition, held in conjunction with European Conference on Computer Vision (ECCV), 2022. The 4th ABAW Competition is a continuation of the Competitions held at IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 and IEEE CVPR 2017 Conferences, and aims at automatically analyzing affect. In the previous runs of this Competition, the Challenges targeted Valence-Arousal Estimation, Expression Classification and Action Unit Detection. This year the Competition encompasses two different Challenges: i) a Multi-Task-Learning one in which the goal is to learn at the same time (i.e., in a multi-task learning setting) all the three above mentioned tasks; and ii) a Learning from Synthetic Data one in which the goal is to learn to recognise the basic expressions from artificially generated data and generalise to real data. The Aff-Wild2 database is a large scale in-the-wild database and the first one that contains annotations for valence and arousal, expressions and action units. This database is the basis for the above Challenges. In more detail: i) s-Aff-Wild2 -- a static version of Aff-Wild2 database -- has been constructed and utilized for the purposes of the Multi-Task-Learning Challenge; and ii) some specific frames-images from the Aff-Wild2 database have been used in an expression manipulation manner for creating the synthetic dataset, which is the basis for the Learning from Synthetic Data Challenge. In this paper, at first we present the two Challenges, along with the utilized corpora, then we outline the evaluation metrics and finally present the baseline systems per Challenge, as well as their derived results. More information regarding the Competition can be found in the competition's website: https://ibug.doc.ic.ac.uk/resources/eccv-2023-4th-abaw/.
1 Introduction
The fourth ABAW Competition extends prior affect-analysis challenges with two new settings: joint learning of three facial-behavior tasks and learning expressions from synthetic data for real-world generalization. It builds on established categorical, dimensional, and action-unit approaches to affective behavior analysis.
- Affect-analysis tasks: Affect analysis spans categorical expressions, continuous valence-arousal dimensions, and action units that represent physical manifestations of facial expressions.The dimensional model is intended to represent both extreme and subtle affective states.
- The fourth ABAW Competition continues earlier competitions covering valence-arousal estimation, basic-expression recognition, and facial action-unit analysis.
- Multi-Task-Learning Challenge: The Multi-Task-Learning Challenge jointly estimates valence and arousal, classifies eight expressions, and detects twelve action units.The eight expressions comprise six basic expressions, neutral, and other.
- Learning from Synthetic Data Challenge: The Learning from Synthetic Data Challenge trains on artificially generated data to recognize six basic expressions and generalize to real-world data.
- Competition basis: Both Challenges use Aff-Wild2, a comprehensive in-the-wild benchmark with annotations for valence-arousal, expressions, and action units.The MTL corpus is a static version, while the LSD corpus is created by manipulating expressions in selected Aff-Wild2 frames.
2 Competition Corpora
The Competition corpora derive from Aff-Wild2 and provide task-specific static or synthetic datasets. The MTL corpus combines three annotation types, while the LSD corpus uses manipulated expression images for training and real images for evaluation.
- Aff-Wild2 basis: Aff-Wild2 is the first comprehensive in-the-wild benchmark annotated for valence-arousal estimation, expression recognition, and action-unit detection.
- Multi-Task Learning Challenge: The MTL corpus, s-Aff-Wild2, contains 221,928 selected images annotated with valence-arousal, eight expressions, and twelve action units.The expression labels include six basic expressions, neutral, and other.
- Multi-Task Learning Challenge: The s-Aff-Wild2 corpus is split into training, validation, and test sets, with annotations initially released for training and validation.
- Preprocessing: Participants receive cropped and cropped-aligned s-Aff-Wild2 images, with aligned images standardized to 112 × 112 × 3 dimensions.RetinaFace supplies face bounding boxes and five facial landmarks for preprocessing.
- Learning from Synthetic Data Challenge: The LSD corpus contains 277,251 synthetically manipulated images labeled with six basic expressions, while validation and test data are real Aff-Wild2 images.The synthetic images form the Challenge training set.
- Learning from Synthetic Data Challenge: LSD model development may use pre-trained models not trained on Aff-Wild2, but refinement and methodology development must use only the provided synthetic data.
3 Evaluation Metrics for each Challenge
The Challenges use task-specific metrics: CCC for valence and arousal, macro F1 for expressions and action units, and macro F1 for synthetic-data expression recognition.
- Multi-Task Learning Challenge: The Multi-Task-Learning criterion sums average valence-arousal CCC, macro F1 across eight expressions, and macro F1 across twelve action units.The criterion is expressed as P_MTL = (ρ_a + ρ_v)/2 + Σ_expr F_1^expr/8 + Σ_au F_1^au/12.
- Multi-Task Learning Challenge: CCC measures agreement between annotated and predicted valence or arousal, ranges from −1 to 1, and higher values are preferred.It uses variances, means, and covariance of the annotations and predictions.
- Evaluation measures: F1 combines precision and recall, ranges from 0 to 1, and higher values are preferred.The paper defines F1 as 2 × precision × recall divided by precision + recall.
- Learning from Synthetic Data Challenge: The Learning from Synthetic Data criterion is macro F1 averaged across the six basic expression categories.It is defined as P_LSD = Σ_expr F_1^expr/6.
4 Baseline Networks and Performance
The paper establishes reproducible TensorFlow baselines for both Challenges: a fixed-feature VGG16 multi-task model and an ImageNet-pretrained ResNet50 expression classifier evaluated on real Aff-Wild2 data.
- Baseline setup: All baselines use open-source machine-learning toolkits and TensorFlow, with approximately five hours of training on a Titan X GPU.The reported learning rate is 10^-4 and batch size is 128.
- Multi-Task Learning Challenge: The Multi-Task-Learning baseline uses VGG16 convolutional weights fixed after VGGFACE pretraining and trains three fully connected layers.Its 22-unit output contains two linear valence-arousal units, eight softmax expression units, and twelve sigmoid action-unit units.
- Multi-Task Learning Challenge: The Multi-Task-Learning baseline is evaluated on the s-Aff-Wild2 validation set using the sum of independent task metrics.No data augmentation is used, and image intensities are normalized to [−1, 1].
- Learning from Synthetic Data Challenge: The Learning from Synthetic Data baseline is an ImageNet-pretrained ResNet50 with a six-unit softmax output for basic-expression predictions.The synthetic-image training uses no data augmentation and normalizes pixel intensities to [−1, 1].
- Learning from Synthetic Data Challenge: Synthetic-data baseline performance is measured on validation and test sets containing only real Aff-Wild2 data using average F1 across six basic expressions.Validation performance is reported in parentheses in Table 5.
5 Conclusion
The fourth ABAW Competition defines two Aff-Wild2-based Challenges: joint learning of affective tasks and recognition of basic expressions from synthetic data with generalization to real data.
- Multi-Task-Learning Challenge: The Multi-Task-Learning Challenge jointly estimates valence and arousal, classifies eight expression categories, and detects twelve action units.The expression categories include six basic expressions, neutral, and other.
- Learning from Synthetic Data Challenge: The Learning from Synthetic Data Challenge trains recognition of six basic expressions on artificially generated data and evaluates generalization to real-world data.Both Challenge corpora are derived from Aff-Wild2.