Source-linked AI summary
Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
Dimitrios Kollias, Stefanos Zafeiriou
TL;DR
Affective computing lacked sufficiently large, diverse, audiovisual datasets annotated across valence-arousal, action units, and expressions. The paper creates Aff-Wild2, trains multi-task and multimodal CNN-based models on it, and adapts ArcFace for emotion recognition. These models achieve state-of-the-art performance across emotion-recognition evaluations, while ArcFace-based networks improve existing state-of-the-art results on expression databases.
Problem
Existing affective-computing databases are limited in size, audiovisual coverage, manual annotation, subject diversity, or annotation across all three main behavior tasks.
Method
The paper extends Aff-Wild into Aff-Wild2, annotates valence-arousal, action units, and expressions, trains multi-task CNN/CNN-RNN audiovisual models, and adapts ArcFace for emotion recognition.
Results
The trained models achieve state-of-the-art emotion-recognition performance across 10 public databases, while ArcFace-based networks improve existing state-of-the-art expression-recognition results.
Takeaways & Limitations
Aff-Wild2 supports joint study of valence-arousal estimation, action-unit detection, and expression classification in a large audiovisual in-the-wild database.
Takeaways & Limitations
Expression-set images could also receive action-unit annotations, but extending them is outside the paper’s scope; some benchmark test sets are unavailable and validation sets are used instead.
Abstract
from arXiv · showhide
Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild database (Aff-Wild) to study continuous emotions such as valence and arousal. Furthermore, we annotate parts of the database with basic expressions and action units. As a consequence, for the first time, this allows the joint study of all three types of behavior states. We call this database Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures that use visual and audio modalities; these networks are trained on Aff-Wild2 and their performance is then evaluated on 10 publicly available emotion databases. We show that the networks achieve state-of-the-art performance for the emotion recognition tasks. Additionally, we adapt the ArcFace loss function in the emotion recognition context and use it for training two new networks on Aff-Wild2 and then re-train them in a variety of diverse expression recognition databases. The networks are shown to improve the existing state-of-the-art. The database, emotion recognition models and source code are available at http://ibug.doc.ic.ac.uk/resources/aff-wild2.
1 Introduction
Affective computing is moving from controlled, limited datasets toward diverse in-the-wild data, but existing resources remain fragmented across modalities, scale, subjects, and behavior tasks. Aff-Wild2 addresses these gaps while supporting multi-task emotion recognition and ArcFace-based modeling.
- In-the-wild datasets are increasingly needed because affective computing models are intended for diverse human participants and spontaneous expressions.
- Existing databases commonly have limited manually annotated data, static rather than audiovisual content, few subjects, or incomplete behavior-task coverage.
- Aff-Wild2 combines valence-arousal, action-unit, and basic-expression annotations, enabling joint study of all three main behavior tasks.The database includes about 398K AU annotations and 403K expression annotations in addition to VA annotations.
- The paper develops multi-task CNN, CNN-RNN, and audiovisual CNN-RNN models trained on Aff-Wild2 and evaluated across 10 public emotion databases.The task losses include binary cross entropy for AUs, MSE and CCC losses for VA, and categorical cross entropy for expression classification.
- The authors adapt ArcFace from face recognition to emotion recognition to seek greater discriminative power than categorical cross entropy.They motivate the adaptation by prior findings that categorical cross entropy is insufficient for discriminating face classes.
2 The Aff-Wild2 database
Aff-Wild2 extends Aff-Wild with new in-the-wild videos and annotations for valence-arousal, action units, and basic expressions. The database combines these datasets with subject-independent partitions and expert annotations.
- Collected dataset and properties: Aff-Wild2 combines Aff-Wild with 260 new YouTube videos, totaling 558 videos, 2,786,201 frames, and 458 subjects.The new videos span subtle and extreme human behaviours in real-world settings.
- Annotation: Parts of Aff-Wild2 were additionally annotated for eight action units across 63 videos and seven basic expressions across 84 videos.These annotations create separate AU and expression datasets alongside the VA set.
- Partition Sets and Distributions: Each VA, AU, and expression set is divided into training, validation, and test subsets using subject-independent partitioning.The VA partitions contain 350, 70, and 138 videos; AU contains 42, 7, and 14; expression contains 51, 11, and 22.
- Scope: The expression image set could also include AU annotations, but extending it is outside the paper’s scope.This boundary is stated explicitly in the database description.
- Annotation: Four experts annotated the new dataset for valence and arousal before it was concatenated with Aff-Wild.Valence and arousal labels were produced from expert annotations, with values ranging continuously from -1 to 1.
3 Proposed Methods
The paper develops visual, audio, recurrent, multi-task, and ArcFace-based networks for jointly recognizing valence-arousal, action units, and expressions.
- Visual Modality Pre-Processing: The visual pipeline detects and aligns faces using five landmarks, then resizes normalized crops to 96 × 96 × 3.Face bounding boxes are extracted with an SSH detector, and pixel intensities are normalized to [-1,1].
- Audio Modality Pre-Processing: The audio pipeline samples mono signals at 44,100Hz and extracts normalized spectrograms from 33ms windows with 11ms overlap.The resulting audio intensities are normalized to [-1,1] to match the visual modality.
- CNN Single- & Multi-Task: CNN models perform one or all three behavior tasks using predictions pooled from a shared feature space.SphereFace-20, VGGFace, and Inception ResNet are trained in single-task and multi-task settings.
- CNN-RNN Multi-Task: MT-VGG-RNN stacks a two-layer GRU on MT-VGG to capture temporal dynamics.Each GRU layer has 128 cells, and the output layer follows the recurrent component.
- CNN-RNN Multi-Modal (A/V) & Multi-Task: A/V-MT-VGG-RNN fuses visual and audio features by concatenating two modality-specific streams before recurrent integration.The concatenated representation is 256-dimensional and is passed through a two-layer GRU with 128 units per layer.
- Standard Loss Functions: The multi-task objective minimizes the sum of the individual losses for valence-arousal estimation, action-unit detection, and expression classification.The losses include MSE and CCC for valence-arousal, binary cross entropy for action units, and categorical cross entropy for expressions.
- ArcFace Loss Function & Networks: ArcFace modifies softmax cross-entropy with an angular margin that promotes intra-class compactness and inter-class discrepancy.The margin m changes θ_yi to θ_yi + m after feature and weight normalization, with feature magnitude re-scaled to s.
- ArcFace Loss Function & Networks: MT-ArcRes uses residual units with an embedding layer and seven expression-class logits to implement the ArcFace-based multi-task architecture.MT-ArcVGG replaces the corresponding rectangular network block with VGGFace layers.
4 Experimental Study
The study evaluates multi-task and multi-modal networks trained on Aff-Wild2 through cross-database tests, and examines ArcFace-based expression recognition. Across tasks and databases, the proposed frameworks generally improve on existing systems, with task- and modality-specific exceptions.
- Experimental design: Networks trained on Aff-Wild2 are evaluated across 10 emotion databases using visual, audio, CNN, CNN-RNN, multi-task, and multi-modal configurations.The experiments cover valence-arousal estimation, action unit detection, and expression recognition.
- Cross-database evaluation: MT-VGG outperforms the state of the art on static-database cross-evaluation for valence-arousal and expression recognition, except expression recognition on AffectNet.The evaluated static databases are AffectNet, RAF-DB, FER2013, and IMFDB.
- Cross-database evaluation: MT-VGG-RNN performs better than MT-VGG for valence-arousal and expression recognition across the examined video databases.The comparison includes Aff-Wild, Aff-Wild2, and AFEW-VA.
- Cross-database evaluation: Audio-visual MT-VGG-RNN improves over visual MT-VGG-RNN for valence, arousal, and expression estimation, while visual input is best for valence and audio input for arousal.The paper attributes the arousal pattern to audio’s thematic constancy in examples such as contrasting fight sequences.
- Cross-database evaluation: MT-VGG generally outperforms compared AU-detection systems across Emotionet, DISFA, BP4DS, and BP4D+, but is slightly worse than the FERA 2017 winner.The paper reports a performance boost in all cases except Emotionet.
- ArcFace expression recognition: ArcFace networks MT-ArcRes and MT-ArcVGG outperform the compared systems, including DLP-CNN, after training on Aff-Wild2 and retraining on expression databases.The comparison includes fine-tuned MT-VGG and state-of-the-art results from the examined databases.
5 Conclusions
The paper introduces Aff-Wild2 as an in-the-wild audiovisual database annotated for valence-arousal, action units, and expressions. Networks trained with it achieve strong cross-database results, while ArcFace-based networks improve expression-recognition performance over existing state of the art.
- Conclusions: Aff-Wild2 is presented as the first and largest in-the-wild audiovisual database annotated for valence-arousal, action units, and expressions.The paper combines dataset construction with multi-task and multi-modal network development.
- Conclusions: Multi-task and multi-modal CNN and CNN-RNN models trained on Aff-Wild2 are tested on 10 databases and reported to beat the state of the art.The evaluation covers the paper’s emotion-recognition tasks.
- Conclusions: Two ArcFace-loss networks trained on Aff-Wild2 and retrained on diverse expression databases improve the existing state of the art.The conclusion specifically identifies ArcFace as the training loss used for these networks.