Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition

I. INTRODUCTION

• Discrete universal expressions and binary Action Units cannot adequately represent complex affective states or varying expression intensity.The paper identifies expression intensity as a crucial cue for interpretation.Source: §I · p. 1
• Dimensional theory represents emotional states continuously through valence and arousal, enabling modeling of subtle and complicated affective behaviors.Valence measures pleasure, while arousal indicates affective activation.Source: §I · p. 1
• Existing dimensional emotion databases are often small, controlled, and limited in subject diversity, pose variation, occlusions, backgrounds, and illumination.Aff-Wild was introduced as the first large-scale in-the-wild database, but the paper extends it further.Source: §I · p. 1
• Aff-Wild2 approximately doubles Aff-Wild’s video frames and subjects, increasing behavioral and participant variability.The union is paired with end-to-end CNN-RNN architectures and an attention mechanism for valence and arousal estimation.Source: §I · pp. 1–2
• Fine-tuning networks trained on Aff-Wild2 on RECOLA substantially exceeds the performance of other networks trained on that database.The paper reports strong database-specific and cross-database results for continuous emotion dimensions.Source: §I · p. 2

B. Additional dataset and properties

• Aff-Wild contained 298 videos, 200 subjects, more than 30 hours, and 1,180,000 frames annotated for valence and arousal.The database was created to capture spontaneous facial behaviors in arbitrary recording conditions.Source: A. The existing Aff-Wild database · p. 2
• 260 additional videos contribute 1,413,000 frames and 258 subjects, including 149 males and 109 females.Eleven videos contained two subjects, all of whom were annotated.Source: B. Additional dataset and properties · p. 2
• The added videos expand variation in age, ethnicity, profession, head pose, illumination, occlusion, and emotional states.Examples include subjects ranging from babies to elderly people and varied public professions.Source: B. Additional dataset and properties · p. 2
• Four annotators assigned continuous valence and arousal values in [−1, +1] through an online joystick procedure.Annotation post-processing considered either direct averaging or median filtering before averaging.Source: B. Additional dataset and properties · p. 3
• Annotated examples show multiple emotional states, rapid emotional changes, and facial occlusions, while reactions produce generally positive arousal and more positive than negative valence.Figure 3 illustrates temporal annotations alongside corresponding video frames; Figure 4 summarizes value distributions.Source: B. Additional dataset and properties · p. 3

• 241 YouTube videos were downloaded and converted to MP4 before trimming advertisements, caption-only scenes, and footage without human faces.Source: 1) Pre-processing at video level · p. 3
• Videos were split when they contained different people or non-continuous appearances across time.Source: 1) Pre-processing at video level · p. 3
• Videos containing two simultaneously visible people were retained to support learning from interpersonal interaction, yielding 260 videos after processing.Source: 1) Pre-processing at video level · p. 3
• All additional videos were converted to a common frame rate of 30 fps.Source: 2) Frame-rate conversion · p. 3

The annotation procedure:

• Valence and arousal were annotated continuously with a tool based on Feeltrace and Gtrace.Source: 1) The annotation procedure · pp. 3–4
• Four computer-scientist experts with facial-expression knowledge performed the annotation task using shared instructions and emotional cues.Source: 1) The annotation procedure · p. 4

Annotation post-processing:

• Annotators reviewed every video a second time to verify or revise labels before final annotation selection.Source: 2) Annotation post-processing · p. 4
• Two final-label strategies were compared: averaging four annotations, or median-filtering each annotation before averaging.Source: 2) Annotation post-processing · p. 4
• Median filtering with a 30-frame window produced the best reported MAIC values, 0.63 for valence and 0.60 for arousal.Source: 2) Annotation post-processing · p. 4
• A representative video segment had a valence MAIC of 0.64, similar to the mean MAIC across the additional data.Source: Figure 5 · p. 4, 2) Annotation post-processing · p. 4

• Aff-Wild2 contains 558 videos, 2,786,201 frames, and 458 subjects, including 279 male and 179 female subjects.Source: E. The properties of Aff-Wild2 database · p. 4
• The database is partitioned subject-independently into 350 training, 70 validation, and 138 test videos.Source: F. Database partition sets · p. 4
• The corresponding splits contain 1,601,000, 405,000, and 780,201 frames for training, validation, and testing, respectively.Source: F. Database partition sets · p. 4

RECOLA

• RECOLA contains multimodal recordings from 46 French-speaking subjects collaborating in dyadic teams, annotated for valence and arousal.Source: G. Other Databases: RECOLA · p. 4
• RECOLA annotations cover five minutes per subject, and valence and arousal range continuously from −1 to +1.Source: G. Other Databases: RECOLA · pp. 4–5

• Faces are detected in every frame, resized to 96 × 96 × 3, and normalized to the range [−1, +1].Source: A. Face detection · p. 5
• CNN-RNN architectures exploit temporal dynamics in videos through recurrent layers added to convolutional visual features.Source: B. The CNN-RNN Deep Neural Architectures · p. 5
• Training used fixed CNN weights, joint CNN-RNN training from random RNN initialization, or joint training initialized from the best fixed-CNN model.Source: B. The CNN-RNN Deep Neural Architectures · p. 5

CNN part of the architectures:

• The CNN experiments used VGGFACE, ResNet-50, and DenseNet-121 with alternative pre-training configurations.Source: 1) CNN part of the architectures · p. 5, Table III · p. 5

RNN part of the architectures:

• The recurrent experiments compared LSTM, GRU, and indRNN models and tested an attention layer on top of CNN-RNN architectures.Source: 2) RNN part of the architectures · p. 5
• The RNNs used two hidden layers with 128 hidden units each.Source: 2) RNN part of the architectures · p. 5

Architecture

• The best Aff-Wild2 architecture uses VGGFACE, GRUs, and an attention layer before valence and arousal predictions.GRUs receive the first fully connected layer’s output from VGGFACE.Source: 3) Best performing architecture/model · p. 5

Evaluation criterion

• CCC measures agreement between predicted and annotated emotion values, ranging from −1 for perfect discordance to +1 for perfect concordance.Higher CCC values indicate better fit between predictions and annotations.Source: C. The evaluation criterion · p. 5
• The CCC formula uses ground-truth and predicted variances, means, and covariance.The symbols sx and sy denote variances, x̄ and ȳ means, and sxy covariance.Source: C. The evaluation criterion · p. 5

Training objective

• Training uses a total loss formed from the CCC values for arousal and valence.The corresponding dimension-specific CCC terms are ρa and ρv.Source: C. The evaluation criterion · p. 5

Model comparisons

• The study compares CNN-RNN models using VGGFACE, ResNet, or DenseNet backbones with LSTM, GRU, or indRNN recurrent units.The reported metric is CCC for valence and arousal.Source: B. Experiments on Aff-Wild2 · p. 6, Table IV · p. 6

VGGFACE experiments

• The VGGFACE1-GRU network provides the best performance among the VGGFACE configurations.Its CNN is pretrained on VGGFACE.Source: B. Experiments on Aff-Wild2 · p. 6

ResNet experiments

• The ResNet2-GRU network provides the best performance among the ResNet configurations.It is pretrained on VGGFACE2.Source: B. Experiments on Aff-Wild2 · p. 6, Table V · p. 6

DenseNet experiments

• The DenseNet-GRU network provides the best performance among the examined DenseNet configurations.DenseNet-121 is paired with recurrent models based on LSTM, GRU, or indRNN units.Source: B. Experiments on Aff-Wild2 · p. 6, Table VI · p. 6

Attention comparison

• 0.55 valence and 0.45 arousal are obtained by VGGFACE1-GRU-attention on the test set.Attention is added above the GRU part of the best-performing CNN-RNN configurations.Source: B. Experiments on Aff-Wild2 · p. 6, Table VII · p. 6

Cross-database evaluation

• The VGGFACE1-GRU-attention model is fine-tuned on RECOLA after training on Aff-Wild2.The comparison includes an AffWildNet model and a ResNet-50-GRU network trained on RECOLA.Source: C. Experiments on RECOLA · pp. 6–7

Results

• The fine-tuned VGGFACE1-GRU-attention architecture outperforms both comparison networks for arousal and valence estimation.No data augmentation or post-processing techniques are used during training.Source: C. Experiments on RECOLA · p. 7, Table VIII · p. 7

Conclusions

• Aff-Wild2 nearly doubles the videos, frames, and subjects included in Aff-Wild.The extension is designed to increase the variability of behaviors and people represented.Source: §V · p. 7
• VGGFACE-GRU-attention achieves the best valence and arousal estimation performance on Aff-Wild2.The model combines a CNN-RNN network with an attention layer.Source: §V · p. 7
• After fine-tuning on other emotion databases such as RECOLA, the architecture outperforms state-of-the-art networks trained on those databases.The conclusion describes transfer from a richly trained database to existing emotion databases.Source: §V · p. 7

Future work

• Future work will add audio and text modalities, additional emotion representations, and ensembles of deep neural networks.Proposed representations include action units and complex emotion categories.Source: §V · p. 7