PaperlayerView exact revision on arXiv

Source-linked AI summary

Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition

Dimitrios Kollias, Stefanos Zafeiriou

arXiv:1811.07770v2cs.CVcs.AIcs.LGstat.ML

TL;DR

The paper addresses the challenge of representing complex, real-world affect and limited diversity in existing emotion databases. It extends Aff-Wild into Aff-Wild2 and trains CNN-RNN architectures with attention for continuous valence and arousal estimation. The reported results support using the extended database and architectures for visual analysis of continuous emotion dimensions.

  • Problem

    Existing expression categories and many emotion databases inadequately capture complex, continuous affect under diverse real-world conditions.

  • Method

    The paper constructs Aff-Wild2 by extending Aff-Wild and develops CNN-RNN architectures with attention for valence and arousal estimation.

  • Results

    Fine-tuning networks trained on Aff-Wild2 on RECOLA outperforms state-of-the-art networks trained on that database.

  • Takeaways & Limitations

    Aff-Wild2 and the presented deep architectures show potential for visual analysis of human behavior in continuous emotion dimensions.

Abstract

from arXiv · show

Automatic understanding of human affect using visual signals is a problem that has attracted significant interest over the past 20 years. However, human emotional states are quite complex. To appraise such states displayed in real-world settings, we need expressive emotional descriptors that are capable of capturing and describing this complexity. The circumplex model of affect, which is described in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion), can be used for this purpose. Recent progress in the emotion recognition domain has been achieved through the development of deep neural architectures and the availability of very large training databases. To this end, Aff-Wild has been the first large-scale "in-the-wild" database, containing around 1,200,000 frames. In this paper, we build upon this database, extending it with 260 more subjects and 1,413,000 new video frames. We call the union of Aff-Wild with the additional data, Aff-Wild2. The videos are downloaded from Youtube and have large variations in pose, age, illumination conditions, ethnicity and profession. Both database-specific as well as cross-database experiments are performed in this paper, by utilizing the Aff-Wild2, along with the RECOLA database. The developed deep neural architectures are based on the joint training of state-of-the-art convolutional and recurrent neural networks with attention mechanism; thus exploiting both the invariant properties of convolutional features, while modeling temporal dynamics that arise in human behaviour via the recurrent layers. The obtained results show premise for utilization of the extended Aff-Wild, as well as of the developed deep neural architectures for visual analysis of human behaviour in terms of continuous emotion dimensions.

I. INTRODUCTION

The paper addresses limitations of discrete and lab-controlled facial-expression analysis by extending large-scale in-the-wild data and using dimensional emotion modeling. It introduces Aff-Wild2 and CNN-RNN-attention architectures for continuous valence and arousal estimation.

  • I. INTRODUCTION: Discrete universal expressions and binary Action Units cannot adequately represent complex affective states or varying expression intensity.The paper identifies expression intensity as a crucial cue for interpretation.
  • I. INTRODUCTION: Dimensional theory represents emotional states continuously through valence and arousal, enabling modeling of subtle and complicated affective behaviors.Valence measures pleasure, while arousal indicates affective activation.
  • I. INTRODUCTION: Existing dimensional emotion databases are often small, controlled, and limited in subject diversity, pose variation, occlusions, backgrounds, and illumination.Aff-Wild was introduced as the first large-scale in-the-wild database, but the paper extends it further.
  • I. INTRODUCTION: Aff-Wild2 approximately doubles Aff-Wild’s video frames and subjects, increasing behavioral and participant variability.The union is paired with end-to-end CNN-RNN architectures and an attention mechanism for valence and arousal estimation.
  • I. INTRODUCTION: Fine-tuning networks trained on Aff-Wild2 on RECOLA substantially exceeds the performance of other networks trained on that database.The paper reports strong database-specific and cross-database results for continuous emotion dimensions.

II. THE AFF-WILD2 DATABASE

Aff-Wild2 combines the original Aff-Wild database with 260 newly collected videos annotated continuously for valence and arousal. The added data broaden subject and recording-condition diversity while preserving in-the-wild emotional variation.

  • B. Additional dataset and properties: Aff-Wild contained 298 videos, 200 subjects, more than 30 hours, and 1,180,000 frames annotated for valence and arousal.The database was created to capture spontaneous facial behaviors in arbitrary recording conditions.
  • B. Additional dataset and properties: 260 additional videos contribute 1,413,000 frames and 258 subjects, including 149 males and 109 females.Eleven videos contained two subjects, all of whom were annotated.
  • B. Additional dataset and properties: The added videos expand variation in age, ethnicity, profession, head pose, illumination, occlusion, and emotional states.Examples include subjects ranging from babies to elderly people and varied public professions.
  • B. Additional dataset and properties: Four annotators assigned continuous valence and arousal values in [−1, +1] through an online joystick procedure.Annotation post-processing considered either direct averaging or median filtering before averaging.
  • B. Additional dataset and properties: Annotated examples show multiple emotional states, rapid emotional changes, and facial occlusions, while reactions produce generally positive arousal and more positive than negative valence.Figure 3 illustrates temporal annotations alongside corresponding video frames; Figure 4 summarizes value distributions.

1) Pre-processing at video level:

The additional videos were cleaned, segmented, retained or split according to scene continuity, and standardized to 30 fps before integration into Aff-Wild2.

  • 241 YouTube videos were downloaded and converted to MP4 before trimming advertisements, caption-only scenes, and footage without human faces.
  • Videos were split when they contained different people or non-continuous appearances across time.
  • Videos containing two simultaneously visible people were retained to support learning from interpersonal interaction, yielding 260 videos after processing.
  • All additional videos were converted to a common frame rate of 30 fps.

1) The annotation procedure:

Four expert annotators continuously labeled valence and arousal, then verified their annotations and selected final labels using inter-annotator agreement.

  • The annotation procedure:: Valence and arousal were annotated continuously with a tool based on Feeltrace and Gtrace.
  • The annotation procedure:: Four computer-scientist experts with facial-expression knowledge performed the annotation task using shared instructions and emotional cues.
  • Annotation post-processing:: Annotators reviewed every video a second time to verify or revise labels before final annotation selection.
  • Annotation post-processing:: Two final-label strategies were compared: averaging four annotations, or median-filtering each annotation before averaging.
  • Annotation post-processing:: Median filtering with a 30-frame window produced the best reported MAIC values, 0.63 for valence and 0.60 for arousal.
  • Annotation post-processing:: A representative video segment had a valence MAIC of 0.64, similar to the mean MAIC across the additional data.

E. The properties of Aff-Wild2 database

Aff-Wild2 combines the original and additional data into a larger database with subject-independent splits and includes RECOLA as a related multimodal comparison dataset.

  • Aff-Wild2 contains 558 videos, 2,786,201 frames, and 458 subjects, including 279 male and 179 female subjects.
  • The database is partitioned subject-independently into 350 training, 70 validation, and 138 test videos.
  • The corresponding splits contain 1,601,000, 405,000, and 780,201 frames for training, validation, and testing, respectively.
  • RECOLA: RECOLA contains multimodal recordings from 46 French-speaking subjects collaborating in dyadic teams, annotated for valence and arousal.
  • RECOLA: RECOLA annotations cover five minutes per subject, and valence and arousal range continuously from −1 to +1.

III. FACIAL AND EMOTION ANALYSIS

The facial and emotion analysis uses face preprocessing followed by CNN-RNN architectures that combine visual representations, temporal modeling, and optional attention.

  • Faces are detected in every frame, resized to 96 × 96 × 3, and normalized to the range [−1, +1].
  • CNN-RNN architectures exploit temporal dynamics in videos through recurrent layers added to convolutional visual features.
  • Training used fixed CNN weights, joint CNN-RNN training from random RNN initialization, or joint training initialized from the best fixed-CNN model.
  • CNN part of the architectures:: The CNN experiments used VGGFACE, ResNet-50, and DenseNet-121 with alternative pre-training configurations.
  • RNN part of the architectures:: The recurrent experiments compared LSTM, GRU, and indRNN models and tested an attention layer on top of CNN-RNN architectures.
  • RNN part of the architectures:: The RNNs used two hidden layers with 128 hidden units each.

3) Best performing architecture/model:

The best Aff-Wild2 model combines VGGFACE convolutional features, GRU-based recurrent processing, and attention to predict valence and arousal. Performance is evaluated with CCC, while training minimizes a loss based on the two emotion dimensions.

  • Architecture: The best Aff-Wild2 architecture uses VGGFACE, GRUs, and an attention layer before valence and arousal predictions.GRUs receive the first fully connected layer’s output from VGGFACE.
  • Evaluation criterion: CCC measures agreement between predicted and annotated emotion values, ranging from −1 for perfect discordance to +1 for perfect concordance.Higher CCC values indicate better fit between predictions and annotations.
  • Evaluation criterion: The CCC formula uses ground-truth and predicted variances, means, and covariance.The symbols sx and sy denote variances, x̄ and ȳ means, and sxy covariance.
  • Training objective: Training uses a total loss formed from the CCC values for arousal and valence.The corresponding dimension-specific CCC terms are ρa and ρv.

IV. EXPERIMENTAL STUDY

Experiments compare CNN-RNN configurations across VGGFACE, ResNet, and DenseNet backbones and recurrent units, then test attention on the strongest models. The VGGFACE1-GRU-attention configuration achieves the best reported Aff-Wild2 test performance.

  • Model comparisons: The study compares CNN-RNN models using VGGFACE, ResNet, or DenseNet backbones with LSTM, GRU, or indRNN recurrent units.The reported metric is CCC for valence and arousal.
  • VGGFACE experiments: The VGGFACE1-GRU network provides the best performance among the VGGFACE configurations.Its CNN is pretrained on VGGFACE.
  • ResNet experiments: The ResNet2-GRU network provides the best performance among the ResNet configurations.It is pretrained on VGGFACE2.
  • DenseNet experiments: The DenseNet-GRU network provides the best performance among the examined DenseNet configurations.DenseNet-121 is paired with recurrent models based on LSTM, GRU, or indRNN units.
  • Attention comparison: 0.55 valence and 0.45 arousal are obtained by VGGFACE1-GRU-attention on the test set.Attention is added above the GRU part of the best-performing CNN-RNN configurations.

C. Experiments on RECOLA

The best Aff-Wild2 model is fine-tuned on RECOLA and compared with two networks previously achieving strong performance there. It outperforms both comparison networks for arousal and valence estimation.

  • Cross-database evaluation: The VGGFACE1-GRU-attention model is fine-tuned on RECOLA after training on Aff-Wild2.The comparison includes an AffWildNet model and a ResNet-50-GRU network trained on RECOLA.
  • Results: The fine-tuned VGGFACE1-GRU-attention architecture outperforms both comparison networks for arousal and valence estimation.No data augmentation or post-processing techniques are used during training.

V. CONCLUSIONS AND FUTURE WORK

The paper introduces Aff-Wild2, an expanded Aff-Wild database, and evaluates CNN-RNN architectures for continuous valence and arousal estimation. VGGFACE-GRU-attention performs best on Aff-Wild2 and transfers successfully to RECOLA.

  • Conclusions: Aff-Wild2 nearly doubles the videos, frames, and subjects included in Aff-Wild.The extension is designed to increase the variability of behaviors and people represented.
  • Conclusions: VGGFACE-GRU-attention achieves the best valence and arousal estimation performance on Aff-Wild2.The model combines a CNN-RNN network with an attention layer.
  • Conclusions: After fine-tuning on other emotion databases such as RECOLA, the architecture outperforms state-of-the-art networks trained on those databases.The conclusion describes transfer from a richly trained database to existing emotion databases.
  • Future work: Future work will add audio and text modalities, additional emotion representations, and ensembles of deep neural networks.Proposed representations include action units and complex emotion categories.
Loading 1811.07770v2…