Source-linked AI summary
Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, Stefanos Zafeiriou
TL;DR
The paper tackles the shortage of large, diverse, in-the-wild data for facial-affect recognition. It introduces the Aff-Wild benchmark and challenge, develops the CNN-RNN AffWildNet, and transfers learned features across dimensional and categorical tasks. AffWildNet achieves state-of-the-art or best reported performance across the evaluated settings, while the benchmark addresses important limitations of controlled and coarse datasets.
Problem
Most existing affect datasets contain naturalistic emotions but were recorded under controlled conditions, limiting coverage of real-world variation and continuous emotional expression.
Method
The paper collects and annotates Aff-Wild for continuous valence and arousal, trains the CNN-RNN AffWildNet, and uses its learned features as priors for other emotion-recognition datasets.
Results
AffWildNet provides best or state-of-the-art performance on Aff-Wild and achieves best or comparable-to-better results after fine-tuning on RECOLA, AFEW-VA, and EmotiW.
Takeaways & Limitations
A single architecture trained on large-scale in-the-wild continuous affect data can support both dimensional and categorical emotion recognition across datasets.
Takeaways & Limitations
Prior datasets remain constrained by controlled recording conditions, limited subject and pose diversity, occlusions, illumination variation, and short videos.
Abstract
from arXiv · showhide
Automatic understanding of human affect using visual signals is of great importance in everyday human-machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) & arousal (i.e., power of the activation of the emotion) constitute popular and effective affect representations. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge that was organized in conjunction with CVPR 2017 on the Aff-Wild database and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional & recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.
1 Introduction
The paper addresses the lack of large, diverse, uncontrolled datasets for automatic facial-affect analysis by introducing Aff-Wild and associated recognition models. It combines continuous valence–arousal annotation with CNN-RNN architectures and evaluates transfer to other dimensional and categorical emotion-recognition settings.
- Motivation: Valence and arousal provide a dimensional representation suited to subtle emotions in everyday human–computer interactions.Valence ranges from positive to negative, while arousal ranges from active to passive.
- Motivation: Existing facial-affect datasets largely use controlled recordings, limiting variation in subjects, poses, occlusions, illumination, backgrounds, and video duration.Naturalistic facial behavior can differ substantially from posed behavior, motivating in-the-wild data collection.
- Aff-Wild benchmark: Eight lay experts annotated Aff-Wild videos continuously for valence and arousal, and the authors organized the Aff-Wild Challenge with CVPR 2017.Challenge participants outperformed the provided baseline, while the authors subsequently developed stronger CNN and CNN-RNN systems.
- Models and transfer: AffWildNet combines convolutional features with recurrent temporal modeling and fuses facial images with facial-landmark positions for end-to-end affect prediction.The authors compare Concordance Correlation Coefficient and Mean Squared Error objectives and report strong performance on Aff-Wild.
- Models and transfer: Features learned from Aff-Wild are used as priors for dimensional recognition on RECOLA and AFEW-VA and categorical recognition on EmotiW.The paper presents this as the first use of a dimensional in-the-wild database and its features as priors for categorical emotion recognition in-the-wild.
2 Existing Databases
Existing emotion databases provide naturalistic or dimensional affect data, but remain limited by controlled recording conditions, small scale, restricted categories, or discrete annotations. Aff-Wild addresses these gaps with a larger in-the-wild collection of continuously annotated valence and arousal videos.
- RECOLA: RECOLA contains 46 subjects and 9.5 hours of multimodal recordings, with continuous arousal and valence annotations.Its training, validation, and test sets are stratified by gender, age, and mother tongue.
- RECOLA: RECOLA is constrained by its tightly controlled laboratory environment, small subject count, and moderate total number of frames.
- AFEW: AFEW contains 1,809 dynamic video clips from close-to-real-world movie and reality-TV scenes, but has limited frame count and only seven emotion categories.Some categories, including fear, disgust, and surprise, contain few samples.
- AFEW-VA: AFEW-VA provides per-frame valence and arousal annotations for more than 30,000 frames across 600 clips, using discrete values from −10 to +10.
- AFEW-VA: AFEW-VA remains limited by small size, only two annotators, and discrete valence-arousal values that coarsely approximate the continuous emotion space.The paper states that continuous values can better model the expressiveness and richness of everyday emotional states.
- Aff-Wild: Aff-Wild comprises 298 YouTube videos exceeding 30 hours, with spontaneous facial behaviors collected under arbitrary recording conditions.The database includes 200 subjects and continuous valence-arousal annotations from eight annotators, ranging over [−1, +1].
- Aff-Wild: Aff-Wild examples combine varied emotional states, rapid emotional changes, occlusions, and continuous valence-arousal annotation over video frames.
3 Data Pre-processing and Annotation
The Aff-Wild videos underwent face and landmark preprocessing, continuous valence/arousal annotation, verification, and selection of the most representative annotations. The resulting labels showed substantially higher agreement among selected annotations than among all annotators.
- Pre-processing: Aff-Wild preprocessing trimmed videos, extracted frames, detected faces and landmarks, and removed frames with failed detections.Bounding-box and landmark failures were identified through detection checks and annotator verification.
- Annotation tool: Annotation timestamps were resampled with nearest-neighbor interpolation because they generally did not match the video frame rate.
- Annotation tool: Six experts annotated valence and arousal continuously using a joystick-based tool with values ranging from −1 to 1.The tool recorded annotation values with their corresponding time instances for each video.
- Annotation validation: Two new experts selected 2–4 annotations per video, whose mean formed the final Aff-Wild labels after all six original annotations were verified.The selection step was intended to obtain annotations with higher correlation.
- Annotation consistency: 0.71 and 0.70 were the mean MAC-S values for valence and arousal, compared with 0.47 and 0.46 for mean MAC-A.MAC-S measures agreement among selected annotators, whereas MAC-A averages inter-annotator correlations across all annotators.
- Annotation consistency: Figure 7 reports cumulative MAC-S and MAC-A distributions over all videos, with mean MAC-S values of 0.71 for valence and 0.70 for arousal.
4 Developing the AffWildNet
The authors developed CNN and CNN-RNN architectures for the Aff-Wild Challenge, culminating in AffWildNet, which models visual features and temporal dynamics. Challenge results exposed the difficulty of in-the-wild affect recognition, while the proposed architecture achieved the best database performance.
- Challenge evaluation: The Aff-Wild Challenge evaluated valence and arousal predictions using CCC as the main criterion, with MSE as a complementary measure.Higher CCC and lower MSE indicate better agreement or prediction error, respectively.
- Challenge results: Six research groups submitted challenge results, and FATAUVA-Net achieved the best mean CCC and mean MSE for valence and arousal among the reported methods.
- Challenge results: All three participating methods had very low arousal CCC, while valence MSE was high and valence CCC was low except for the winning method.The results illustrate the difficulty of recognizing affect under illumination changes, occlusions, and varied head poses.
- AffWildNet architecture: AffWildNet combines convolutional and recurrent layers to predict continuous affect while modeling temporal information in video sequences.The authors developed CNN and CNN-RNN alternatives and identified AffWildNet as their best-performing architecture.
- Ablation studies: The experiments compared end-to-end CNN and CNN-RNN training, alternative CNN and recurrent backbones, and the use of facial landmarks.The tested feature extractors included ResNet-50, VGG-Face, and VGG-16, while recurrent options included LSTM and GRU.
- Ablation studies: Data augmentation contributed to AffWildNet’s state-of-the-art performance on Aff-Wild.
5 Feature Learning from Aff-Wild
Aff-Wild features were used to fine-tune models across dimensional and categorical emotion datasets with differing label formats. The fine-tuned AffWildNet achieved strong cross-dataset performance, including superiority over comparison architectures on AFEW-VA and EmotiW.
- Cross-dataset feature learning: Aff-Wild pretraining was used as a feature prior for valence/arousal recognition on RECOLA and AFEW-VA, and categorical recognition on EmotiW.The authors motivate this transfer by variability in annotators, label ranges, and discrete versus continuous annotations across databases.
- AFEW-VA: AFEW-VA labels were scaled from [−10, +10] to [−1, +1] to match Aff-Wild, while remaining discrete rather than continuous.
- AFEW-VA: The fine-tuned AffWildNet greatly outperformed a VGG-Face-based CNN on AFEW-VA in CCC-based valence and arousal evaluation.Table 13 evaluates both models on the AFEW-VA training set.
- EmotiW: On EmotiW, AffWildNet was fine-tuned by replacing its output layer with seven neurons for the basic emotion categories.The model used video frames and frame aggregation rather than audio.
- EmotiW: AffWildNet outperformed the compared CNN and CNN-RNN architectures in total validation accuracy on the EmotiW 2017 Grand Challenge.The comparisons used networks based on ResNet-50, VGG-16, and VGG-Face trained on the same AFEW 5.0 dataset.
- EmotiW: Its total accuracy was much higher than the baseline value of 0.3881 and better than all vanilla architectures’ performances.
- Cross-dataset feature learning: The reported results support fine-tuning AffWildNet for both dimensional and categorical emotion recognition.
6 Conclusions and Future Work
The paper establishes Aff-Wild and AffWildNet for in-the-wild affect analysis, then shows that the learned architecture and database transfer to dimensional and categorical emotion recognition.
- Aff-Wild contains 298 videos from 200 subjects totaling more than 30 hours, providing a large in-the-wild resource for affect recognition.
- AffWildNet achieves the best valence and arousal performance on Aff-Wild under both Concordance Correlation Coefficient and Mean Squared Error criteria among networks trained on the same database.
- Fine-tuning and retraining AffWildNet produces best results on other dimensional databases, including RECOLA and AFEW-VA, compared with state-of-the-art pre-trained and fine-tuned networks.
- Fine-tuning AffWildNet also produces state-of-the-art performance for categorical emotion recognition and comparable or better results than EmotiW 2017 challenge winners.
- The same deep architecture is used for dimensional and categorical emotion analysis by training on continuous valence-arousal regression and then addressing seven-category classification.
A.1 Baseline: CNN-M
The CNN-M baseline is specified through its convolutional, pooling, and fully connected layers using TensorFlow parameter notation.
- The baseline architecture contains convolutional, batch normalization, pooling, and fully connected layers whose exact structure is given in Table 15.
- Convolutional-layer parameters are reported as filter and stride tuples containing spatial dimensions, input channels, and output channels or feature maps.
- Table 15 also reports the number of hidden units in the fully connected layers using TensorFlow platform notation.
A.2 ResNet-50
The ResNet-50-based CNN architecture uses residual learning with shortcut connections and reports convolutional layers by filter dimensions and feature-map counts.
- ResNet-50: Residual learning stacks blocks in which each output combines a learned residual function with an identity mapping from the input.
- ResNet-50: The valence-arousal CNN-only architecture is based on ResNet-50 and includes two fully connected layers, labeled V and A for valence and arousal.
- ResNet-50: The identity mapping can use 1×1 convolutions to project the input and match the residual function's dimensions.
- ResNet-50: The ResNet-50 model begins with a 7 × 7 convolutional layer with 64 feature maps followed by 3×3 max pooling.
- ResNet-50: Its subsequent four bottleneck blocks each contain 1×1, 3×3, and 1×1 convolutional layers, with a shortcut after every block.
A.3 VGG-Face/VGG-16
The VGG-Face/VGG-16 CNN architecture uses thirteen convolutional and pooling layers followed by three fully connected layers, with dropout and task-specific linear outputs.
- VGG-Face/VGG-16: The VGG-Face/VGG-16-based CNN contains thirteen convolutional and pooling layers and three fully connected layers.
- VGG-Face/VGG-16: Table 16 reports the convolutional, pooling, and fully connected-layer parameters and output units using TensorFlow notation.
- VGG-Face/VGG-16: The final fully connected layer uses a linear activation to provide the final estimates, while remaining fully connected layers use ReLU activations.
- VGG-Face/VGG-16: Dropout follows the first fully connected layer to avoid over-fitting.