PaperlayerView exact revision on arXiv

Source-linked AI summary

Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework

Dimitrios Kollias, Stefanos Zafeiriou

arXiv:2103.15792v1cs.CVcs.AIcs.LG

TL;DR

The paper tackles limited large-scale evidence for affect recognition in uncontrolled real-world settings. It develops large in-the-wild databases and deep networks, including a holistic multitask framework, and reports improved state-of-the-art performance across affect tasks and datasets.

  • Problem

    Affect recognition has been limited by the scarcity of large real-life datasets, while facial affect must be analyzed across dimensional, categorical, and action-unit representations.

  • Method

    The paper creates and uses Aff-Wild and Aff-Wild2 to train uni-task, multitask, audiovisual, and holistic deep neural architectures for affect recognition.

  • Results

    The proposed systems improve state-of-the-art affect recognition, with FaceBehaviorNet exceeding state-of-the-art by 3.5% average CCC on Aff-Wild and 4% on AffectNet without coupling loss.

  • Takeaways & Limitations

    Jointly modeling valence-arousal, expressions, and action units supports affect recognition across existing in-the-wild datasets.

Abstract

from arXiv · show

Affect recognition based on subjects' facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This paper exploits these advances and presents significant contributions for affect analysis and recognition in-the-wild. Affect analysis and recognition can be seen as a dual knowledge generation problem, involving: i) creation of new, large and rich in-the-wild databases and ii) design and training of novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. The paper focuses on large in-the-wild databases, i.e., Aff-Wild and Aff-Wild2 and presents the design of two classes of deep neural networks trained with these databases. The first class refers to uni-task affect recognition, focusing on prediction of the valence and arousal dimensional variables. The second class refers to estimation of all main behavior tasks, i.e. valence-arousal prediction; categorical emotion classification in seven basic facial expressions; facial Action Unit detection. A novel multi-task and holistic framework is presented which is able to jointly learn and effectively generalize and perform affect recognition over all existing in-the-wild databases. Large experimental studies illustrate the achieved performance improvement over the existing state-of-the-art in affect recognition.

I. INTRODUCTION

The paper addresses affect recognition in-the-wild by combining large databases with deep neural architectures for dimensional, categorical, and action-unit analysis. It targets robust interpretation of complex, time-varying facial affect in uncontrolled human-computer interaction settings.

  • Facial affect is difficult because emotion patterns are complex, time varying, and dependent on users and context.
  • The paper considers categorical expressions, facial Action Units, and dimensional valence-arousal representations.
  • In-the-wild affect recognition requires systems robust to varied recording conditions, contexts, and timing.
  • Existing in-the-wild datasets have limitations including small size, limited annotation, few subjects, and restricted modalities.
  • The authors present new DNN systems for dimensional, categorical, and AU-based affect recognition.
  • A holistic multitask framework jointly learns the three emotion representations using publicly available datasets containing over 5 million images.

B. Action Units

Action Units provide a physiological basis for analyzing facial behavior, while in-the-wild databases address the diversity and spontaneity absent from controlled settings. Aff-Wild2 extends this effort with annotations spanning continuous affect, expressions, and action units.

  • Action Units: Facial Action Units encode facial muscle movements and can represent expressions as combinations of constituent actions.
  • Action Units: Action-unit detection is challenging because these actions are brief, unconscious, and expressed through subtle facial appearance changes.
  • In-the-wild datasets: Naturalistic facial expressions can differ radically from posed behaviors, motivating databases collected in unconstrained settings.
  • In-the-wild datasets: Existing in-the-wild datasets separately study expressions, action units, or continuous valence-arousal, with most offering annotations for only one task.
  • Aff-Wild: Aff-Wild comprised 298 videos from 200 subjects, exceeding 30 hours and 1,224,100 frames, with framewise valence-arousal annotations.
  • Aff-Wild2: Aff-Wild2 was introduced as the first and only database annotated for valence-arousal, action units, and expressions.
  • Aff-Wild2: Aff-Wild2 contains 558 videos from 458 subjects and around 2,800,000 frames showing subtle and extreme behaviors in real-world settings.

IV. INTELLIGENT SIGNAL PRE-PROCESSING & PERFORMANCE EVALUATION METRICS

The paper preprocesses visual and, in some experiments, audio signals before evaluating affect models with task-specific metrics. Valence-arousal estimation uses CCC, which penalizes correlated predictions that are shifted in value.

  • Pre-processing: Visual preprocessing includes face detection, alignment, resizing, and normalization, with no data augmentation performed.
  • Pre-processing: Face bounding boxes are extracted with an SSH detector, while facial landmarks are used for alignment.
  • Pre-processing: Audio preprocessing samples mono signals at 44,100Hz and extracts normalized spectrograms using 33ms windows with 11ms overlap.
  • Evaluation metrics: CCC measures agreement between annotations and predictions by scaling correlation with mean squared difference.
  • Evaluation metrics: CCC ranges from −1 to 1, with +1 indicating perfect concordance and higher values indicating better fit.
  • Evaluation metrics: The CCC equation incorporates Pearson correlation, variances, means, and covariance of annotations and predictions.
  • Evaluation metrics: Expression classification uses F1 Score or mean confusion-matrix diagonal, while AU detection uses F1 Score or mean F1 and Accuracy.

V. UNI-TASK AFFECT ANALYSIS: DIMENSIONAL AFFECT RECOGNITION

The uni-task dimensional-affect analysis develops AffWildNet, a CNN-RNN model that predicts valence and arousal from in-the-wild facial data. It combines facial landmarks with CNN features, models temporal context, and outperforms competing architectures while generalizing to other datasets.

  • AffWildNet architecture: AffWildNet is a CNN-RNN architecture that predicts valence and arousal from facial sequences.Its CNN features and facial landmarks are mapped through an FC layer before a two-layer GRU models temporal variations.
  • Design choices: Facial landmarks are fused with CNN features, while the CCC-based loss aligns training with the principal evaluation criterion.The loss uses CCC for arousal and valence, denoted ρa and ρv.
  • Training: AffWildNet is trained end-to-end by jointly optimizing its CNN and RNN components.This contrasts with approaches that separately train the components or freeze CNN weights.
  • Evaluation: AffWildNet improves CCC and MSE over the developed comparison architectures on the Aff-Wild test set and outperforms FATAUVA-Net, the Aff-Wild Challenge winner.The reported improvement is attributed to better capture of Aff-Wild’s temporal dynamics.
  • Generalization: AffWildNet successfully generalizes to other emotion-recognition datasets and contexts.The paper characterizes its learned features as a robust prior for dimensional and categorical emotion recognition.

B. Multi-Component Architectures: CNN plus Multi-RNN networks

The multi-component approach extracts features at multiple CNN levels, processes them with separate RNNs, and fuses the resulting representations. These architectures improve over ResNet-RNN, while model-level fusion with an RNN outperforms decision-level fusion and challenge-winning methods on valence and arousal.

  • CNN plus Multi-RNN networks: CNN plus Multi-RNN networks process low-, mid-, and high-level CNN features through multiple RNNs.The design treats lower-layer features as rich and time varying, while higher-layer features are more task-specific.
  • Architectures: CNN-3RNN-2nd-pool_last-pool_fc independently processes FC, last-pooling, and second-pooling features before combining them.CNN-1RNN instead concatenates features from these layers before a single RNN processes them.
  • Fusion: Model-level fusion concatenates network features and feeds them to an RNN or fully connected layer, whereas decision-level fusion averages predictions using validation-set weights.The model-level fusion network predicts valence-arousal values per frame and computes median final estimates.
  • Results: Around 6% and 9% average performance improvements are reported for CNN-1RNN-2nd-pool_last-pool_fc and CNN-3RNN-2nd-pool_last-pool_fc, respectively, versus ResNet-RNN.CNN-3RNN performs better by exploiting low- and high-level feature time variations independently.
  • Fusion results: Model-level fusion with an RNN outperforms decision-level fusion and the OMG-Emotion Challenge winning methodologies on both valence and arousal estimation.The ensemble is trained end-to-end so concatenated features can be optimized jointly.
  • Modality effects: Visual-only models perform worse for arousal than valence estimation because audio cues appear more discriminating for arousal.This comparison concerns correlation coefficient performance.

A. A Multi-Task Approach to Affect Recognition

The multi-task approach jointly learns valence-arousal estimation, action unit detection, and basic expression classification from shared representations, including audiovisual inputs. Across databases, the developed multi-task networks generally outperform competing methods, with modality-specific differences for valence and arousal.

  • Multi-task learning jointly addresses valence-arousal estimation, action unit detection, and basic expression classification using shared representations.The approach exploits correlations among the three behavior tasks.
  • A/V-MT-VGG-GRU combines raw video-image and audio-spectrogram streams, fuses their features with a two-layer GRU, and predicts all three tasks.The fused feature vector is 256-dimensional, and each GRU layer contains 128 units.
  • Cross-database evaluation trained networks on Aff-Wild2 and fine-tuned them on nine databases, identifying the multi-task networks as the strongest pre-trained framework across settings.The comparison covered databases with differing modalities and task annotations.
  • MT-VGG-GRU outperformed the compared methods across all tasks in the Aff-Wild2 evaluation, for both audio-only and visual-only training settings.The comparison included MT-VGG, A/V-MT-VGG-GRU, and state-of-the-art methods from the ABAW Competition.
  • The A/V-MT-VGG-GRU achieved the best Aff-Wild valence and arousal performance, surpassing AffWildNet.Visual training performed best for valence, whereas audio training performed best for arousal in the reported comparison.
  • MT-VGG and MT-VGG-GRU surpassed several state-of-the-art baselines, although MT-VGG underperformed the state-of-the-art method for expression recognition on AffectNet.Reported comparisons included AlexNet, VGG-FACE-mSVM, and ResNet-34.

B. A Holistic Approach to Affect Recognition in-the-wild

The holistic framework jointly models expressions, action units, and valence-arousal while encoding task relatedness through expert knowledge or empirical evidence. FaceBehaviorNet is trained end-to-end across partially annotated in-the-wild databases and substantially improves performance, including compound-expression generalization with little or no direct training data.

  • Task relatedness is incorporated through expert-derived emotion–AU relationships or empirical relationships inferred from external dataset annotations.The two strategies are explicitly evaluated in the framework.
  • Co-annotation and distribution matching couple heterogeneous task predictions during training, including relationships between emotions and action units.Co-annotation reuses emotion or AU information to supervise the related task, with observational AUs re-weighted by annotator agreement.
  • FaceBehaviorNet simultaneously predicts 7 basic expressions, 17 action units, and continuous valence-arousal using more than 5M images with partial or non-overlapping annotations.The network is trained end-to-end on all publicly available in-the-wild databases described by the authors.
  • FaceBehaviorNet greatly outperforms single-task networks and generalizes to compound-expression recognition in zero-shot and few-shot settings.The reported generalization uses feature representations learned during joint training when little or no compound-expression data is available.
  • The framework jointly treats seven basic expressions, 17 binary action units, and continuous valence-arousal as interconnected affective tasks.Its formulation covers classification, detection, and regression within one facial behavior model.
  • The holistic objective combines emotion, action-unit, and valence-arousal losses as LMT = LEmo + λ1LAU + λ2LVA.The total objective is designed for joint optimization of the three tasks.

1) Coupling of basic emotions and AUs via co-annotation:

The co-annotation strategy aligns emotion and action-unit predictions during training by using each sample’s available labels to supervise related task outputs.

  • Co-annotation aligns emotion and action-unit predictions by jointly using softmax emotion scores and AU activation predictions for each sample.The strategy is intended to couple the two tasks during training.

2) Coupling of basic emotions and AUs via distribution matching:

Distribution matching couples predicted basic emotions and facial Action Units by requiring their AU distributions to remain consistent. The approach uses emotion-linked AU structure as soft targets during training.

  • 2) Coupling of basic emotions and AUs via distribution matching:: The method matches predicted AU probabilities p(y_i^au|x) with a distribution q(y_i^au|x) derived from predicted basic emotions.Each AU is modeled through its association with basic emotion categories.
  • 2) Coupling of basic emotions and AUs via distribution matching:: AU-emotion associations are deterministic: an AU receives value 1 when prototypical or observational for an emotion and 0 otherwise.For example, AU2 is associated with surprise and fear.
  • 2) Coupling of basic emotions and AUs via distribution matching:: If happiness is predicted with probability 1, the corresponding distribution activates AUs 12, 25 and 6 while setting other listed AUs to zero.This illustrates how emotion predictions impose structured AU targets.
  • 2) Coupling of basic emotions and AUs via distribution matching:: The distributions are aligned by minimizing cross-entropy with soft targets, using all available training samples.The authors describe this as a distillation-style matching procedure.

3) A mix of the two strategies, co-annotation and distribution matching:

The co-annotation strategy converts AU annotations into soft emotion labels, while distribution matching compares those emotion-derived AU distributions with the network’s emotion predictions.

  • 3) A mix of the two strategies, co-annotation and distribution matching:: For each AU-annotated image, the method computes soft scores for each basic emotion from its prototypical and observational AUs.For happiness, the score combines AU12, AU25 and AU6, optionally using unequal weights such as 0.51 for AU6.

4) Coupling of categorical emotions and AUs with continuous affect:

FaceBehaviorNet jointly couples valence-arousal, seven basic expressions and 17 Action Units across heterogeneous databases. Coupling losses improve performance across tasks and support transfer to compound-expression datasets.

  • 4) Coupling of categorical emotions and AUs with continuous affect:: FaceBehaviorNet jointly predicts valence and arousal, seven basic expressions and 17 Action Units.Continuous affect is implicitly coupled with categorical expressions and AUs through joint training.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: Training concatenates batches from VA-Set, AU-Set and EXPR-Set so every iteration includes images annotated for all three task types.This ensures all loss terms contribute during optimization despite partial annotations.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: The datasets are aligned by sampling batch sizes of 401 for VA-Set, 247 for AU-Set and 103 for EXPR-Set per epoch.Training used an end-to-end learning rate of 10^-4 and 0.5 dropout in fully connected layers.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: FaceBehaviorNet outperformed independently trained VGG-FACE single-task models on all databases, indicating coherent correlations among the facial behavior tasks.The authors link the improvement to simultaneous end-to-end training with heterogeneous databases.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: With coupling losses, FaceBehaviorNet outperformed state-of-the-art methods across the evaluated databases and remained superior under two task-relatedness settings.Without coupling losses, performance exceeded some baselines but remained inferior on several datasets.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: Fine-tuned FaceBehaviorNet exceeded DLP-CNN by 1% and 4% on RAF-DB, depending on whether coupling losses were used.The comparison concerns few-shot compound-expression learning.
  • 4) Coupling of categorical emotions and AUs with continuous affect:: On EmotioNet zero-shot compound-expression classification, the coupling approach exceeded NTechLab by 6% in F1 score and 10% in UAR.The comparison used the basic and compound evaluation set.

VII. CONCLUSIONS

The paper develops in-the-wild affect-recognition databases and deep neural architectures spanning dimensional, categorical and AU-based emotion representations. Its holistic FaceBehaviorNet jointly estimates all three facial behavior dimensions and achieves state-of-the-art performance.

  • VII. CONCLUSIONS: The work addresses affect recognition in-the-wild using databases annotated for valence-arousal, seven basic expressions and facial Action Units.It combines database development with deep neural architectures trained on these annotations.
  • VII. CONCLUSIONS: AffWildNet models in-the-wild affect dynamics, while multi-task DNNs trained on Aff-Wild2 outperform state-of-the-art affect-recognition methods.Aff-Wild2 is described as the largest in-the-wild audiovisual database in the paper.
  • VII. CONCLUSIONS: Future work includes scalable coarse-to-fine architectures, database extension or domain adaptation for specific affects, unsupervised learning, uncertainty analysis and improved model transparency.The paper notes that DNN decisions are not straightforward to justify.
Loading 2103.15792v1…