Source-linked AI summary
Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
Dimitrios Kollias, Viktoriia Sharmanska, Stefanos Zafeiriou
TL;DR
Facial expressions, action units, and continuous affect have largely been studied separately despite documented interconnections. The paper introduces FaceBehaviorNet, an end-to-end network jointly trained across heterogeneous tasks and datasets with co-annotation and distribution matching. It reports consistent gains over single-task and state-of-the-art methods, including compound-expression recognition with zero or little task-specific data.
Problem
Facial-behavior research has primarily collected datasets for individual tasks, although expressions, action units, and valence-arousal are interconnected.
Method
FaceBehaviorNet jointly predicts expressions, action units, and valence-arousal using over 5M in-the-wild images and couples tasks through co-annotation and distribution matching.
Results
FaceBehaviorNet consistently outperformed single-task and state-of-the-art methods, including compound-expression recognition in zero-shot and few-shot settings.
Takeaways & Limitations
Jointly learned features generalized to compound-expression recognition beyond the training tasks when no or little training data was available.
Takeaways & Limitations
Continuous affect was coupled implicitly through joint training; explicitly modeling its relationship with expressions and action units was left beyond the paper’s scope.
Abstract
from arXiv · showhide
Automatic facial behavior analysis has a long history of studies in the intersection of computer vision, physiology and psychology. However it is only recently, with the collection of large-scale datasets and powerful machine learning methods such as deep neural networks, that automatic facial behavior analysis started to thrive. Three of its iconic tasks are automatic recognition of basic expressions (e.g. happy, sad, surprised), estimation of continuous emotions (e.g., valence and arousal), and detection of facial action units (activations of e.g. upper/inner eyebrows, nose wrinkles). Up until now these tasks have been mostly studied independently collecting a dataset for the task. We present the first and the largest study of all facial behaviour tasks learned jointly in a single multi-task, multi-domain and multi-label network, which we call FaceBehaviorNet. For this we utilize all publicly available datasets in the community (around 5M images) that study facial behaviour tasks in-the-wild. We demonstrate that training jointly an end-to-end network for all tasks has consistently better performance than training each of the single-task networks. Furthermore, we propose two simple strategies for coupling the tasks during training, co-annotation and distribution matching, and show the advantages of this approach. Finally we show that FaceBehaviorNet has learned features that encapsulate all aspects of facial behaviour, and can be successfully applied to perform tasks (compound emotion recognition) beyond the ones that it has been trained in a zero- and few-shot learning setting.
1 Introduction
The paper addresses interconnected facial-behavior tasks with a holistic framework that jointly models heterogeneous annotations and task relationships. FaceBehaviorNet combines expressions, action units, and continuous affect, using large in-the-wild datasets and coupling strategies.
- Contribution: The framework explores relatedness among heterogeneous classification, detection, and regression tasks while operating across partially annotated datasets.It also encodes explicit task relationships to support transparency and expert input.
- Motivation: Facial behavior analysis has mainly studied expressions, continuous valence-arousal, and action units as separate tasks despite evidence that they are interconnected.Prior work links expressions and action units through FACS, compound-emotion studies, and learned representations.
- Method: The proposed coupling strategies use task relatedness from cognitive and psychological studies or infer it empirically from external annotations.The contributions include co-annotation and distribution matching as algorithmic approaches for coupling tasks.
- Contribution: FaceBehaviorNet jointly predicts 7 basic expressions, 17 action units, and continuous valence-arousal in-the-wild.The end-to-end network uses publicly available databases totaling over 5M images with partial or non-overlapping annotations.
- Results: Joint training outperforms single-task networks and supports compound-expression recognition in zero-shot and few-shot settings.The learned representation generalizes beyond the tasks used for training when no or little compound-expression training data is available.
2 Related work
Related work spans holistic computer-vision frameworks, multi-task learning, and facial-analysis systems that combine selected tasks. The paper distinguishes its goal of jointly learning three heterogeneous facial-behavior tasks from prior approaches focused on narrower task combinations.
- Facial behavior analysis: Earlier facial-analysis methods used emotion labels to supplement action-unit annotations or improve action-unit classifiers, but did not jointly learn all three facial-behavior tasks.The three tasks are expression recognition, action-unit detection, and valence-arousal estimation.
- Multi-task learning: Multi-task learning transfers knowledge through shared representations, but prior face-analysis applications were comparatively limited.The cited literature frames joint learning as a way for related tasks to improve one another.
- Closest approaches: FATAUVA-Net combined face attributes, action units, and quantized valence-arousal, while another framework jointly modeled emotion recognition and valence-arousal on AffectNet.These systems differ from the paper’s broader holistic target and task configuration.
3 The Proposed Approach
FaceBehaviorNet jointly models seven basic emotions, 17 facial action units, and continuous valence-arousal, using shared training across partially annotated datasets. It couples task predictions through co-annotation and distribution matching, with continuous affect only implicitly coupled to the other tasks.
- Multi-task formulation: FaceBehaviorNet jointly learns seven basic emotions, 17 binary action units, and continuous valence-arousal from images with task-specific annotations.The multi-task formulation uses separate objectives for categorical emotion recognition, AU detection, and continuous affect estimation.
- Task relatedness: Cognitive-study relations map each basic emotion to prototypical and observational action units, with observational weights reflecting annotator agreement.For happiness, AU12 and AU25 are prototypical, while AU6 is observational with weight 0.51.
- Coupling strategies: Co-annotation transfers labels between emotions and action units: emotion-labeled images receive corresponding AU labels, while compatible AU sets can receive an emotion label.When multiple emotions are possible, the assigned emotion has the largest requirement of prototypical and observational AUs.
- Coupling strategies: Distribution matching aligns predicted AU probabilities with an emotion-derived AU distribution defined by prototypical and observational relations.The approach uses soft targets so AU predictions become consistent with the network’s predicted basic-emotion distribution.
- Coupling strategies: Soft co-annotation combines both strategies by deriving a soft emotion distribution from AU evidence and matching it to predicted emotion probabilities.AU contributions can be weighted, such as the happiness score using AU12, AU25, and weighted AU6.
- Continuous affect: Continuous valence and arousal are coupled implicitly through joint training, while explicitly modeling their relationship with categorical emotions and AUs remains beyond this work’s scope.One dataset includes annotations for both categorical and continuous emotions.
4 Experimental Study
The experiments combine heterogeneous in-the-wild facial-behavior datasets and evaluate FaceBehaviorNet with task-coupling strategies against single-task and state-of-the-art systems. Joint training improved performance broadly, while combining soft co-annotation with distribution matching produced the strongest results and enabled compound-emotion recognition with limited supervision.
- Databases: The experiments use databases containing valence-arousal, basic-expression, and action-unit annotations across varied image conditions, totaling over 5M images.The selected datasets include differences in resolution, pose, orientation, and lighting; annotations may be partial or non-overlapping.
- Ablation Study: Using both soft co-annotation and distribution-matching losses achieved the best overall results, improving average performance by more than 2% over using either loss alone.Co-annotation alone performed worst among the coupling-loss variants; soft co-annotation helped AUs more, whereas distribution matching helped expressions more.
- Single-task comparison: Jointly trained FaceBehaviorNet outperformed independently trained VGG-FACE single-task models across all databases despite heterogeneous, ambiguous, and sometimes error-prone annotations.The reported ambiguities include disagreement over some emotions, inconsistent valence-arousal values, and difficult AU annotation.
- Results: Adding coupling losses made FaceBehaviorNet superior to all state-of-the-art methods under both cognitive-study and empirically inferred task-relatedness settings.The authors report that performance was boosted independently of which task-relatedness table was used.
- Zero-Shot and Few-Shot Learning: Fine-tuned FaceBehaviorNet outperformed DLP-CNN by 1.2% and 3.7% in the reported few-shot comparisons.The comparison used RAF-DB’s small training set and included FaceBehaviorNet trained with and without coupling losses.
- Zero-Shot and Few-Shot Learning: With coupling losses, FaceBehaviorNet outperformed NTechLab by 5.7% in F1 score and 8.6% in Unweighted Average Recall for compound-emotion recognition.The same coupled approach also outperformed VGG-FACE-mSVM by 4.8% in the reported compound-emotion experiment.
5 Conclusions
FaceBehaviorNet jointly trains expression recognition, action-unit detection, and valence-arousal estimation using over 5M in-the-wild images. It consistently outperformed single-task and state-of-the-art methods and generalized to compound-expression recognition in zero- and few-shot settings.
- FaceBehaviorNet jointly performs basic expression recognition, action unit detection, and valence-arousal estimation in an end-to-end network.The training uses publicly available in-the-wild databases containing over 5M images.
- The model uses co-annotation and distribution matching to couple facial-behavior tasks during training.
- FaceBehaviorNet consistently outperformed single-task networks and state-of-the-art methodologies.
- Jointly learned features generalized to compound-expression recognition under zero-shot and few-shot learning settings.