Source-linked AI summary
Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
Dimitrios Kollias, Viktoriia Sharmanska, Stefanos Zafeiriou
TL;DR
Heterogeneous multi-task learning lacks a general way to jointly train detection, classification, and regression tasks when annotations barely overlap. The paper uses task relatedness, weakly supervised co-training, and prediction-distribution matching in FaceBehaviorNet, which achieves broad improvements across facial behavior tasks and databases while supporting compound-expression zero-/few-shot learning.
Problem
Heterogeneous tasks often have little or non-overlapping annotations, while multi-task learning typically assumes related homogeneous tasks with overlapping ground truth.
Method
FaceBehaviorNet uses task relatedness from expert knowledge or dataset studies to couple heterogeneous tasks through weakly supervised label co-annotation and prediction-distribution matching.
Results
FaceBehaviorNet greatly outperforms single-task and state-of-the-art methods across all tasks and 10 databases, including databases not used in training.
Takeaways & Limitations
Joint training learns facial behavior features that generalize to compound-expression recognition in zero-shot and few-shot settings.
Takeaways & Limitations
The paper leaves explicit relationships between categorical emotions, action units, and continuous affect as a research direction beyond its scope.
Abstract
from arXiv · showhide
Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a DNN. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We explore task-relatedness as a means for co-training, in a weakly-supervised way, tasks that contain little, or even non-overlapping annotations. Task-relatedness is introduced in MTL, either explicitly through prior expert knowledge, or through data-driven studies. We propose a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions. Based on this approach, we build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks. We develop case studies for: i) continuous affect estimation, action unit detection, basic emotion recognition; ii) attribute detection, face identification. We illustrate that co-training via task relatedness alleviates negative transfer. Since FaceBehaviorNet learns features that encapsulate all aspects of facial behavior, we conduct zero-/few-shot learning to perform tasks beyond the ones that it has been trained for, such as compound emotion recognition. By conducting a very large experimental study, utilizing 10 databases, we illustrate that our approach outperforms, by large margins, the state-of-the-art in all tasks and in all databases, even in these which have not been used in its training.
1 INTRODUCTION
The paper addresses heterogeneous facial behavior tasks with weakly supervised co-training based on task relatedness, enabling knowledge exchange despite limited or non-overlapping annotations. FaceBehaviorNet jointly learns facial behavior and outperforms single-task and state-of-the-art methods across extensive evaluations.
- Motivation: Prior holistic models commonly combine datasets or rely on annotations covering multiple tasks, but facial behavior datasets lack complete annotations for all three heterogeneous tasks.Action-unit annotation is particularly costly because it requires skilled annotators.
- Task relatedness: Task relatedness is inferred from domain knowledge or dataset annotations and used to couple heterogeneous tasks during co-training.The paper considers relationships among expressions, action units, and affect dimensions.
- Proposed contribution: Distribution matching and label co-annotation provide weakly supervised coupling for tasks with little or even non-overlapping annotations.The approach is designed for heterogeneous tasks spanning detection, classification, and regression.
- Framework: FaceBehaviorNet simultaneously predicts 7 basic expressions, 17 action units, and continuous valence-arousal dimensions in-the-wild.The network also automatically annotates available databases for all tasks, with annotations intended for public release.
- Results: FaceBehaviorNet greatly outperforms single-task networks in all tasks and databases, including databases not used during training, and outperforms state-of-the-art methods across 10 databases.The study reports that coupling losses prevent negative transfer and enhance jointly trained capabilities.
2 RELATED WORK
Related work includes multi-task and weakly supervised approaches for face analysis, but prior systems generally address narrower task combinations or rely on overlapping annotations. The paper targets a single holistic framework for jointly learning heterogeneous facial behavior tasks.
- Facial behavior learning: Prior work used emotion labels to complement action-unit annotations, but generally focused on action-unit detection rather than jointly learning all three facial behavior tasks.The three tasks are action-unit detection, emotion recognition, and valence-arousal estimation.
- Multi-task learning: Earlier face-analysis MTL systems jointly handled tasks such as face detection, landmarks, gender, age, attributes, or identification, often using generated labels across datasets.These examples establish related precedents for multi-task and label-transfer learning.
- Weak supervision: Knowledge-augmented models combined expression classification, action-unit detection, and knowledge-based weak supervision through separate neural and Bayesian components.An AU-based expression-recognition model used action-unit detection results.
- Facial behavior learning: A unified competition model addressed action units, expressions, and valence-arousal with incomplete labels by first training a teacher multi-task model.The described system used Aff-Wild2, which contains annotations for all three tasks but not for every image-task pair.
- Alternative architectures: Other approaches used multimodal networks or attention-based architectures for subsets of the tasks, with training data containing overlapping annotations.One model combined image and audio modalities, while another used AffectNet annotations for emotion and valence-arousal.
3 THE PROPOSED APPROACH
The proposed approach formulates heterogeneous MTL over task-specific data distributions and combines standard task losses with coupling losses. Task relationships are incorporated through co-annotation and distribution matching within a shared deep network.
- MTL formulation: For m tasks, the framework learns task-specific hypotheses while minimizing a weighted average of expected losses over their task distributions.Each task has observations drawn from its own input-label distribution, and task weights govern contributions to the overall loss.
- Task types: Task labels support binary classification, mutually exclusive k-class classification, and regression settings.Binary and multiclass labels use corresponding indicator or one-hot representations.
- Task losses: The framework uses MAE, MSE, correlation-based, binary cross-entropy, or softmax cross-entropy losses according to the task type.In neural networks, the hypotheses are represented by functions of learned network weights.
- Task coupling: Task relationships are inferred from domain knowledge or dataset annotations and used to couple tasks through co-annotation and distribution-matching losses.The coupling losses can be incorporated into any deep neural network performing MTL.
3.1 Case Study I: Affective Computing
The affective-computing case jointly learns seven basic emotions, 17 action units, and continuous valence-arousal, coupling tasks through relatedness-informed co-annotation and distribution matching.
- The model jointly learns seven basic emotions, 17 binary action units, and continuous valence-arousal dimensions.
- Task-Relatedness: Task relatedness is obtained from cognitive studies linking emotions with prototypical and observational action units, or from Aff-Wild2 annotations.
- Co-annotation: Co-annotation transfers emotion labels to action-unit targets and action-unit labels to emotion targets when the relatedness rules support those labels.
- Distribution Matching: Distribution matching represents action-unit predictions as a mixture over emotion categories and aligns them with the network’s action-unit predictions.
- Distribution Matching: The matching loss uses soft targets, drawing on prototypical and observational action-unit relationships during training.
Via Mixing the two strategies, co-annotation and distribution
The paper also combines co-annotation and distribution matching, then extends FaceBehaviorNet to compound-emotion recognition through zero-shot scoring and few-shot transfer.
- Mixing the two strategies: A hybrid strategy can convert action-unit annotations into soft emotion labels and match them against predicted emotion distributions.
- Mixing the two strategies: The soft emotion label is computed from indicators of prototypical and observational action units, optionally weighted by annotator agreement.
- Coupling categorical emotions, AUs with continuous affect: Continuous affect is coupled implicitly through joint training, while an explicit relationship with categorical emotions and action units remains beyond the work’s scope.
- Compound Expressions: Zero- and Few-shot Learning: For compound emotions, candidate scores combine associated action-unit predictions, the two component basic-emotion predictions, and valence for selected classes.
- Compound Expressions: Zero- and Few-shot Learning: The predicted compound class is the class with the maximum candidate score.
- Robust Prior for Few-Shot Learning: FaceBehaviorNet serves as a pretrained prior for fine-tuning compound-emotion recognition on in-the-wild datasets containing fewer than 5K images.
3.2 Case Study II: Face Recognition
The face-recognition case jointly detects 40 facial attributes and classifies 10,177 identities, using CelebA annotations to infer their task relatedness for distribution matching.
- The model jointly detects 40 binary facial attributes and classifies 10,177 identities.
- Task-Relatedness: CelebA provides overlapping identity and attribute annotations from which attribute distributions are calculated for each identity.
- Task-Relatedness: Table 2 illustrates identity-attribute relationships such as gender, age, lipstick, necklace, and five-o’clock-shadow patterns.
- Coupling via Distribution Matching: Distribution matching uses identity softmax predictions and attribute sigmoid predictions, modeling attributes as a mixture over identities.
- Coupling via Distribution Matching: The method aligns the two prediction distributions with a cross-entropy soft-target loss using all available training samples.
3.3 FaceBehaviorNet structure
Figure 1 presents FaceBehaviorNet as a residual-unit holistic model for multi-task, multi-domain, and multi-label face analysis.
- FaceBehaviorNet uses residual units and organizes outputs for multiple facial behavior tasks.In affective computing, outputs include valence-arousal estimates and seven basic-expression logits.
4 EXPERIMENTAL STUDY
The experimental study evaluates FaceBehaviorNet across affective-computing and face-recognition databases, using diverse annotations and task-specific evaluation criteria. The datasets span large-scale in-the-wild, laboratory-controlled, and attribute-labeled face data.
- Affective-computing databases: The affective-computing study uses databases annotated for valence-arousal, basic expressions, and action units across varied image conditions.The images vary in resolution, pose, orientation, lighting, gender, ethnicity, race, and age.
- Affective-computing databases: Aff-Wild contains around 1.25M frames from 298 videos and 200 subjects, with valence-arousal annotations ranging in [−1, 1].Aff-Wild2 extends this setting to 545 videos and around 2.8M frames.
- Affective-computing databases: AffectNet contains around 1M facial images, including 400K manually annotated for seven discrete expressions plus contempt and valence-arousal.Its training set contains around 321K images and its validation set 5K balanced across emotion categories.
- Affective-computing databases: DISFA provides 27 videos with 4,845 frames each and codes action-unit intensity on a six-point discrete scale.Action-unit intensities equal to or greater than 2 are treated as occurrences.
- Affective-computing databases: BP4D contains 61 subjects and 223K frames annotated for occurrence and intensity of 27 action units, with subject-partitioned training, development, and test sets.The database was also used in FERA 2015 with 11 action units.
- Evaluation: Evaluation uses CCC, confusion-matrix mean diagonal, accuracy, or F1 score depending on the database and task.These metrics follow the relevant challenge or database evaluation criteria.
4.3 Pre- and Post-Processing
Pre- and post-processing clean inconsistent affect annotations, reduce redundant video frames, standardize face inputs, and smooth per-frame predictions.
- Data cleaning: AffectNet images with inconsistent valence-arousal and discrete-expression annotations are removed using rule-based consistency checks.The rules include a radius threshold for neutral images and valence-sign constraints for several negative expressions.
- Data subsampling: For Aff-Wild videos, every retained frame is followed by four skipped frames because the CNN does not exploit temporal dependencies.This subsamples consecutive frames with similar valence-arousal values.
- Face pre-processing: Faces are detected with SSH, aligned using five landmarks, resized to 112 × 112 × 3 pixels, and normalized to [−1, 1].CelebA uses the database’s distributed aligned data.
- Prediction post-processing: Median filtering is applied to Aff-Wild’s per-frame predictions because FaceBehaviorNet does not model temporal dependencies.The filtering operates after prediction rather than within the CNN.
4.4 Training Implementation Details
Training combines batches from valence-arousal, action-unit, and expression datasets at each iteration, aligning their different sizes for end-to-end optimization.
- Multi-dataset batching: Each training iteration concatenates one batch from VA-Set, AU-Set, and EXPR-Set before feeding the combined input to FaceBehaviorNet.The three sets contain images annotated for valence-arousal, action units, or seven basic expressions, respectively.
- Multi-dataset batching: Batch sizes of 200 for AU-Set, 124 for VA-Set, and 52 for EXPR-Set align sampling so every set is covered after one epoch.The different batch sizes compensate for the sets’ different sizes.
- Optimization: FaceBehaviorNet is trained end-to-end with Momentum using 0.9 momentum and a learning rate of 10^-4.The optimizer settings are reported as part of the training implementation details.
4.5 Case Study I: Affective Computing
FaceBehaviorNet’s coupling losses improved joint prediction of valence-arousal, basic expressions, and action units across affective-computing databases. The combined soft co-annotation and distribution-matching losses also reduced negative transfer and supported strong comparisons with single-task and state-of-the-art methods.
- Ablation Study: Coupling losses improved FaceBehaviorNet over training without coupling losses across all databases and both task-relatedness scenarios.The improvement held for individual losses and combinations, supporting coupling of the three heterogeneous tasks.
- Ablation Study: More than 2.5% average performance improvement resulted from combining soft co-annotation and distribution-matching losses versus using either alone.The combined losses achieved the best results in both task-relatedness settings across all databases.
- Comparison with State-of-the-Art and Single-Task Methods: FaceBehaviorNet without coupling losses outperformed state-of-the-art methods on BP4D+ by 5%, BP4D by 25% for AU detection, and AffectNet by 2% for arousal estimation.In other databases, performance was slightly worse or worse than state-of-the-art methods.
- Comparison with State-of-the-Art and Single-Task Methods: The uncoupled multi-task network outperformed single-task models for AU detection and valence-arousal estimation but underperformed for basic expression classification.The expression-classification deficit indicates negative transfer in that task.
- Comparison with State-of-the-Art and Single-Task Methods: The coupling losses addressed negative transfer without relying on performance-indicator trade-offs, while task-relatedness information was incorporated into training.The authors note that VA and AU data dominated expression recognition because they contained more labeled data.
- Comparison with State-of-the-Art and Single-Task Methods: 10.3% average improvement over single-task networks was achieved by FaceBehaviorNet with both coupling losses, with database differences ranging from 5% to 22%.The coupled model also exceeded the uncoupled multi-task model by up to the database-specific margins reported across the evaluation set.
4.6 Case Study II: Face Recognition
FaceBehaviorNet jointly learns facial attribute detection and identity classification, outperforming separate single-task networks and benefiting substantially from distribution-matching coupling.
- Face Recognition: FaceBehaviorNet without coupling outperformed two single-task networks on identity classification and attribute detection across Accuracy and F1 Score.The comparison used CelebA and evaluated Total Accuracy and average F1 Score.
- Distribution Matching: 4.57% and 5.9% gains in identity classification followed distribution-matching coupling for Accuracy and F1 Score, respectively.For attribute detection, the corresponding gains were 1.24% and 2.3%.
- Distribution Matching: 1.24% and 2.3% gains in attribute detection followed distribution-matching coupling for Accuracy and F1 Score, respectively.The coupling-loss model was compared with its counterpart trained without that loss.
- Architecture Ablation: FaceBehaviorNet was the best architecture in an ablation against VGG-FACE, ResNet-50, and DenseNet-121 on all tasks and metrics.The comparison trained the networks multitask without coupling loss on CelebA.
5 CONCLUSIONS
The paper proposes weakly supervised heterogeneous multi-task co-training through task relatedness and distribution matching. FaceBehaviorNet applies this framework to affective computing and face recognition, generalizes to compound expressions with zero- or few-shot learning, and outperforms comparison methods across databases and tasks.
- Framework: The framework co-trains heterogeneous detection, classification, and regression tasks by exchanging predictions through distribution matching based on distillation.Task relatedness comes from expert knowledge or empirical studies.
- Case Studies: FaceBehaviorNet jointly addresses basic expression recognition, action unit detection, valence-arousal estimation, facial attribute detection, and face identification.The case studies use publicly available in-the-wild facial behavior databases.
- Results: FaceBehaviorNet consistently outperformed comparison methods by large margins across databases and both case studies, while mitigating bias and negative transfer.The conclusion reports this pattern across the affective-computing and face-recognition studies.
- Generalization: The learned feature representation generalized to compound-expression recognition under zero-shot and few-shot learning settings.This evaluated tasks beyond those used for joint training.