Source-linked AI summary
Investigating Bias and Fairness in Facial Expression Recognition
Tian Xu, Jennifer White, Sinan Kalkan, Hatice Gunes
TL;DR
Facial-expression datasets often lack fair demographic coverage, motivating a systematic study of bias and fairness. The paper compares baseline, attribute-aware, and disentangled models on RAF-DB and CelebA, with and without augmentation. The disentangled approach performs best for demographic-bias mitigation, while augmentation alone improves baseline accuracy without removing bias.
Problem
Facial-expression datasets are rarely collected with even demographic coverage, leaving bias and fairness insufficiently investigated beyond binary smiling classification.
Method
The study compares baseline, attribute-aware, and disentangled approaches with and without data augmentation on RAF-DB and CelebA.
Results
The disentangled approach is best for mitigating demographic bias; attribute-aware and disentangled models with augmentation outperform the baseline in accuracy and fairness, while augmentation alone does not mitigate baseline bias.
Takeaways & Limitations
Bias mitigation is most suitable when demographic attributes or subgroup data are unevenly distributed, whereas CelebA shows little room for improvement because its task and distribution yield high accuracy and fairness.
Takeaways & Limitations
The findings are data- and model-driven and require evaluation on additional facial-expression datasets and machine-learning models.
Abstract
from arXiv · showhide
Recognition of expressions of emotions and affect from facial images is a well-studied research problem in the fields of affective computing and computer vision with a large number of datasets available containing facial images and corresponding expression labels. However, virtually none of these datasets have been acquired with consideration of fair distribution across the human population. Therefore, in this work, we undertake a systematic investigation of bias and fairness in facial expression recognition by comparing three different approaches, namely a baseline, an attribute-aware and a disentangled approach, on two well-known datasets, RAF-DB and CelebA. Our results indicate that: (i) data augmentation improves the accuracy of the baseline model, but this alone is unable to mitigate the bias effect; (ii) both the attribute-aware and the disentangled approaches fortified with data augmentation perform better than the baseline approach in terms of accuracy and fairness; (iii) the disentangled approach is the best for mitigating demographic bias; and (iv) the bias mitigation strategies are more suitable in the existence of uneven attribute distribution or imbalanced number of subgroup data.
1 Introduction
Facial expression recognition relies on datasets that often underrepresent demographic groups, creating bias in model performance. This paper addresses the gap by systematically comparing three bias-aware approaches beyond prior work focused mainly on smiling classification.
- Dataset Bias: Uneven demographic representation gives models more training opportunities for majority groups and can lower performance for minority groups.Existing datasets may also lack demographic labels, making bias difficult to assess and mitigate.
- Research Gap: Bias in facial analysis has been studied across recognition, gender, age, kinship, and image-quality tasks, but rarely for facial expression recognition.Prior expression-recognition studies cited here focused only on smiling versus non-smiling using CelebA.
- Study Design: The study compares baseline, attribute-aware, and representation-disentangling networks with and without data augmentation.Experiments use RAF-DB and CelebA, which provide expression labels alongside gender, age, and/or race labels.
- Contribution: The work extends fairness analysis to multi-class facial expressions and evaluates sensitive attributes as model inputs or targets for fairer representations.These contributions go beyond binary smiling classification.
2 Related Work
Prior research establishes that facial-analysis systems can inherit demographic bias from data, while expression-specific bias studies remain limited. Existing mitigation strategies span data balancing, augmentation, sensitive-information handling, and model-level approaches.
- Bias in Machine Learning: Machine-learning applications are widely recognized as vulnerable to biases in training data, motivating concern about whether data reflect relevant demographic dimensions.This concern has also prompted preventive regulatory attention.
- Data-Level Mitigation: Dataset balancing can use under-sampling, over-sampling, weighting, generation, augmentation, or balanced datasets, but balanced samples do not guarantee demographic fairness.The cited literature distinguishes label imbalance from demographic bias.
- Feature-Level Strategies: Removing sensitive attributes may leave implicitly correlated information that continues to hinder fairness in predictions.This motivates approaches that make models explicitly aware of sensitive attributes.
- Facial Affect Recognition: Bias research in automatic affect recognition remains sparse, including evidence that ethnicity can impede human expression judgments and that minority-group recognition may be skewed.Related work includes controlled facial-attribute manipulation and targeted augmentation for bias analysis.
- Compared Approaches: The paper’s three approaches are illustrated as baseline, attribute-aware, and disentangled designs.The figure provides a visual overview of the compared mitigation strategies.
3 Methodology
The methodology compares a standard expression classifier with attribute-aware and disentangled alternatives. The latter use sensitive-attribute information either explicitly at classification or adversarially during representation learning to reduce demographic information in shared features.
- Compared Models: The study compares a baseline, an Attribute-aware Approach, and a Disentangled Approach to investigate facial-expression bias mitigation.The approaches are presented as alternative model designs for the same recognition task.
- Baseline Approach: The baseline uses ResNet-18 with cross-entropy loss to predict one expression label from each facial image.The predicted class probabilities and indicator function define the expression-classification objective.
- Attribute-aware Approach: The attribute-aware model supplies sensitive-attribute representations as an additional input to the classification layer.An attribute vector is projected to the image-feature size and added to the ResNet-18 feature vector.
- Disentangled Approach: The disentangled model jointly learns expression features while preventing shared representations from predicting sensitive attributes.It uses a primary expression branch and parallel attribute branches with confusion and attribute-predictive objectives.
- Disentangled Approach: The confusion objective encourages equal probability across sensitive-attribute classes, while an adversarial predictive loss prevents a trivial solution.The two objectives operate in opposition to remove sensitive information without simply collapsing the representation.
- Optimization: The overall loss combines expression, sensitive-attribute, and confusion losses, with α controlling the confusion-loss contribution.The shared representation is intended to retain expression information and discard sensitive-attribute information.
4 Implementation Details
The experiments evaluate accuracy and fairness on RAF-DB and CelebA after standardized face preprocessing, optional augmentation, and common training settings. Fairness is defined through equal opportunity across demographic groups, including the largest accuracy gap for non-binary attributes.
- Datasets: RAF-DB contributes 14,388 basic-emotion images, split into 11,512 training and 2,876 testing samples.The task is recognition of seven facial-expression categories using race, age, and gender annotations.
- Datasets: CelebA contains 202,599 images from 10,177 identities, with Smiling, Male, and Young selected as expression, gender, and age attributes.Its binary smiling task supplements RAF-DB because full expression annotations are unavailable.
- Preprocessing: Both datasets are aligned, cropped, and normalized to 100 × 100 pixels before being fed into the networks.This standardizes face position and input size across experiments.
- Augmentation: Augmentation combines random cropping, small rotations, horizontal mirroring, and histogram equalization through a weighted summation.The crop size is 96 × 96, rotations range from -15° to 15°, and histogram equalization increases global contrast.
- Training: All three models use PyTorch and Adam with batch size 64, initial learning rate 0.001, scheduled decay, and early stopping.Maximum training is 200 epochs, with dataset-specific decay and stopping schedules.
- Evaluation: The evaluation reports both accuracy and fairness because standard metrics do not expose demographic performance differences.Fairness uses equal opportunity, aiming for similar results across sensitive-attribute groups.
- Evaluation: For non-binary sensitive attributes, fairness measures the largest accuracy gap among demographic groups relative to the dominant group.The dominant group is identified by summing class-wise accuracy.
5 Experiments and Results
Experiments on RAF-DB and CelebA compare baseline, attribute-aware, and disentangled approaches with and without data augmentation. On RAF-DB, augmentation improves accuracy but not fairness alone, while attribute-aware and disentangled models improve subgroup accuracy and fairness; CelebA shows little difference because its task is simpler and subgroup imbalance is less consequential.
- RAF-DB expression recognition: Data augmentation increased expression-recognition accuracy across almost all RAF-DB categories, with Happiness highest and Fear and Disgust lowest.The augmented baseline performed best, although its advantage over augmented attribute-aware and disentangled models was minimal.
- RAF-DB demographic analysis: Disentangled with data augmentation achieved the best RAF-DB class-wise accuracy across demographic groups, followed by the attribute-aware approach.Class-wise accuracy weights expression categories equally, reducing the influence of over-represented classes.
- RAF-DB fairness assessment: RAF-DB fairness gains from attribute-aware and disentangled approaches were strongest for unevenly distributed Age and Joint Gender-Race subgroups.The effect was less pronounced for Gender, whose subgroup distribution was more even.
- RAF-DB fairness assessment: Adding data augmentation improved baseline accuracy by approximately 7% on RAF-DB but did not mitigate its demographic bias.Both attribute-aware and disentangled approaches achieved further fairness improvements when equipped with augmentation.
- CelebA experiments: On CelebA, all six models produced comparable accuracy, class-wise subgroup accuracy, and fairness results for the Smiling versus Non-Smiling task.The target attribute was well balanced, while Gender and Age were unevenly distributed.
- CelebA discussion: CelebA offered less opportunity for improvement because it is more than ten times larger than RAF-DB, its binary task is simple, and its subgroup imbalance is less suited to the mitigation approaches.The authors therefore found visual failure-case examples uninformative for this dataset.
6 Conclusion
This paper investigates how demographic attributes are distributed in facial expression datasets and compares bias and fairness across three model approaches on RAF-DB and CelebA. Results show that mitigation benefits depend on dataset imbalance and model choice, while broader validation is needed because findings are data- and model-driven.
- The study compares baseline, attribute-aware, and disentangled approaches with and without data augmentation on RAF-DB and CelebA.
- Data augmentation improves baseline accuracy but does not alone mitigate bias; augmented attribute-aware and disentangled models outperform the baseline in accuracy and fairness.
- The disentangled approach is the best among the compared methods for mitigating demographic bias.
- CelebA-DB models show no significant accuracy or fairness differences, and data augmentation contributes little because the dataset is already large.
- The conclusions should be tested on additional facial-expression datasets and machine-learning models because current findings are data- and model-driven.