PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Annotation Bias in Facial Expression Recognition

Yunliang Chen, Jungseock Joo

arXiv:2108.08504v1cs.CVcs.AIcs.LGcs.MM

TL;DR

Human annotations are often assumed unbiased, but this paper investigates systematic annotation bias in facial expression recognition and compares lab-controlled with in-the-wild datasets. It proposes AUC-FER, which uses facial action units and triplet loss; the authors report stronger bias mitigation than existing techniques.

  • Problem

    Most debiasing work addresses image-composition bias, while the assumption that human-generated labels are unbiased leaves systematic annotation bias in facial expression recognition underexamined.

    Source: §1 · pp. 1–2

  • Method

    AUC-FER uses facial action units and incorporates triplet loss to learn expression classifications for people with similar action units.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    AUC-FER is reported to reduce annotation bias more effectively than existing debiasing methods, while systematic bias is reported to be more severe in in-the-wild than lab-controlled datasets.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Systematic annotation bias in facial-expression datasets can be learned by trained models and should not be assumed to be random noise.

    Source: §1 · p. 2, §5 · p. 8

    The study focuses on systematic annotation bias, and future work could examine variation across annotator subgroups and how emotion definitions interact with bias.

    Source: §5 · p. 8

Abstract

from arXiv · show

The performance of a computer vision model depends on the size and quality of its training data. Recent studies have unveiled previously-unknown composition biases in common image datasets which then lead to skewed model outputs, and have proposed methods to mitigate these biases. However, most existing works assume that human-generated annotations can be considered gold-standard and unbiased. In this paper, we reveal that this assumption can be problematic, and that special care should be taken to prevent models from learning such annotation biases. We focus on facial expression recognition and compare the label biases between lab-controlled and in-the-wild datasets. We demonstrate that many expression datasets contain significant annotation biases between genders, especially when it comes to the happy and angry expressions, and that traditional methods cannot fully mitigate such biases in trained models. To remove expression annotation bias, we propose an AU-Calibrated Facial Expression Recognition (AUC-FER) framework that utilizes facial action units (AUs) and incorporates the triplet loss into the objective function. Experimental results suggest that the proposed method is more effective in removing expression annotation bias than existing techniques.

1. Introduction

The introduction argues that image-composition debiasing does not address biased annotations, which can also shape model predictions. It investigates systematic annotation bias in facial expression recognition and proposes AUC-FER, which uses facial action units to reduce that bias.

1. Introduction

  • Large web-scraped, crowd-annotated datasets provide diverse images quickly and at low cost, but can introduce dataset bias.Traditional lab-controlled datasets are often too small and insufficiently diverse for robust model training.Source: §1 · p. 1
  • Most debiasing methods focus on image-composition bias in P(X), while biased labels can leave the joint distribution P(X, Y) biased.The introduction distinguishes dataset bias from bias in the annotation process.Source: §1 · p. 2
  • The authors identify systematic annotation bias as a concern in subjective tasks and note that objective measures such as action units can be used in their approach.They also propose adapting the framework to other subjective-labeling tasks when suitable objective measures are available.Source: §1 · p. 2
  • In-the-wild facial-expression datasets often contain systematic annotation bias, unlike the assumption that annotation errors are merely random noise.The paper compares in-the-wild and lab-controlled datasets and reports that the bias is more severe in the former.Source: §1 · pp. 1–2
  • Trained models learn systematic annotation bias, so it cannot be ignored when assessing model bias.The paper frames this as a concern for fairness in facial expression recognition.Source: §1 · pp. 1–2
  • AUC-FER uses facial action units and is reported to mitigate expression annotation bias more effectively than existing debiasing techniques.The framework also incorporates triplet loss into its objective.Source: §1 · p. 2

2. Related Work

Related work covers fairness and debiasing methods, prior assumptions about crowd annotations, and facial-expression research. It situates this paper's focus on systematic annotation bias alongside work on dataset composition bias and expression recognition.

Fairness

  • Fairness research distinguishes group fairness from individual fairness, which requires similar individuals to receive similar outcome distributions [27] [20].Counterfactual fairness asks whether a decision would remain unchanged if a person belonged to a different demographic group, all else held fixed.Source: §2 · p. 2

Debiasing techniques

  • Debiasing work spans composition-bias methods such as transfer learning and domain adaptation, fairness-aware representations, and label correction or reweighting [80] [81] [82] [69] [91] [35] [60].For historically biased labels, data massaging changes labels before training or applies weights or sampling during training.Source: §2 · pp. 2–3

Annotation bias

  • Prior work on crowd-annotated in-the-wild datasets generally treats annotation errors as random noise, while this paper examines whether systematic label bias exists.Earlier work also studied annotation accuracy, multi-annotator label compilation, and in-batch annotation bias.Source: §2 · p. 3

Facial expression recognition and facial action units

  • Facial action units objectively code fundamental muscle actions associated with expressions; the cited framework of six prototypical emotions includes anger, disgust, fear, happiness, sadness, and surprise.Prior work also reports gender-linked differences in perceived happiness and in detecting angry and happy expressions.Source: §2 · p. 3
  • Expression-recognition composition bias includes underrepresented, harder-to-learn classes such as disgust, anger, fear, and surprise, prompting methods such as re-sampling and confusion loss [87].Prior psychological findings and model-bias observations motivate the hypothesis that systematic annotation bias may also contribute to gender bias in trained models [17].Source: §2 · p. 3

3. Annotation Bias in Expression Datasets

The section tests whether expression labels differ by gender after facial action units are controlled, using OpenFace AUs as an objective benchmark. It finds systematic happy and angry annotation differences in in-the-wild datasets, unlike lab-controlled datasets.

3. Annotation Bias in Expression Datasets

  • The analysis uses facial action units as objective expression benchmarks, focusing on AU6 and AU12 for happiness and the corresponding action units for anger.The paper also examines age and race, but notes that most evaluated datasets are dominated by younger and white people.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1. Facial Action Units (AUs) Recognition

  • OpenFace AU6 and AU12 recognition is imperfect but shows no significant male–female accuracy differences on EmotioNet, supporting its use as a proxy benchmark for happiness annotations.OpenFace thresholds for AU presence were selected to optimize classification accuracy against EmotioNet’s expert-coded annotations.Source: §3.1 · p. 4

3.2. Expression Annotation Bias

  • Conditioning emotion labels on AUs distinguishes annotation bias from data-composition bias, because the AUs quantify expression independently of the emotion labels.The paper defines unbiased expression annotations as labels independent of gender when the AUs are held fixed.Source: §3.2 · p. 4

3.3. Evaluation on Various Datasets

  • Happy-label proportions differ by gender after controlling for AU6 and AU12 in all four in-the-wild datasets; lab-controlled datasets show no such difference, and anger labels show the same in-the-wild versus lab pattern.For happiness, the difference is smaller in RAF-DB than in ExpW and AffectNet; males are more likely than females to be labeled angry in all in-the-wild datasets.Source: §3.3 · pp. 5–6, Table 2 · p. 6
  • Younger people are more likely to be labeled happy than older people, with the bias varying by dataset; the analysis finds no systematic race-related annotation bias.The authors note that further analysis requires more balanced age and race representation.Source: §3.3 · p. 6
  • The authors suggest that holistic emotion judgments may be more affected by perceived gender than judgments of local facial action units.They offer this as an explanation for why happy and angry labels show bias while many AU labels do not.Source: §3.3 · p. 6

4. Bias Correction

The section shows that annotation bias can transfer to trained models, then presents AUC-FER as a way to address it without changing labels. AUC-FER achieves lower discrimination scores than other debiasing methods in the reported experiments.

4.1. Learned Bias in Trained Models

  • Models trained on raw ExpW show significant prediction bias on CFD, while models trained on relabeled ExpW show little bias.The test set balances happiness proportions between genders; its ground-truth happy proportion is 0.3629 for both genders.Source: §4.1 · p. 7, Table 3 · p. 7

4.2. Bias Correction

  • AUC-FER combines classification cross-entropy and triplet loss with a weight λ, avoiding the need to modify training labels.The triplet-loss margin α controls the embedding-distance constraint.Source: §4.2 · p. 7
  • AUC-FER uses AU similarity to encourage similar model treatment across samples, even when their genders and labels differ.Its triplets pair images with the same AU presence and contrast them with an image having different AU presence.Source: §4.2 · p. 7

4.3. Experiments

  • AUC-FER achieves the lowest discrimination score, reducing bias by 64–89% versus baseline models and coming close to relabeling results.The experiments also report that AUC-FER outperforms other debiasing techniques for angry-expression recognition when trained on AffectNet-Automatic.Source: §4.3 · p. 8, Table 4 · p. 8

5. Discussion

The discussion argues that facial-expression datasets contain systematic annotation biases, and presents AUC-FER as a way to reduce them using facial action units. It also identifies limits and directions for further study.

5. Discussion

  • Systematic annotation bias appears in many facial-expression datasets, is more severe in in-the-wild data, and can also bias trained models.The authors characterize their work as the first in computer vision to demonstrate annotators’ perceptual bias as a potential source of model bias.Source: §5 · p. 8
  • AUC-FER uses action units as auxiliary measures, which the authors argue are fairer calibration signals but may be inaccurate or noisy.They suggest extending the framework with measures such as body keypoints for subjective tasks including activity recognition or image captioning.Source: §5 · p. 8
  • The paper focuses on systematic annotation bias; future work could examine annotator subgroups, compound emotions, and combinations with composition-bias methods.The authors note that six prototypical expressions do not capture the full range of facial expressions.Source: §5 · p. 8

Supplementary Material: Understanding and Mitigating Annotation Bias

The supplied passages identify a supplementary-material section and provide author, affiliation, and version metadata.

Supplementary Material

1. Angry Annotation Bias Between Genders

Angry-label bias is prominent in the in-the-wild datasets examined, while the lab-controlled datasets show no gender difference after controlling for action units. AUC-FER achieves the lowest discrimination score among the compared methods.

1.1. OpenFace Accuracy for Angry AUs

  • Recalibrating AU thresholds removes statistically significant gender differences in AU accuracy, so subsequent evaluations calibrate AU4 only.AU5 shows little difference before and after calibration; AU7 and AU23 are not assessed in this accuracy comparison because EmotioNet lacks their labels.Source: §1.1 · p. 13, Table 1 · p. 13

1.2. Annotation Bias of the Angry Expression

  • The angry-label comparisons condition gender proportions on marginal AU values and use chi-square tests, rather than testing all joint AU combinations.The authors report that many joint combinations have insufficient data and that correlated AUs make some combinations more common.Source: §1.2 · p. 13
  • For KDEF and CFD, angry-label distributions are independent of gender after controlling for AUs, whereas ExpW, RAF-DB, and AffectNet show significantly higher angry-label proportions for males.In-the-wild datasets also show large male-female discrepancies in angry-label proportions as AU intensities vary.Source: §1.2 · pp. 13–14, Table 2 · p. 14

1.3. Bias Correction for the Angry Expression

  • In Table 3, all debiasing methods reduce bias relative to baseline, and AUC-FER achieves the lowest discrimination score, corresponding to a 72% bias reduction versus baseline.Source: §1.3 · p. 15

2. Happy Annotation Bias Across Age Groups

Across in-the-wild datasets, younger age groups generally receive more happy labels, although the datasets are heavily weighted toward younger people. The analysis also finds a larger bias for AU6 than AU12.

Age-group method

  • Because only RAF-DB includes age labels, the authors trained a ResNet-34 age classifier using FairFace and applied it to KDEF, CFD, ExpW, and AffectNet, grouping predictions as less than 19, 20–39, 40–59, and more than 60.Source: §2 · p. 15

Age-group scope

  • Because the lab-controlled datasets contain too few images of people older than 40, the study cannot evaluate age-related annotation bias in those datasets.Only 26 of 1,207 images were predicted to depict people older than 40.Source: §2 · p. 16

Happy-label results by age

  • Younger age groups have higher proportions of happy labels in general, with statistically significant differences in most cases.The datasets are heavily dominated by younger people, and OpenFace fails to produce AU labels for a small fraction of images.Source: §2 · p. 16, Table 4 · p. 16

Age comparisons and AU patterns

  • Younger people are more likely than older people to be annotated as happy, although the strength of the pattern varies across datasets.The figures indicate that AU6 exhibits a larger bias than AU12.Source: §2 · p. 16, Figure 3 · p. 17, Figure 2 · p. 17

3. Happy Annotation Bias Across Racial Groups

The happy-label analysis finds no consistent pattern of racial or ethnic annotation bias across the in-the-wild datasets. The study restricts this analysis to those datasets because the lab-controlled datasets are small and lack population diversity.

Scope and data limitations

  • The racial-bias analysis is limited to in-the-wild datasets because lab-controlled datasets are small and lack diversity, including only 100 Asian and 100 Latino images in CFD.Many in-the-wild datasets other than ExpW are also heavily dominated by white people, making statistical tests challenging.Source: §3 · pp. 17–18

Racial-group method

  • The analysis uses six racial or ethnic groups, combining East and Southeast Asian categories, and also reports results for three major groups.RAF-DB supplies labels for only White, Black, and Asian groups; the authors additionally examine three major groups because some datasets have few images in several categories.Source: §3 · p. 18

Happy-label results across racial groups

  • Racial and ethnic differences in happy-label proportions are generally minor, with no consistent bias pattern across the in-the-wild datasets.Some differences are statistically significant, but the reported direction is not consistent across groups and datasets.Source: §3 · pp. 18–19, Figure 4 · p. 19
Loading 2108.08504v1…