PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study on Face Recognition Biases Beyond Demographics

Philipp Terhörst, Jan Niklas Kolf, Marco Huber, Florian Kirchbuchner, Naser Damer, Aythami Morales, Julian Fierrez, Arjan Kuijper

arXiv:2103.01592v1cs.CV

TL;DR

Face-recognition bias research has largely focused on demographics, despite the importance of understanding a broader range of facial attributes. This study analyzes 47 attributes across two models with control groups to assess validity. It finds that many non-demographic attributes affect performance, with threshold patterns differing between the models.

  • Problem

    Prior work has largely focused on demographic bias, leaving the influence of a wider range of facial attributes on face-recognition performance to be examined.

    Source: §I · p. 1

  • Method

    The study compares FaceNet and ArcFace verification performance across 47 attributes using MAAD-Face and control-group validity assessments.

    Source: §I · p. 1, D. Control Groups · p. 3

  • Results

    Many non-demographic attributes strongly affect recognition performance; FaceNet’s attribute-related differentials are relatively stable across thresholds, while ArcFace’s often depend on the threshold.

    Source: §I · pp. 1–2

  • Takeaways & Limitations

    The findings motivate further advances in face-recognition robustness to covariates, explainability, and fairness.

    Source: §I · p. 2

    The evaluation covers FaceNet and ArcFace on MAAD-Face annotations based on VGGFace2 images.

    Source: §I · p. 1

Abstract

from arXiv · show

Face recognition (FR) systems have a growing effect on critical decision-making processes. Recent works have shown that FR solutions show strong performance differences based on the user's demographics. However, to enable a trustworthy FR technology, it is essential to know the influence of an extended range of facial attributes on FR beyond demographics. Therefore, in this work, we analyse FR bias over a wide range of attributes. We investigate the influence of 47 attributes on the verification performance of two popular FR models. The experiments were performed on the publicly available MAADFace attribute database with over 120M high-quality attribute annotations. To prevent misleading statements about biased performances, we introduced control group based validity values to decide if unbalanced test data causes the performance differences. The results demonstrate that also many non-demographic attributes strongly affect the recognition performance, such as accessories, hair-styles and colors, face shapes, or facial anomalies. The observations of this work show the strong need for further advances in making FR system more robust, explainable, and fair. Moreover, our findings might help to a better understanding of how FR networks work, to enhance the robustness of these networks, and to develop more generalized bias-mitigating face recognition solutions.

I. INTRODUCTION

The introduction motivates examining face-recognition bias beyond demographics, and the related work describes a field that has largely focused on demographic attributes. The paper studies 47 attributes across two models and reports that many non-demographic attributes also affect performance.

I. INTRODUCTION

  • Prior work reported demographic performance differences in face recognition, [4],,,,,, and research has developed several demographic-bias mitigation approaches.Approaches include adversarial learning, margin-based methods, data augmentation, metric learning, and score normalization [26], [41],, [45],, [34],,,,,.Source: §I · p. 1
  • The study examines differential outcomes for 47 attributes using FaceNet and ArcFace on MAAD-Face, with control groups and attribute-correlation analysis to qualify the results.MAAD-Face contains over 120M high-quality annotations for 3.3M face images; the study also presents a visual summary of performance differences and validity.Source: §I · p. 1
  • Many non-demographic attributes strongly affect recognition performance, while FaceNet’s attribute-related differentials are relatively stable across thresholds and ArcFace’s often depend on the threshold.The authors relate many differences to face visibility, temporal variability, and abnormality.Source: §I · pp. 1–2

II. RELATED WORK

  • Earlier face-recognition bias research mainly focused on demographics [18], while Terhörst et al. showed that non-demographic characteristics are also stored in face templates and might affect recognition performance.Source: §II · p. 2

A. Estimating Bias in Face Recognition

Earlier bias studies examined demographic factors, and mitigation work targeted demographic bias. This paper extends analysis to 47 attributes across two face embeddings.

A. Estimating Bias in Face Recognition

  • Studies of age reported lower biometric performance on children’s faces,, [16],, while gender studies reported weaker performance on female faces,, [2], [58].Ethnicity studies found weaker performance for underrepresented ethnicities and darker-skinned cohorts.Source: A. Estimating Bias in Face Recognition · p. 2

C. How This Work Contributes to the State of the Art

  • Table I contrasts earlier work on specific demographic attributes with this study’s wider range of demographic and non-demographic attributes.The comparison identifies testing identities and images for the reviewed studies.Source: Table I · p. 2

A. Estimating Bias in Face Recognition

  • Joint demographic studies reported error increases when race, gender, and age factors were jointly biased, and FRVT consistently found elevated false positives for female subjects and subjects at the outer ends of the age spectrum.Source: A. Estimating Bias in Face Recognition · p. 2

B. Mitigating Bias in Face Recognition

C. How This Work Contributes to the State of the Art

  • This work investigates 47 attributes across two popular face embeddings, extending prior analyses that focused on no more than seven attributes [43].The authors frame this wider range as a way to examine potential sources of differential outcomes beyond demographic factors.Source: C. How This Work Contributes to the… · p. 3

III. EXPERIMENTS ON MEASURING DIFFERENTIAL OUTCOME

The experiments use MAAD-Face annotations to compare verification performance for FaceNet and ArcFace across operating points. Positive and negative control groups help assess whether imbalanced attribute labels affect the validity of performance comparisons.

A. Database and Considered Attributes

  • MAAD-Face supplies over 120M high-quality annotations for 3.3M face images and 47 attributes spanning demographics, skin, hair, face geometry, facial regions, and accessories.Source: A. Database and Considered Attributes · p. 3

B. Face Recognition Models

  • FaceNet and ArcFace embeddings are compared using cosine similarity after face-image alignment, scaling, and cropping.Both models use ResNet-100 architecture and were trained on MS1M.Source: B. Face Recognition Models · p. 3

C. Evaluation Metrics

  • Verification performance is reported as FNMR at fixed FMR and EER, across EER and FNMR at 10^-3 and 10^-4 FMR operating points.For each operating point, performance is computed separately for positively and negatively annotated samples.Source: C. Evaluation Metrics · p. 3

D. Control Groups

  • For each attribute, six positive and six negative control groups match the sample counts of the corresponding labeled groups; similar control-group performances support validity, while strong differences indicate possible imbalance effects.Attributes with validity below 0.9 are treated as not valid, and validity values are reported so operators can choose a threshold suited to their applications.Source: D. Control Groups · pp. 3–4

E. Investigations

The investigations combine correlation analysis, performance comparisons, validity assessment, and interpretation of attribute-related differences across two face-recognition systems.

E. Investigations

  • The investigations examine correlations between attribute annotations to identify whether apparent attribute bias might originate from correlated labels.Source: E. Investigations · p. 4
  • For each attribute, the study compares recognition performance for positively and negatively labeled groups and interprets differences alongside validity values.The validity context is intended to avoid misinterpretations from unbalanced test data.Source: E. Investigations · p. 4
  • The study provides a visual summary relating attribute impacts to result validity, discusses possible explanations and differences between the two systems, and derives future research directions.Source: E. Investigations · p. 4

IV. RESULTS

FaceNet showed recognition differences across demographic and non-demographic attributes, while the attribute correlations and control-group validity criteria qualify how those differences should be interpreted. Correlated labels include both co-occurring appearances and semantically exclusive attributes.

A. Investigating the Correlation of Facial Attributes

  • The attribute correlations include positive associations among lipstick, earrings, heavy makeup, youth, and attractiveness, and negative associations with square face, male, and bags under eyes.The authors also note negative correlations between 5 o Clock Shadow and No Beard, and between Eyeglasses and No Eyewear.Source: A. Investigating the Correlation of… · p. 4

B. The Impact of Facial Attributes on Recognition

  • Positive and negative attribute groups were compared at three operating points, with control groups used to assess whether unbalanced data could explain performance differences.A result is considered valid when the relative performance between control classes is below 10% (validity ≥0.9).Source: B. The Impact of Facial Attributes on… · p. 4

1) Biases in FaceNet Embeddings:

  • FaceNet performance varied with demographics, accessories, hair, facial hair, face shape, and other facial attributes; some observed differences were attributed to visibility or correlated attributes.Asian and Black faces performed worse than White faces, and Young faces worse than Middle-aged faces; accessories and face-covering hair were associated with degraded performance, while several facial features performed better.Source: 1) Biases in FaceNet Embeddings · pp. 4–5
  • High cheekbones, double chins, chubby faces, and visible foreheads performed better, as did several periocular features and nose attributes.The authors suggest that these features may provide additional recognition information, while an obstructed forehead degrades performance.Source: 1) Biases in FaceNet Embeddings · p. 5
  • Smiling and closed-mouth expressions performed better than non-neutral expressions, which the authors suggest may reflect the expression distribution in the face databases.They contrast this with, where extreme expressions intended to avoid identification resulted in low comparison scores.Source: 1) Biases in FaceNet Embeddings · p. 5

2) Biases in ArcFace Embeddings:

ArcFace performance varied across demographic and non-demographic attributes, with several effects depending on decision threshold. The authors treated the Round Face result as invalid because unbalanced data could explain much of the difference.

2) Biases in ArcFace Embeddings:

  • ArcFace showed attribute-related performance differences, including threshold-dependent gender effects and weaker performance for Young than Middle-aged or Senior faces.The reported ethnicity differences favor White faces over Asian and Black faces, but the authors regard those results as invalid because control groups also show strong performance differences.Source: 2) Biases in ArcFace Embeddings · p. 5
  • Receding hairlines, wavy hair, and sideburns supported recognition, while bangs degraded it, with effects changing across decision thresholds.Wavy Hair's positive effect vanished at lower FMRs, while Bangs' negative effect increased.Source: 2) Biases in ArcFace Embeddings · p. 5
  • Gray Hair performed above average and Black Hair below average; Blond Hair and Brown Hair effects varied with decision threshold.Blond Hair improved recognition at high FMRs but fell below average at lower FMRs; Brown Hair's positive effect vanished at lower FMRs.Source: 2) Biases in ArcFace Embeddings · p. 5
  • Oval and Square Faces had positive, threshold-dependent effects, while the Round Face result was excluded because unbalanced data explained much of its performance difference.High cheekbones, double chins, chubby faces, and a fully visible forehead improved performance; Brown Eyes performed worse, while several facial anomalies performed better.Source: 2) Biases in ArcFace Embeddings · p. 5
  • Eyeglasses and hats degraded ArcFace recognition, while heavy makeup improved it; the authors link degradations to facial occlusion and attribute changes over time.They suggest frequent heavy-makeup use may produce more consistent training and testing images than hats or eyeglasses.Source: 2) Biases in ArcFace Embeddings · pp. 5–6

C. Performance Analysis

Figure 2 summarizes relative performance at the FMR corresponding to 10^-3 FNMR and encodes significance, direction, magnitude, and validity of attribute differences.

C. Performance Analysis

  • Figure 2 uses red and green to mark significant weaker or stronger performance for the positive attribute class, yellow for minor effects, and grey for results with validity below 90%.Relative performance is based on FMR at 10^-3 FNMR; validity compares positive and negative attribute-related control groups.Source: C. Performance Analysis · p. 6

1) FaceNet vs ArcFace:

FaceNet and ArcFace share network structure and training data but use different training principles. The authors suggest those principles may account for observed differences in attribute-related outcomes.

1) FaceNet vs ArcFace:

  • FaceNet uses triplet-loss learning to minimize intra-class variation and maximize inter-class variation, whereas ArcFace adds an angular large-margin principle aimed at improving robustness.Because the models share network structure and training data, the authors suggest their different training principles may underlie differences in recognition behavior.Source: 1) FaceNet vs ArcFace · p. 6

2) The Effect of Attributes on Recognition:

Across the two models, most investigated attributes strongly affected recognition, with differential outcomes often varying by decision threshold for ArcFace but remaining relatively constant for FaceNet. The effects span demographic and non-demographic attributes, and the cause of some effects remains unclear.

Demographics

  • Middle-aged, senior, White, and male faces performed better, while young, Asian, Black, and female faces performed worse; Asian/Black comparisons on ArcFace were invalid due to data imbalance.The demographic differential outcomes were stronger for FaceNet than ArcFace.Source: 2) The Effect of Attributes on… · p. 6

Visibility-related attributes

  • Fully visible foreheads, receding hairlines, no eyewear, and baldness were associated with improved recognition, while some partial occlusions degraded performance.The reported occlusion effects included obstructed foreheads, bangs, and wavy hair for FaceNet, and eyeglasses or bangs for ArcFace.Source: 2) The Effect of Attributes on… · p. 6

Temporary attributes

Anomalous characteristics

Facial expressions

  • Smiling faces and faces with closed mouths performed above average, while non-neutral expressions were associated with degraded performance.The authors suggest training data that usually contains neutral or smiling faces might explain this pattern.Source: 2) The Effect of Attributes on… · p. 6

Colors

  • Hair- and eye-color effects were strong and sometimes threshold-dependent; the reason for some attribute effects remains unclear.For example, the reported effect of blond hair changes direction at different false match rates.Source: 2) The Effect of Attributes on… · p. 6

Contrasting attribute effects

  • Some effects differed across models: wavy hair had opposite outcomes, no-beard faces performed below average, and wearing a necktie improved performance.The authors suggest necktie results might reflect correlations with hidden factors such as environment, but note the high validity and strong differential outcome.Source: 2) The Effect of Attributes on… · p. 11

Overall results

  • Most attributes strongly affected both models; FaceNet’s differential outcomes stayed relatively constant across thresholds, while ArcFace’s often varied by threshold.ArcFace also showed a stronger overall recognition performance than FaceNet.Source: 2) The Effect of Attributes on… · pp. 6, 12

3) Future challenges for face recognition:

The experiments point to robustness, fairness, and explainability challenges for face recognition. The authors call for broader approaches that consider many attributes together and use prior or general knowledge.

Need for robustness

  • Systems need greater robustness to partial occlusions, facial expressions beyond neutral or smiling, and temporary attributes that change daily appearance.Addressing these factors could enhance applicability in more real-life scenarios.Source: 3) Future challenges for face… · p. 12

Need for fairness

Need for explainability

Comprehensive approaches and transfer learning

  • The authors advocate comprehensive approaches that consider attributes together and exploit prior or general knowledge, rather than studying isolated attributes.They identify their analysis of 47 attributes as an example of a more comprehensive approach.Source: 3) Future challenges for face… · p. 12

V. CONCLUSION

The conclusion reports an analysis of 47 attributes across FaceNet and ArcFace, using control-group validity values to address unbalanced test data. Many attributes beyond demographics affected recognition, but some effects remain unexplained.

Study scope and method

  • The study compared two face-recognition models across 47 attributes using MAAD-Face, which has over 120M high-quality attribute annotations.The authors considered attribute correlations and used control-group validity values to minimize the effect of unbalanced testing data.Source: §V · p. 12

Findings

  • Many non-demographic attributes affected both models, with FaceNet’s differential outcomes relatively constant across thresholds and ArcFace’s strongly threshold-dependent.Source: §V · p. 12

Conclusion and limitations

  • Some attribute effects remain unexplained, and the authors call for further advances in face-recognition robustness, explainability, and fairness.Source: §V · p. 12
Loading 2103.01592v1…