Source-linked AI summary
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
Kimmo Kärkkäinen, Jungseock Joo
TL;DR
Existing public face datasets underrepresent non-White races, limiting reliable comparison across demographic groups. FairFace constructs a balanced, labeled face dataset and evaluates it on novel datasets, finding better generalization and more consistent accuracy across race and gender groups.
Problem
Existing public face datasets strongly underrepresent non-White races, limiting model applicability and comparison across demographic groups.
Method
FairFace constructs a 108,501-image dataset with seven race groups, labeled for race, gender, and age, using primarily YFCC-100M images and adaptive country sampling.
Results
FairFace achieves better generalization for race, gender, and age on novel datasets, with balanced accuracy across race; gender classification has less than 1% male–female and White–non-White discrepancy.
Takeaways & Limitations
The dataset supports discovering and mitigating race and gender bias in computer vision systems and enables more consistent demographic performance.
Takeaways & Limitations
Perfect balance across all possible attribute co-occurrences is infeasible for in-the-wild datasets.
Abstract
from arXiv · showhide
Existing public face datasets are strongly biased toward Caucasian faces, and other races (e.g., Latino) are significantly underrepresented. This can lead to inconsistent model accuracy, limit the applicability of face analytic systems to non-White race groups, and adversely affect research findings based on such skewed data. To mitigate the race bias in these datasets, we construct a novel face image dataset, containing 108,501 images, with an emphasis of balanced race composition in the dataset. We define 7 race groups: White, Black, Indian, East Asian, Southeast Asian, Middle East, and Latino. Images were collected from the YFCC-100M Flickr dataset and labeled with race, gender, and age groups. Evaluations were performed on existing face attribute datasets as well as novel image datasets to measure generalization performance. We find that the model trained from our dataset is substantially more accurate on novel datasets and the accuracy is consistent between race and gender groups.
1. Introduction
Existing face datasets overrepresent Caucasian faces, limiting reliable performance across demographic groups. FairFace addresses this gap with a balanced, diverse dataset and reports stronger, more consistent attribute-classification results.
- Motivation: Existing public face datasets are strongly biased toward Caucasian faces, with non-White groups such as Latino significantly underrepresented.Around 80% of faces in most large-scale databases are described as lighter-skinned, including White faces.
- Motivation: This imbalance can prevent models from applying reliably to some subpopulations and complicate comparisons across demographic groups.The paper connects biased training data with biased models and fairness concerns.
- Dataset contribution: 108,501 facial images form a dataset emphasizing balanced representation across seven groups: White, Black, Indian, East Asian, Southeast Asian, Middle East, and Latino.Images were collected primarily from YFCC-100M, with additional examples from Twitter and online newspapers.
- Evaluation and contributions: The paper shows existing face attribute datasets and models generalize poorly to unseen data containing more non-White faces.It also reports better average performance and greater consistency across racial groups for the new dataset.
- Evaluation and contributions: The dataset’s inclusion of previously missing racial groups broadens the applicability of computer-vision methods to demographic analyses.The paper specifically highlights Latino and Middle Eastern groups and distinguishes East Asian from Southeast Asian.
2. Related Work
Face attribute recognition supports applications ranging from security to demographic research, but its datasets and models raise concerns about racial and gender imbalance. FairFace is presented as a mitigation strategy that improves diversity and generalization without claiming perfect balance.
- Face Attribute Recognition: Face attribute recognition classifies human attributes such as gender, race, age, emotions, and expressions from facial appearance.The task is reviewed here primarily through its underlying datasets.
- Face Attribute Recognition: Attribute recognition contributes to verification, person re-identification, device authentication, surveillance, marketing, and social-science demographic studies.These uses make balanced performance relevant to systems and analyses involving different demographic groups.
- Dataset Bias and Fairness: Commercial systems and public datasets have exhibited asymmetric performance, especially favoring male and light-skinned faces.The paper associates these disparities with biases introduced during image selection, capture, and dataset construction.
- Dataset Bias and Fairness: The paper focuses on balanced accuracy, defined as attribute-classification accuracy being independent of race and gender.It places this work within research on auditing bias and constructing better datasets.
- Dataset Bias and Fairness: Perfectly balancing every possible attribute co-occurrence is infeasible in an in-the-wild dataset, so the paper aims to mitigate rather than entirely solve dataset bias.FairFace instead collects more diverse non-White faces and reports improved generalization to novel datasets not dominated by White faces.
3. Dataset Construction
FairFace constructs its dataset around an explicit seven-group race taxonomy and a collection process designed to reduce selection and racial imbalance. Faces were sampled from YFCC100M, adaptively collected across countries, and annotated through worker agreement and refinement.
- 3.1. Race Taxonomy: The study defines seven race groups: White, Black, Indian, East Asian, Southeast Asian, Middle East, and Latino.
- 3.1. Race Taxonomy: Race is treated as distinct from ethnicity and as a non-discrete concept requiring explicit definition before data collection.
- 3.1. Race Taxonomy: Skin color is rejected as a sufficient race proxy because illumination affects it, within-group variation is large, and it is one-dimensional.
- 3.2. Image Collection and Annotation: The dataset seeks to reduce selection bias by detecting faces from YFCC100M without preselecting images or populations.
- 3.2. Image Collection and Annotation: 7,125 randomly sampled faces initialized collection, after which country sampling was adaptively adjusted to avoid White-race dominance.
- 3.2. Image Collection and Annotation: Three Mechanical Turk workers verified race, gender, and age, with disagreement triggering reannotation and possible image removal.
4. Experiments
Experiments compare FairFace with existing datasets using consistent architectures, cross-dataset tests, external datasets, and diversity analyses. FairFace generally improves novel-data accuracy and reduces performance disparities across demographic groups.
- Dataset Analysis: Existing face attribute datasets are predominantly White, while gender composition is comparatively more balanced.Race composition was assessed from reported statistics or annotations of 3,000 random samples per dataset; gender ratios generally ranged from 40%–60% male.
- Experimental Setup: Models were trained with an identical ResNet-34 architecture and evaluated through cross-dataset classification across race, gender, and age tasks.FairFace’s finer racial groups were merged when necessary to match other datasets, and CelebA was used only for gender classification because it lacks race annotations.
- Novel-Dataset Generalization: FairFace models outperform models trained on other datasets for race, gender, and age classification on novel datasets from different sources.Even FairFace models trained on 9k or 18k images outperform other datasets, including CelebA, which is larger than FairFace.
- Balanced Accuracy: FairFace produces more consistent race, gender, and age accuracy across race groups, measured by standard deviations across subpopulations.The study frames consistency using conditional use accuracy equality or equalized odds and reports the lowest maximum accuracy disparity for FairFace in Table 2.
- Balanced Accuracy: FairFace achieves less than 1% accuracy discrepancy between male–female and White–non-White groups for gender classification.Other models show stronger male and White-group advantages; LFWA+ has the largest reported gender performance gap at 32%.
- Data Coverage and Diversity: In embedding analyses, FairFace faces are broadly spread, whereas UTKFace faces are more tightly clustered and LFWA+ diversity appears partly influenced by a White-oriented embedding.The comparison uses t-SNE visualizations and pairwise L1 distances on 128-dimensional facial embeddings.
5. Conclusion
The paper introduces a face dataset balanced on race, gender, and age, and reports better generalization and more balanced accuracy across race than existing in-the-wild datasets.
- The dataset is balanced on race, gender, and age for face attribute research.
- It achieves better generalization for gender, race, and age classification on novel datasets containing more non-White faces.
- Models trained on the dataset produce balanced accuracy across race, whereas other datasets often yield asymmetric accuracy between racial groups.
- The dataset is intended to help discover and mitigate race and gender bias in computer vision systems used in society.