Source-linked AI summary
Diversity in Faces
Michele Merler, Nalini Ratha, Rogerio S. Feris, John R. Smith
TL;DR
Face-recognition systems need balanced and diverse data, but existing datasets show demographic skews and facial diversity lacks comprehensive quantitative representation. DiF addresses this gap with one million annotated faces and ten interpretable coding schemes, whose analysis shows craniofacial measures capture greater variability than age and gender alone.
Problem
Existing face datasets have skewed age, gender, and skin-color distributions, while comprehensive quantitative representations of facial diversity remain needed for fair and accurate recognition.
Method
DiF samples YFCC-100M faces and applies ten scientifically grounded, computationally feasible, continuous-valued, human-interpretable facial coding schemes.
Results
Craniofacial measures show higher diversity and more evenness than age and gender measures, capturing information that age and gender alone do not.
Takeaways & Limitations
DiF provides a large annotated basis for studying facial diversity, data coverage, and balance in face-recognition research.
Takeaways & Limitations
Because YFCC-100M contributors are themselves skewed, DiF is not completely balanced on its own.
Abstract
from arXiv · showhide
Face recognition is a long standing challenge in the field of Artificial Intelligence (AI). The goal is to create systems that accurately detect, recognize, verify, and understand human faces. There are significant technical hurdles in making these systems accurate, particularly in unconstrained settings due to confounding factors related to pose, resolution, illumination, occlusion, and viewpoint. However, with recent advances in neural networks, face recognition has achieved unprecedented accuracy, largely built on data-driven deep learning methods. While this is encouraging, a critical aspect that is limiting facial recognition accuracy and fairness is inherent facial diversity. Every face is different. Every face reflects something unique about us. Aspects of our heritage - including race, ethnicity, culture, geography - and our individual identify - age, gender, and other visible manifestations of self-expression, are reflected in our faces. We expect face recognition to work equally accurately for every face. Face recognition needs to be fair. As we rely on data-driven methods to create face recognition technology, we need to ensure necessary balance and coverage in training data. However, there are still scientific questions about how to represent and extract pertinent facial features and quantitatively measure facial diversity. Towards this goal, Diversity in Faces (DiF) provides a data set of one million annotated human face images for advancing the study of facial diversity. The annotations are generated using ten well-established facial coding schemes from the scientific literature. The facial coding schemes provide human-interpretable quantitative measures of facial features. We believe that by making the extracted coding schemes available on a large set of faces, we can accelerate research and development towards creating more fair and accurate facial recognition systems.
1 Introduction
Data-driven AI systems depend on the data used to train them, making robust and diverse training data essential for fair and accurate face recognition. DiF addresses this need with one million annotated faces and ten scientifically grounded facial coding schemes.
- Robust and diverse training data are essential because AI systems learn what they are taught, affecting both accuracy and fairness.
- Face recognition requires sufficiently large, balanced, and diverse training data to represent the many ways faces differ.
- DiF provides annotations for one million face images sampled from YFCC-100M, covering intrinsic features, predictions, subjective annotations, pose, and resolution.
- The paper develops and analyzes DiF, reviewing existing datasets, constructing the dataset, implementing ten coding schemes, and statistically analyzing their outputs.
2 Related Work
Prior face-recognition datasets have enabled high benchmark accuracy but often represent narrow and uneven demographic distributions. These skews limit how readily benchmark performance translates to deployment and can produce biased models.
- LFW reached 99.8% accuracy, yet benchmark success on narrow datasets does not readily translate into equivalent deployment accuracy.
- Prominent face datasets differ substantially in age distributions, including 36.5% of IMDB-Face faces aged 20–30 versus 16.2% for IJB-C.
- LFW is 77.4% male, while all but two surveyed datasets contain more than 80% lighter faces under their respective skin-color groupings.
- PPB balances gender and skin type using dermatologist labels, but its adult-only parliamentary-photo source leaves age distribution skewed.
- Skewed training data produce biased models, with reported performance drops across face groupings and gender-estimation bias against dark-skinned females relative to white-skinned males.
3 DiF Data Set Construction
DiF was constructed as a scientifically grounded, multimodal dataset for studying facial diversity. Images from YFCC-100M were filtered and processed into approximately one million mostly frontal faces, but the source remains imperfectly balanced.
- DiF uses ten coding schemes selected for scientific basis, computational feasibility, continuous-valued outputs, and human interpretability.
- YFCC-100M was chosen for its scale, broad variety of photos, and Creative Commons availability, although Flickr contributors introduce source-population skew.
- The initial dataset excludes non-face, black-and-white or grayscale, and significantly blurred images to focus on intrinsic facial variation rather than color-processing variation.
- Candidate images were processed with Faster-RCNN face detection, DLIB pose and landmark extraction, size and pose filtering, and eye-based affine rectification.
- The pipeline produced rectified faces and expanded crops, filtering 100 million photos to approximately one million mostly frontal faces with adequate size.
4 Facial Coding Scheme Implementation
The DiF implementation uses ten scientifically grounded, computationally feasible, interpretable coding schemes to quantify multiple facial modalities. Craniofacial measures for schemes 1–3 are extracted from 19 landmarks mapped from DLIB’s 68 key-points.
- Ten coding schemes capture multiple facial-feature modalities using scientifically grounded, computationally feasible, numerically representable, and interpretable measures.
- The landmark table lists anatomical terms and abbreviations used to compute craniofacial measurements for coding schemes 1–3.
- DLIB’s 68 detected key-points are mapped to 19 facial landmarks for extracting craniofacial features.
- Coding scheme 1 comprises eight vertical craniofacial distances spanning the forehead, eyes, nose, mouth, and chin.
4.2 Coding Scheme 2: Craniofacial Areas
Coding scheme 2 represents facial structure through craniofacial areas, while the surrounding implementation describes landmark-based ratios and symmetry measures as additional quantitative facial descriptors.
- 4.2 Coding Scheme 2: Craniofacial Areas: Coding scheme 2 comprises twelve craniofacial area dimensions derived from facial landmarks mapped from DLIB key-points.
- Coding Scheme 3: Craniofacial Ratios: Coding scheme 3 comprises eight facial ratios using mapped DLIB key-points as landmarks; these ratios were used to estimate age progression in faces aged 0 to 18.
- Coding Scheme 4: Facial Symmetry: Coding scheme 4 contains two facial symmetry measures computed from density difference and edge orientation similarity.
- Coding Scheme 4: Facial Symmetry: Density difference reflects left-right relative intensity variation, whereas edge orientation similarity reflects zero-crossings of the intensity field.
- Coding Scheme 4: Facial Symmetry: Higher density-difference values indicate greater asymmetry, while higher edge-orientation-similarity values indicate greater symmetry.
4.5 Coding Scheme 5: Facial Regions Contrast
Coding scheme 5 quantifies contrast around the lips, eyes, and eyebrows in CIE-Lab channels. Prior work links facial contrast to age perception across several cultural and racial groups.
- Prior work found similar age-related facial-contrast changes across Chinese, Latin American, Black South African, and Caucasian faces.
- High-contrast faces were judged younger than low-contrast faces, and increasing age-related contrast features made faces look younger across tested groups.
- Contrast is computed as the difference between average pixel intensities in outer and inner regions around each facial part.
- The method repeats measurements across the L, a, and b CIE-Lab channels, producing nine total measures.
- Coding scheme 5 contains three facial-region contrast measures covering the lips, eyes, and eyebrows.
4.6 Coding Scheme 6: Skin Color
Coding scheme 6 represents facial skin color with Individual Typology Angle measurements. Pixel-level ITA values are aggregated over a masked facial region into one score per face.
- Because ITA is a point measurement, the method computes it for every skin pixel within a masked face region.
- A deep neural network extracts the facial skin mask, while DLIB key-points identify the chin, cheeks, and forehead regions.
- Regional ITA values are smoothed to reduce outliers, and each region’s peak value becomes its ITA score.
- The four regional ITA scores are averaged to produce a single ITA score for each face.
- Coding scheme 6 measures skin color using Individual Typology Angle (ITA).
4.7 Coding Scheme 7: Age Prediction
Coding scheme 7 uses the DEX convolutional neural network to produce a continuous age measure for each face, while acknowledging the model was trained in a narrow context and may not be fair.
- Coding scheme 7 uses the DEX model, a CNN fine-tuned on IMDB-wiki, to predict age from 0 to 100 years.DEX is based on a pre-trained VGG16-face identity network subsequently fine-tuned for age prediction.
- The authors use DEX initially as a continuous age-diversity measure, while noting that its narrow training context makes fairness uncertain.They frame balanced data and fairer models as requiring an iterative understanding of diversity.
- Each face is pre-processed by expanding its bounding box by 40% horizontally and vertically, resizing to 256x256, and center-cropping to 224x224.
- Age prediction is computed as the expected value of 101 softmax class probabilities corresponding to discrete years 0 through 100.The output combines probabilities p_i with year values y_i.
4.8 Coding Scheme 8: Gender Prediction
Coding scheme 8 adapts the age-prediction pipeline for gender, using the same preprocessing, neural network, and training process but producing a continuous score rather than a binary label.
- Gender prediction uses the same preprocessing steps, neural network model, and training pipeline as age prediction.
- The DEX model outputs a continuous gender score between 0 and 1 instead of only reporting male or female.
- Coding scheme 8 therefore represents gender prediction as a continuous-valued measure for diversity analysis.
4.9 Coding Scheme 9: Subjective Annotation
Coding scheme 9 captures age and gender through subjective human annotations rather than a neural-network predictive model.
- The study uses Figure Eight crowdsourcing to obtain human-labeled age and gender annotations for every DiF face image.
- Gender is annotated with two class labels: male and female.
- Age is annotated using seven groups ranging from [0-3] through [61-], together with a continuous age value.
5 Statistical Analysis
The statistical analysis quantifies diversity and evenness across the DiF coding schemes using Shannon and Simpson measures, alongside means and variances. Craniofacial features generally show greater diversity and evenness than age, gender, and pose measures, while facial contrast captures variation beyond simple age labels.
- Measurement framework: Shannon H and Simpson D measure diversity, while Shannon E and Simpson E measure evenness across feature distributions.For a uniform 20-class distribution, both evenness measures equal 1.0.
- Craniofacial features: The craniofacial distance, area, and ratio schemes contain 28 features, with their diversity scores summarized alongside the remaining coding dimensions in Table 12.
- Craniofacial features: 5.902 is the largest Simpson D value for craniofacial ratios, while their Shannon H values range from 1.781 to 1.783 and map to approximately six classes.
- Facial attributes: 5.510 is the highest Simpson D value for facial symmetry, whose highest evenness values are 0.918 for Simpson E and 0.975 for Shannon E.
- Age and gender: 4.368 and 1.601 are the Simpson D and Shannon H values for predicted age, with evenness around 0.624 and an equivalent of five classes.
- Discussion: Facial contrast has higher diversity and better evenness than neural-network or subjective age measures, whereas pose has lower diversity because mostly frontal faces were selected.
6 Summary and Future Work
DiF provides a methodology for analyzing facial diversity across one million images and 47 feature dimensions. Future work will compare datasets and use these analyses to identify coverage and balance gaps.
- DiF’s ten facial coding schemes yield 47 feature dimensions for measuring diversity across one million face images.The analysis uses diversity, evenness, and variance to characterize the dataset.
- The authors caution that all feature dimensions need not be maximally diverse and even.They present the approach as a needed methodology rather than a definitive target for every dimension.
- The proposed statistical analysis is intended to support comparisons of diversity across currently used face datasets.The authors plan to apply the same analysis to other datasets.
- These measures can help determine whether a dataset falls short in coverage and balance.The authors frame this as a way to assess whether one dataset is better than another.
- Future work may expand DiF, iteratively sample to fill gaps, and use generated faces to synthesize training data.The paper identifies larger datasets, proactive gap-filling, and GAN-based synthesis as possible directions.