Towards Measuring Fairness in AI: the Casual Conversations Dataset

1 INTRODUCTION

• Imbalanced training datasets can make high-performing models perform poorly for underrepresented groups, motivating fairness work in computer vision.The introduction argues that results from unbalanced datasets may not reflect real-world outcomes.Source: §1 · p. 1
• The paper asks whether deepfake detectors reliably detect faces across age, gender, and skin-tone groups, and how often face-detector bias contributes to failure.It also asks whether detector vulnerabilities can be measured.Source: §1 · p. 1
• Casual Conversations contains recordings of 3,011 diverse individuals who consented to likeness use, with participant-provided age and gender labels plus apparent skin-tone and low-light annotations.The apparent skin-tone labels use the Fitzpatrick scale.Source: §1 · p. 1
• The dataset is also proposed for image inpainting, temporally consistent models, audio understanding, facial-attribute classification, and low-light scenarios.The authors present these as potential application areas beyond robustness evaluation.Source: §1 · p. 2

2 RELATED WORK

• Prior fairness work links imbalanced category distributions to subgroup favoritism and highlights lighter-skin representation gaps in IJB-A and Adience.The review also covers ethical concerns and fairness mitigation approaches.Source: §2 · p. 2
• Studies report demographic differentials in face-recognition systems, including higher false-match rates for African-American cohorts and persistent accuracy gaps between “man” and “woman” categories.The review notes that most earlier work focused on false negatives.Source: §2 · p. 2
• Existing facial-attribute datasets commonly use third-party labels, which may diverge from people’s self-identification and introduce annotation inaccuracies.The text contrasts these labels with the paper’s participant-provided age and gender annotations.Source: §2 · p. 2
• They chose apparent skin tone rather than ethnicity, which they describe as subjective and potentially confusing; annotators voted on sampled frames from an average of 15 recordings per actor under varying ambient lighting.They note color correction may still be necessary to improve inter-rater agreement.Source: §2 · p. 3
• The review identifies unresolved questions about deepfake-detector robustness across age, gender, apparent skin tone, and ambient lighting.It also notes uncertainty about how face detectors perform across subgroups.Source: §2 · p. 3
• Prior apparent age and gender classifiers had not been evaluated across apparent skin types and ambient lighting, motivating tests of their robustness on this dataset.The reviewed methods include an end-to-end CNN, a lightweight multi-task CNN, and a hybrid face-recognition framework.Source: §2 · p. 3

Dataset composition

• 3,011 subjects contributed approximately fifteen one-minute videos each, recorded in five U.S. cities; some videos include two people but label only the subject of interest.Participants were adults aged 18 or older and represented diverse ages, genders, and skin tones.Source: §3 · p. 3

Annotations

• Age and gender labels come from subjects themselves, while the dataset also records apparent skin tone and whether videos were captured in low ambient lighting.Gender categories are Male, Female, Other, and N/A; the authors note this categorization is oversimplified.Source: §3 · p. 3
• The Fitzpatrick scale supplies six apparent-skin-tone types, but the paper notes limitations in representing diversity beyond Western theories and for darker skin tones.The authors provide full, non-bucketed types so users can decide how to group them.Source: §3 · pp. 3, 5
• Eight raters annotated apparent skin type, with the most-voted type from weighted votes serving as the ground-truth annotation.Uncertain votes count as half.Source: §3 · p. 5

Scope and use

• The mini dataset contains 6,022 videos, including 37.3% dark videos after lighting-focused resampling; the full dataset is limited to evaluation because it is relatively small and U.S.-collected.Resampling selects one video pair per subject and affects the lighting distribution; the dataset cannot be used to train models with its provided labels.Source: §3 · p. 5

Distribution

• Paired category distributions are generally fairly uniform across subgroups, alongside decently balanced age and gender distributions and near-equal combined Fitzpatrick-type groups.The authors argue that uniform annotation distributions reduce bias's impact on measurements and support robustness evaluation.Source: Figure 3 · p. 4, §3 · p. 5

4 EXPERIMENTS

• All DFDC winners struggle on darker skin tones; all but NTechLab are particularly biased toward paler Type I skin, while Eighteen Years Old performs best on ages 46–85.NTechLab has the best weighted precision on this private-test subset, despite not winning the original competition.Source: §4 · p. 5, Table 1 · p. 6
• NTechLab is more consistent across age, gender, and lighting, while NTechLab and Eighteen Years Old have the lowest overall log-loss; log-loss rises sharply on darker skin tones.The winner comparison covers 4,945 overlapping DFDC test videos.Source: §4 · p. 5, Table 2 · p. 6
• Most false-negative ratios align with category distributions except for darker skin types V and VI, and example failures show all top winners struggling with darker-skinned subjects.The false negatives are benign real samples; Fig. 4 displays unperturbed face crops.Source: §4 · pp. 5–6, Figure 4 · p. 5
• The DFDC subgroup differences have several possible contributors, including imbalance in the rest of DFDC, training-time sampling or augmentation, and models trained without fairness labels.The authors do not isolate a single cause for the unequal performance.Source: §4 · p. 6

4.1 Apparent age and gender prediction

• On the mini dataset, state-of-the-art apparent gender precision is more than 20% lower on average for darker skin types V and VI; LightFace is more successful on apparent age and gender.Levi & Hassner is less accurate overall, though its apparent-gender classification is almost as good as LMTCNN’s.Source: §4.1 · p. 6

• The dataset measures robustness across age, gender, apparent skin type, and lighting, with uniform distributions over all categories and age and gender labels supplied by participants.Source: §5 · p. 6
• All five DFDC winners showed bias toward lighter skin tones, mostly failing on darker-skinned subjects.Source: §5 · p. 6
• Apparent age and gender models also showed bias toward lighter-skinned subjects; gender classifiers performed best on people over 45 and similarly on darker and brighter videos.Source: §5 · p. 6
• The authors state that the dataset enables researchers to develop and thoroughly evaluate models for more inclusive and responsible AI.Source: §5 · p. 6