Source-linked AI summary
An Examination of Fairness of AI Models for Deepfake Detection
Loc Trinh, Yan Liu
TL;DR
Deepfake detectors are increasingly relevant, but evidence on their performance across diverse racial and gender groups is limited. The paper audits datasets and three detectors using demographically informed facial data, finding racial performance disparities alongside dataset representation bias and subgroup discrimination associated with Blended Images.
Problem
Existing deepfake-detection research gives limited attention to performance across gender and race, despite the importance of low false-positive rates in detection.
Source: §1 · p. 2Method
The authors audit dataset representation and evaluate three popular deepfake detectors using facial datasets balanced by race and gender.
Source: Auditing datasets · p. 3, §4.3 · p. 4Results
Up to 10.7% difference in error rate is found between racial subgroups, and FaceForensics++ is overwhelmingly composed of Caucasian subjects, with female Caucasians accounting for 36.6% of videos.
Source: §1 · p. 2Takeaways & Limitations
The findings underscore the importance of benchmark representation and intersectional auditing for demographic transparency and accountability in AI systems.
Source: §5 · pp. 6–7The authors cannot pinpoint the exact source of bias, so Blended Images alone may not be responsible for misclassification and large false-positive disparities.
Source: Results · p. 6
Abstract
from arXiv · showhide
Recent studies have demonstrated that deep learning models can discriminate based on protected classes like race and gender. In this work, we evaluate bias present in deepfake datasets and detection models across protected subgroups. Using facial datasets balanced by race and gender, we examine three popular deepfake detectors and find large disparities in predictive performances across races, with up to 10.7% difference in error rate between subgroups. A closer look reveals that the widely used FaceForensics++ dataset is overwhelmingly composed of Caucasian subjects, with the majority being female Caucasians. Our investigation of the racial distribution of deepfakes reveals that the methods used to create deepfakes as positive training signals tend to produce "irregular" faces - when a person's face is swapped onto another person of a different race or gender. This causes detectors to learn spurious correlations between the foreground faces and fakeness. Moreover, when detectors are trained with the Blended Image (BI) dataset from Face X-Rays, we find that those detectors develop systematic discrimination towards certain racial subgroups, primarily female Asians.
1 Introduction
Deepfake detectors are increasingly used against realistic deceptive media, yet their performance across racial and gender groups has received limited attention. This study audits subgroup disparities and dataset bias, finding racial performance gaps, underrepresentation in FaceForensics++, irregular face swaps, and discrimination associated with Blended Images.
1 Introduction
- Deepfakes can portray altered identities or malicious or embarrassing actions, and the introduction describes them as a vehicle for misinformation.The authors motivate automated detection amid increasingly realistic synthetic media and documented malicious uses.Source: §1 · p. 1
- Research has given little attention to how deepfake systems perform across gender and race, despite low false-positive detection remaining challenging.The introduction notes that subgroup false-positive disparities could mean members of a particular group are more likely to be mistakenly classified as fake.Source: §1 · p. 1
- FaceForensics++ is overwhelmingly composed of Caucasian subjects, with female Caucasians accounting for 36.6% of videos.The authors identify dataset representation as a central contribution of their audit.Source: §1 · p. 2
- Deepfake-generation approaches tend to produce irregular swaps across race or gender, leading detectors to learn spurious correlations between foreground faces and fakeness.An irregular deepfake swaps a person's face onto someone of a different race or gender.Source: §1 · p. 2
- Up to 10.7% difference in error rate separates racial subgroups in deepfake detection classifiers evaluated on datasets balanced by gender and race.The introduction frames this as a predictive disparity across racial groups.Source: §1 · p. 2
- Training detectors with Face X-Rays Blended Images is associated with systematic discrimination, primarily toward female Asian subjects.The contribution specifically identifies female Asian subjects as affected.Source: §1 · p. 2
2 Related Work
Prior work develops deepfake detectors and studies generalization, robustness, and algorithmic fairness, but this paper examines fairness across demographic groups in deepfake detection.
2.1 Deepfake detection
- Deepfake detectors use hand-crafted facial cues, CNNs, and methods targeting spatial, blending, or temporal artifacts.Examples include MesoInception4, Xception, Face X-ray, and dynamic prototypes.Source: §2.1 · p. 2
2.2 Generalizability and robustness of detectors
- Detector performance can drop on unseen manipulation types, while prior work addresses generalization and adversarial vulnerability.ForensicTransfer and Face X-ray target generalization; small tailored perturbations can fool detectors.Source: §2.2 · p. 2
2.3 Algorithmic fairness and consequences
- Fairness research has studied discrimination and documentation practices, while this work audits deepfake detection using face-swap racial distributions and subgroup-specific fakes.The cited context includes recommendations for model cards and datasheets.Source: §2.3 · p. 2
3 Deepfake Detection
The paper evaluates three popular deepfake detectors trained on FaceForensics++ and cross-tests their generalizability on datasets with unseen manipulations. Figure 1 presents demographic distributions and example and average faces for RFW and UTKFace.
3 Deepfake Detection
- Three detectors—MesoInception4, Xception, and Face X-Ray—are trained on FaceForensics++ and cross-tested on datasets containing unseen manipulations.The cross-test datasets include Google’s DeepfakeDetection, Celeb-DF, and DeeperForensics-1.0.Source: §3 · p. 2
- Figure 1 shows examples and average faces from RFW and UTKFace, with racial groups ordered Caucasian, African, Asian, and Indian from top to bottom.The figure also presents the datasets’ gender and racial distributions.Source: Figure 1 · p. 3
4 Deepfake Detection Audit
The audit evaluates three deepfake detectors across demographic groups using facial datasets and subgroup metrics. It finds substantial racial disparities, especially in false-positive and error rates for detectors trained with Blended Images.
Auditing datasets
- The audit adapts RFW and UTKFace, preserving RFW’s test distribution and downsampling UTKFace to balance racial groups.UTKFace images were used to create 40K blended images per subgroup, with swapped faces constrained to the same gender and racial group.Source: Auditing datasets · pp. 3–4
- Figure 2 shows blended images and their Face X-Rays for examples with low and high artifact levels.The audit’s goal is to evaluate detectors on real and manipulated images with faithful demographic labels.Source: Figure 2 · p. 4, Auditing datasets · p. 4
Evaluation metrics
- The evaluation reports AUC and subgroup accuracy, TPR, FPR, and error rate, choosing a threshold that maximizes balanced accuracy on FaceForensics++ validation data.AUC is also included because it is robust to class imbalance.Source: Evaluation metrics · p. 4
Gender groups audit
- Gender-group error rates differ by only 0.1–0.3%, but Xception + BI has a 14.0% FPR for female subjects versus 7.7% for male subjects.Xception + BI error rates are 9.9% for female subjects and 9.8% for male subjects.Source: Gender groups audit · p. 4
Racial and intersectional subgroups audit
- Across detectors, intersectional subgroup error-rate differences reach 10.7%, while BI-trained models show especially high FPRs for Asian faces and low TPRs for African faces.Asian faces have the highest FPR, especially female Asian faces, with 5.2–8.1% differences across BI-trained detectors; African faces have the lowest TPR, especially male African faces, with 4.7–10.7% differences.Source: Racial and intersectional subgroups… · p. 4, §4.3 · p. 4, Table 1 · p. 5
4.1 Key findings on evaluated detectors
- All detectors trained with BI perform worst on African faces, especially male African faces, with 3.5–6.7% error-rate differences from the best subgroup.The worst error rates are 41.3% for female African faces with MesoNet + BI, and 13.0% and 13.6% for male African faces with Xception + BI and FaceXRay + BI, respectively.Source: §4.1 · p. 3, Racial and intersectional subgroups… · p. 4
Analysis of results
- Despite AUC up to 0.962 and detection accuracy up to 90.1%, intersectional analysis reveals disparities that overall metrics do not capture.The authors argue that a single dataset-wide metric is not enough to justify broad commercial deployment.Source: Analysis of results · p. 4
- The authors report that BI training improved overall predictive performance for the evaluated detectors but negatively affected predictions on real videos and images.They suggest that detectors’ focus on manipulation artifacts may contribute to unintended consequences when those artifacts are absent from genuine media.Source: Analysis of results · p. 4
4.4 Training distribution and methodology bias
The section examines demographic representation in FaceForensics++ (FF++) and the swap patterns in FF++ and Blended Image (BI). It reports substantial representation imbalance and frequent cross-race or cross-gender swaps, while cautioning that BI alone may not explain disparities in false-positive rates.
Evaluation Methodology
- The analysis manually labels FF++ subjects by perceived gender and race, using three annotators and majority labels; inter-rater reliability is 75.93%.A swap is defined as irregular when a face is placed onto a face of a different gender or race.Source: Evaluation Methodology · p. 6
Results
- 61.7% of FF++ real videos contain Caucasian subjects, including 36.0% female Caucasians; fewer than 5% contain African or Indian subjects.Female subjects account for 58.3% of the videos, and male Indian subjects are the least represented subgroup.Source: Results · p. 6, Figure 4 · p. 6
- 59.44% of FF++ fakes and 65.45% of BI images are irregular swaps, with female Asian foreground faces often swapped onto female Caucasian faces.For BI female Asian foregrounds, 35.3% are swapped onto female Caucasian faces, 21% onto female Asian faces, and 14% onto female Hispanic faces; the authors say this pattern may encourage a correlation between Asian facial features and fakeness.Source: Results · p. 6, §4.4 · p. 6
- The authors caution that BI alone may not be responsible for misclassification and large disparities in false-positive rates, and suggest racially aware blending as future work.Source: Results · p. 6
5 Conclusion
The conclusion reports disparities in detector performance across races and representation bias in FaceForensics++. It also connects irregular swaps to the training data and calls for benchmark representation and intersectional auditing.
5 Conclusion
- The authors report large predictive disparities across races, representation bias in FF++, and that most fakes are irregular swaps between different genders or races.They argue for benchmark representation and intersectional auditing to increase demographic transparency and accountability in AI systems.Source: §5 · pp. 6–7
A Appendix
The appendix includes a table presenting basic information about the training and testing datasets.
A Appendix
- Table 2 presents basic information about the training and testing datasets.Source: Table 2 · p. 8
A.1 Detection methods
The appendix describes three detection methods: MesoInception4, XceptionNet, and Face X-Ray. They differ in architecture and supervision, with Face X-Ray additionally using boundary-focused signals from blended images.
A.1 Detection methods
- MesoInception4 is a shallow network designed to focus on mesoscopic image properties.Source: §A.1 · p. 8
- XceptionNet is an ImageNet-trained CNN fine-tuned on FF++ for deepfake detection, using separable convolutions and residual connections.It requires facial-landmark detection and frame cropping before images are fed to the network.Source: §A.1 · p. 8
- Face X-Ray generates blended images by combining masked regions from two faces, then uses mask-derived Face X-Rays as supervised labels for swap boundaries.Source: §A.1 · p. 8
- Unlike the other two models, Face X-Ray is additionally trained with cross-entropy loss over Face X-Rays, encouraging soft binary maps focused on visual boundary artifacts.Source: §A.1 · p. 8
A.2 Training and testing datasets
Training uses FaceForensics++ (FF++), while evaluation also tests generalization on three datasets with different video sources and manipulation characteristics.
A.2 Training and testing datasets
- FF++ contains 1,000 real videos and 4,000 fake videos manipulated with four face-manipulation methods.The methods are DeepFakes, Face2Face, FaceSwap, and NeuralTextures.Source: §A.2 · p. 8
- FF++ provides ground-truth masks identifying manipulated face regions, which are useful for training Face X-Ray.Source: §A.2 · p. 8
- Cross-dataset testing uses DeepfakeDetection, DeeperForensics-1.0, and Celeb-DF in addition to FF++.The datasets include Google-released videos, paid-actor recordings, and celebrity videos with reduced visual artifacts, respectively.Source: §A.2 · p. 8
A.3 Pre-processing
Videos are preprocessed by extracting frames and cropping faces using facial landmarks detected by MTCNN.
A.3 Pre-processing
- Frames are extracted and faces are cropped using facial landmarks detected by MTCNN.Source: §A.3 · p. 8
- The same preprocessing procedure is used for all models.Source: §A.3 · p. 8
- MesoInception4 and XceptionNet are also trained with the additional BI dataset for comparison with Face X-Ray.Source: §A.3 · p. 8
A.4 Metrics and generalization results
The study evaluates detector generalization using AUC, pAUC at 10% FPR, and equal error rate. Face X-Ray performs well on FF++ and generalizes to unseen manipulations, while BI training improves two other architectures.
A.4 Metrics and generalization results
- Face X-Ray performs very well on FF++ and generalizes to unseen manipulations during cross-testing.The reported comparison also notes AUC above 0.8 on the challenging Celeb-DF dataset.Source: §A.4 · p. 8, Table 3 · p. 9
- Training with the BI dataset substantially improves performance for MesoInception4 and XceptionNet.The passage attributes this expectation to BI's diverse facial forgeries and augmentation, including random downsampling and JPEG compression.Source: §A.4 · p. 8
- Generalization is measured with AUC, pAUC at 10% FPR, and equal error rate.Source: §A.4 · p. 8