Source-linked AI summary
The DeepFake Detection Challenge (DFDC) Dataset
Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, Cristian Canton Ferrer
TL;DR
Convincing face-swapped videos can now be created cheaply, but scalable detection requires large, representative, ethically sourced training data. The paper constructs and releases the DFDC Dataset and runs a public benchmark competition, finding that DFDC-trained models generalize to real in-the-wild Deepfakes despite detection remaining difficult.
Problem
Cheap, convincing face-swapped videos create a need for scalable detection, but producing the large training corpora required by detection models is often cost prohibitive.
Method
The paper constructs the DFDC Dataset from consenting paid actors using multiple face-swapping methods and evaluates community-submitted detectors through a benchmark competition.
Results
Models trained on DFDC achieved average precision 0.753 and ROC-AUC 0.734 on real videos, demonstrating generalization to real videos.
Takeaways & Limitations
The released dataset and benchmark provide a basis for training and assessing Deepfake detectors across varied manipulations and unseen real videos.
Takeaways & Limitations
Only 960 of roughly 3,500 original identities were included because of time and computational constraints.
Abstract
from arXiv · showhide
Deepfakes are a recent off-the-shelf manipulation technique that allows anyone to swap two identities in a single video. In addition to Deepfakes, a variety of GAN-based face swapping methods have also been published with accompanying code. To counter this emerging threat, we have constructed an extremely large face swap video dataset to enable the training of detection models, and organized the accompanying DeepFake Detection Challenge (DFDC) Kaggle competition. Importantly, all recorded subjects agreed to participate in and have their likenesses modified during the construction of the face-swapped dataset. The DFDC dataset is by far the largest currently and publicly available face swap video dataset, with over 100,000 total clips sourced from 3,426 paid actors, produced with several Deepfake, GAN-based, and non-learned methods. In addition to describing the methods used to construct the dataset, we provide a detailed analysis of the top submissions from the Kaggle contest. We show although Deepfake detection is extremely difficult and still an unsolved problem, a Deepfake detection model trained only on the DFDC can generalize to real "in-the-wild" Deepfake videos, and such a model can be a valuable analysis tool when analyzing potentially Deepfaked videos. Training, validation and testing corpuses can be downloaded from https://ai.facebook.com/datasets/dfdc.
1. Introduction
Deepfakes made convincing face-swapped videos accessible with consumer hardware, creating a need for scalable automated detection. The authors address training-data constraints by releasing a large, diverse dataset and organizing a benchmark competition.
- Consumer-grade GPUs and off-the-shelf software can produce convincing face-swapped videos without specialized expertise.
- Scalable detection is needed because expert forensic review cannot handle hundreds of thousands of videos uploaded daily.
- The DFDC Dataset uses footage from 3,426 paid actors who agreed to computer-based face manipulation, addressing consent and scale concerns.
- The dataset contains varied-quality clips and multiple face-swapping methods, while a benchmark competition evaluates submitted detectors on unseen real Deepfakes.
- DFDC is over an order of magnitude larger than other available datasets in both frames and videos.
2. Previous work
Earlier Deepfake datasets were limited by small scale, few identities, uncertain consent, and restricted source conditions. Third-generation datasets such as DFDC sought to combine much larger scale, participant agreement, and more realistic variation.
- Pairwise autoencoder generation made earlier datasets expensive to scale because each identity pair required roughly one day of single-GPU retraining.
- First- and second-generation datasets generally contained few swapped identities, limiting detection generalization.
- The authors classify DFDC and DF-1.0 as third-generation datasets combining increased scale, better quality, and agreement from depicted individuals.
- Third-generation datasets contain tens of thousands of videos and tens of millions of frames, with paid actors continuing as a dataset-construction practice.
- DFDC counts 100,000 unique target/source swaps, whereas DF-1.0 contains 1,000 unique fake videos when perturbations are ignored.
- DF-1.0 uses internet-sourced target videos whose free-use status is unclear, while DFDC recorded consenting participants specifically for dataset construction.
- DF-1.0 uses controlled studio footage, whereas DFDC includes indoor and outdoor videos across real-world lighting conditions.
3. DFDC Dataset
The DFDC Dataset was constructed from consenting subjects and diverse face-swapping, audio-swapping, blending, post-processing, and augmentation methods to support large-scale detection research.
- Dataset construction: The dataset used commissioned videos of consenting individuals filmed in varied natural settings, rather than publicly available footage.Subjects agreed to filming, inclusion in a machine-learning dataset, and face manipulation.
- Dataset scale: 3,426 subjects contributed 48,190 videos averaging 68.8 seconds, totaling 38.4 days of footage and over 25 TB of raw data.Most videos were shot in 1080p, and each subject contributed an average of 14.4 videos.
- Manipulation methods: The dataset included multiple face-swapping families, with most swapped videos produced by Deepfake Autoencoders alongside morphable-mask, GAN-based, and StyleGAN methods.The included GAN-based methods were Neural Talking Heads, FSGAN, and StyleGAN.
- Post-processing: Audio swapping, sharpening, and edge-focused re-blending expanded manipulation diversity and improved or preserved full-frame video composition.Audio swaps were independent of face manipulation, while sharpening increased perceptual quality with nearly no additional cost.
- Training and splits: Training pair selection was constrained by computational cost, so subjects were paired within sets based partly on similar appearance.Over 800 GPUs were used to train 6,683 pairwise models instead of training all potential pairings.
4. Metrics
The paper argues that extreme class imbalance makes precision-oriented detection metrics more informative than accuracy for realistic video traffic, while log loss was used for competition ranking.
- Metric motivation: When fake videos are rarer than one in a million, false positives can outnumber true positives even for highly accurate models.This imbalance reduces the practical utility of accuracy as a detection metric.
- Metric motivation: Precision is more indicative than Fβ score or false positive rate when evaluating detectors against realistic video distributions.The paper notes that constructing a dataset matching organic-traffic statistics is impractical.
- Weighted precision: Weighted precision wP = TP / (TP + αFP) uses α = x/y to approximate precision under the larger negative-to-positive ratio of organic traffic.TP, FP, and FN denote true positives, false positives, and false negatives; recall remains R = TP / (TP + FN).
- Augmentation analysis: Videos containing both augmenters and distractors were excluded from the Figure 4 analysis of average log loss.Noise and face augmentations were among the most difficult, while flips, blurring, and rotation were among the easiest.
- Evaluation protocol: Competition submissions were ranked by log loss, whereas weighted precision and recall were additionally reported to assess detector performance on realistic video distributions.Log loss was chosen because weighted precision could be extremely small and noisy, and ranking required only relative performance.
5. Results
The dataset spans a wide range of face-swap quality across several methods. DFAE methods were generally the most flexible and effective, while GAN-like methods showed stronger setting-dependent weaknesses.
- A wide quality range across methods is more useful than a small, hand-tuned collection of only high-quality examples.The qualitative results are intended to represent varied outputs rather than cherry-picked failures or successes.
- MM/NN produced convincing single frames but often showed facial discontinuities and occasional mask-fitting failures.
- DFAE methods generally produced the best and most flexible results across lighting conditions and identities, with good temporal coherence.They still showed weaknesses with glasses and extreme poses.
- FSGAN worked well in good lighting and handled extreme head poses, but struggled with darker scenes and produced flat-looking results.
- NTH had the most consistent quality among GAN-like methods but inserted similar-looking eyes across subjects and performed poorly in darker settings.
- StyleGAN produced the worst overall results, commonly showing unconstrained eye gaze and illumination mismatches.
6. Large scale benchmarking
The DFDC competition evaluated Deepfake detectors at large scale using public and private test sets, including real videos. Although performance dropped on real videos, the strongest models generalized beyond the DFDC distribution.
- The private test set was used to assess how leading detectors would perform on a more realistic video distribution.
- 2,114 teams participated in the public Deepfake detection competition, whose private test set included additional subjects and real videos.Teams could use external data if it complied with competition policies.
- 60% of submissions achieved private-test log loss ≤ 0.69, approximately matching constant 0.5 probability predictions, and many submissions were effectively random.
- Good public-test performance correlated with good private-test performance.
- On real videos, the best models reached average precision 0.753 and ROC-AUC 0.734 despite an expected performance drop.These results indicate that DFDC-trained models can generalize to real videos.
- Detection performance on DFDC videos correlated with performance on real videos, supporting DFDC as a resource for training detectors for real Deepfake videos.
- The top submissions combined face detection, frame-level feature extraction, augmentation, and ensemble-based modeling approaches.
7. Future work
Future work focuses on assessing video quality, expanding identity coverage, and potentially releasing the original raw dataset with additional annotations.
- Perceptual quality assessment of the dataset remains ongoing after delays caused by time constraints and COVID-19 circumstances.
- Only 960 of roughly 3,500 original identities are currently included because of time and computational constraints.
- The authors plan to explore releasing the original raw dataset to the research community.