Source-linked AI summary
The Deepfake Detection Challenge (DFDC) Preview Dataset
Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, Cristian Canton Ferrer
TL;DR
Deepfake detectors need robust, diverse data and metrics that reflect the rarity and cost of false positives in real-world traffic. This paper introduces an actor-consented DFDC preview dataset, defines weighted evaluation metrics, and tests existing detection models as reference baselines. The preview combines varied videos and facial manipulations for early community development and comparison.
Problem
Existing deepfake datasets and metrics are limited by generation-method coverage, data diversity, and test-set class distributions that may not represent organic traffic or false-positive costs.
Method
The paper constructs an actor-consented preview dataset with varied footage and face swaps, defines weighted precision alongside recall, and evaluates existing detection models.
Results
The paper provides a preview dataset of around 5000 original and manipulated videos and reference benchmark results from existing deepfake detection methods.
Takeaways & Limitations
The preview is intended to help researchers familiarize themselves with DFDC data, obtain early results, and compare methods with proposed baselines.
Takeaways & Limitations
Because the preview has few true negatives, weighted precision is reported with α = 100, a constant subject to change for the final dataset.
Abstract
from arXiv · showhide
In this paper, we introduce a preview of the Deepfakes Detection Challenge (DFDC) dataset consisting of 5K videos featuring two facial modification algorithms. A data collection campaign has been carried out where participating actors have entered into an agreement to the use and manipulation of their likenesses in our creation of the dataset. Diversity in several axes (gender, skin-tone, age, etc.) has been considered and actors recorded videos with arbitrary backgrounds thus bringing visual variability. Finally, a set of specific metrics to evaluate the performance have been defined and two existing models for detecting deepfakes have been tested to provide a reference performance baseline. The DFDC dataset preview can be downloaded at: deepfakedetectionchallenge.ai
1. Introduction
Deepfakes are a broad and consequential form of manipulated visual content, creating demand for robust detection data. The DFDC preview addresses this need with a diverse, actor-consented dataset and early benchmark results.
- Malicious deepfake videos can intimidate, harass, or propagate disinformation affecting elections and financial systems.
- Deepfakes encompass diverse face-swapping and facial-manipulation methods, motivating detection models trained on robust data spanning multiple generation methods.
- Existing datasets leave room for improvement in diversity, actor participation agreements, dataset size, and evaluation metrics.
- The DFDC challenge provides a freely available, actor-consented dataset of labeled face videos for developing and evaluating manipulation detectors.
- The paper introduces a preview of around 5000 original and manipulated videos and reports benchmark results from existing deepfake detection methods.
2. Dataset Construction
The preview dataset was built from crowdsourced actor footage designed to vary demographic and visual conditions, then processed into face swaps and clips for training and testing.
- Crowdsourced actors contributed videos varying in gender, skin tone, age, lighting, head pose, and background.
- Actors agreed to the use and modification of their likenesses, and no publicly available or social-media data was used.
- The preview selected 66 individuals for separate training and testing sets to avoid cross-set face swaps, using two undisclosed swap methods.
- Face swaps were generated between subjects with similar facial attributes, with each identity swapped onto the other’s videos.
- Method A swaps were restricted to videos with an average face size ratio below 0.25 because close-up faces produced easy-to-spot fakes.
- After removing the first five seconds, videos were divided into multiple 15-second clips while training clips retained their original resolution and quality.
3. Evaluation metrics
Standard deepfake metrics are sensitive to test-set class balance and may not reflect organic traffic, especially the cost of false positives. The paper therefore emphasizes weighted precision alongside recall.
- Test-set positive and negative distributions strongly influence existing deepfake metrics, limiting quantitative insight into production traffic performance.
- Organic traffic contains far fewer deepfakes than typical datasets, so evaluation metrics must account for this prevalence difference.
- Weighted precision is defined as wP = TP / (TP + αFP), while recall is R = TP / (TP + FN).
- Because the preview has few true negatives and false positives cause large weighted-precision variation, results use α = 100, a constant subject to change.
- The paper reports log(wP) at recall levels 0.1, 0.5, and 0.9 to represent different detection and labeling objectives.
4. Baseline
The paper establishes an initial baseline using three simple video-manipulation detection models and jointly tunes frame-level and video-level thresholds for evaluation.
- Three models provide the initial baseline: TamperNet, XceptionNet face detection, and XceptionNet full-image models.TamperNet is a small DNN trained on low-level image manipulations; both XceptionNet models were trained on FaceForensics data, with one frame sampled per second.
- Frame-based detection requires tuning both the per-frame detection threshold and the frames-per-video threshold.The thresholds are coupled: a lower per-frame threshold generally requires a higher frames-per-video threshold, and conversely.
- Cross-validation selects frame and video thresholds that maximize log-WP while maintaining the desired recall.The frames-per-video threshold is evaluated only on frames containing a detectable face to normalize for video length.
5. Closing remarks
The paper presents the DFDC preview as an early release intended to familiarize researchers with the data, provide initial results, and enable comparison with proposed baselines.
- The preview is intended to help researchers become familiar with the data before the later DFDC release.
- It provides early results and comparisons against the proposed baselines.