Source-linked AI summary
Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics
Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, Siwei Lyu
TL;DR
DeepFake detectors need large-scale data that better represents high-quality face-swapping videos because existing datasets contain conspicuous artifacts unlike those circulated online. The paper constructs Celeb-DF with an improved synthesis process and evaluates detection methods across datasets, finding that Celeb-DF is substantially more challenging while identifying visual-quality measurement and future anti-forensic work as boundaries.
Problem
Existing DeepFake datasets contain visual artifacts and do not closely resemble high-quality videos circulated online, limiting their relevance for developing and evaluating detectors.
Method
The paper constructs Celeb-DF from celebrity videos using an improved DeepFake synthesis method and evaluates current detectors across Celeb-DF and existing datasets.
Results
Celeb-DF is challenging to most existing detection methods, while second-generation datasets have average AUC scores below 70% compared with around 80% for first-generation datasets.
Takeaways & Limitations
Celeb-DF reduces the visual-quality gap between benchmark data and DeepFakes circulated online and provides a challenging basis for detector evaluation.
Takeaways & Limitations
The paper notes that no reference-free face-image quality metric is yet widely agreed upon and adopted, so it uses a referenced Mask-SSIM measure with interpretive limitations.
Abstract
from arXiv · showhide
AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5,639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.
1. Introduction
DeepFakes use DNNs to synthesize one person’s face onto another while retaining the target’s expressions and poses, creating serious risks and motivating detection research. Existing datasets contain conspicuous artifacts and may not reflect online videos, so Celeb-DF introduces a larger, higher-quality benchmark and finds current detectors challenged by it.
- DeepFakes replace a target’s face with a donor’s synthesized face while retaining facial expressions and head poses, potentially creating false impressions of real activities.The paper links these identity-based illusions to political, social, financial, and legal consequences.
- Growing concerns about DeepFakes have driven development of detection methods and increased demand for large-scale datasets.The paper identifies dataset availability as an enabling factor for detector development.
- Existing datasets commonly show low-quality synthesized faces, splicing boundaries, color mismatches, visible original-face regions, and inconsistent face orientations.The paper attributes these artifacts to imperfect synthesis steps and insufficient curation.
- Because existing-dataset artifacts differ from DeepFakes circulated online, high detection performance on those datasets may not remain relevant after deployment in the wild.This motivates evaluating detectors on more realistic data.
- Celeb-DF contributes 5,639 DeepFake videos spanning more than 2 million frames, generated from videos of 59 diverse celebrities using an improved synthesis method.The dataset is intended for developing and evaluating DeepFake detection algorithms, with fewer notable visual artifacts than existing datasets.
- Evaluation results show that Celeb-DF is challenging for most existing detection methods despite high or near-perfect accuracy on previous datasets.The paper presents this as a comprehensive evaluation across Celeb-DF and other existing datasets.
2. Backgrounds
DeepFake videos are generated through an automated auto-encoder pipeline, while detection methods use several types of visual, physiological, and data-driven cues. Existing datasets support detector development but differ in scale, synthesis algorithms, and generation quality.
- DeepFake Video Generation: The basic maker is widely used because it is simpler and available through multiple open-source implementations.The paper identifies FakeApp, DFaker, faceswap-GAN, faceswap, and DeepFaceLab as examples.
- DeepFake Video Generation: The basic DeepFake maker detects and aligns target faces, encodes them, decodes donor identities, then warps and blends synthesized faces into the original frames.A shared encoder captures identity-independent attributes, while each subject has a dedicated decoder.
- DeepFake Detection Methods: Detection methods target splicing artifacts, physical or physiological inconsistencies, signal-level artifacts, and data-driven visual patterns.Examples include abnormal eye blinking, incoherent head poses, behavioral landmark patterns, and synthesis-related signal artifacts.
- Existing DeepFake Datasets: Existing datasets range from UADFV’s 49 real and 49 fake videos to DFD’s 3,068 DeepFake videos, using different synthesis tools and source populations.DF-TIMIT, FF-DF, DFD, and DFDC provide additional datasets with distinct sizes and synthesis configurations.
- Existing DeepFake Datasets: The released DFDC information was incomplete at the time of the paper’s CVPR submission, and its synthesis details were not fully available.The paper notes that the full DFDC set had not yet been released and that synthesis information was based on the first-round release.
- Existing DeepFake Datasets: DeepFake detection requires large-scale datasets for both training and evaluation, motivating comparisons among available collections.The paper groups UADFV, DF-TIMIT, and FF-DF as first-generation datasets, and DFD, DFDC, and Celeb-DF as second-generation datasets.
3. The Celeb-DF Dataset
Celeb-DF was constructed to provide more relevant data for developing and evaluating DeepFake detection methods by reducing visual artifacts found in existing datasets. Its improved synthesis process produces higher-quality videos, whose visual quality is assessed with Mask-SSIM despite limitations in absolute interpretation.
- Dataset Motivation: The dataset targets artifacts in existing datasets, including low-quality faces, visible splicing boundaries, color mismatch, exposed original-face regions, and inconsistent face orientations.These artifacts are attributed to imperfect synthesis steps and insufficient curation.
- Dataset Composition: Celeb-DF contains 590 real videos and 5,639 DeepFake videos, corresponding to over two million frames from interviews of 59 diverse celebrities.Videos average approximately 13 seconds at 30 frames per second.
- Synthesis Method: The 256 × 256 synthesized faces are smoother and less affected by resizing and rotation when accommodating target faces.The higher resolution is achieved with encoder and decoder models having more layers and increased dimensions.
- Synthesis Method: Color augmentation, post-processing, and color transfer reduce mismatch between synthesized donor faces and the original target faces.Figure 5 contrasts synthesis without and with color correction.
- Visual Quality: Celeb-DF has the highest average Mask-SSIM among compared datasets, although absolute Mask-SSIM values may not meaningfully measure synthesized-face quality because identity changes contribute to the score.Differences in Mask-SSIM are treated as reflecting relative visual-quality differences, and DFD and DFDC scores use only videos with exact correspondences.
4. Evaluating DeepFake Detection Methods
The evaluation compares nine publicly available or author-provided DeepFake detection methods across datasets using frame-level AUC. Celeb-DF is the most challenging dataset overall, while DSP-FWA has the highest average performance across datasets and compression affects methods differently.
- 4.1. Compared DeepFake Detection Methods: Nine DeepFake detection methods are evaluated using publicly available code and corresponding pretrained models or models obtained from the authors.The study uses inference code, published pretrained models, and default parameters rather than retraining methods across all datasets.
- 4.2. Evaluation Metrics: Frame-level AUC is used because all methods classify individual frames, avoiding differences from video-level score aggregation and cross-dataset output calibration.Classification scores are rounded to five decimal places to improve robustness to numerical imprecision.
- 4.3. Results and Analysis: Celeb-DF is generally the most challenging dataset, with the lowest overall detection performance among the evaluated datasets.The authors relate this difficulty to improved synthesis quality that reduces artifacts such as low resolution and color mismatch.
- 4.3. Results and Analysis: Average AUC scores are below 70% for second-generation datasets DFD, DFDC, and Celeb-DF, versus around 80% for first-generation datasets UADFV, DF-TIMIT, and FF-DF.Some methods achieve near-perfect detection on the first-generation datasets.
- 4.3. Results and Analysis: DSP-FWA achieves the highest overall average AUC across all DeepFake datasets at 87.4%.The result indicates improved overall detection performance for the most recent method evaluated.
- 4.3. Results and Analysis: Performance decreases as H.264 compression increases; FWA and DSP-FWA degrade significantly, whereas Xception-c23 and Xception-c40 are not significantly affected.The Xception methods were trained on compressed H.264 videos, which the authors identify as the reason for their greater robustness in this setting.
5. Conclusion
The paper introduces Celeb-DF as a challenging large-scale dataset designed to narrow the visual-quality gap between synthetic and online DeepFake videos. It concludes that current detectors remain challenged and identifies dataset expansion, synthesis improvements, and anti-forensic techniques as future directions.
- 5. Conclusion: Celeb-DF is presented as a challenging large-scale dataset for developing and evaluating DeepFake detection methods.The dataset is intended to reduce the visual-quality gap between DeepFake datasets and videos circulated online.
- 5. Conclusion: Future work prioritizes enlarging Celeb-DF and improving the visual quality, efficiency, and model structure of its synthesis algorithm.The authors also aim to incorporate anti-forensic techniques to anticipate methods that hide traces used by detectors.