PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection

Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, Baoyuan Wu

arXiv:2307.01426v2cs.CV

TL;DR

Deepfake detection lacks standardized data processing, experimental settings, and evaluation protocols, hindering fair comparison and reproducibility. DeepfakeBench provides a unified modular benchmark integrating datasets, detectors, and evaluation tools. Its analyses show that augmentation and backbone choices materially affect results, while the current benchmark primarily operates at the frame level.

  • Problem

    Deepfake detection lacks standardized data processing, experimental settings, and evaluation strategies, making fair comparison and reproducibility difficult.

  • Method

    DeepfakeBench unifies data management, implements 15 detectors across 9 datasets, and standardizes frame-level metrics and evaluation protocols.

  • Results

    Augmentation and backbone choices substantially affect detector performance across within-domain, compressed-data, and cross-domain evaluations.

  • Takeaways & Limitations

    DeepfakeBench supports fairer, more transparent, and more reproducible comparisons while enabling systematic detector analysis.

  • Takeaways & Limitations

    The current benchmark primarily evaluates frame-level detection; video-level detectors and temporal evaluation metrics remain future extensions.

Abstract

from arXiv · show

A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results. Specifically, there is a lack of uniformity in data processing pipelines, resulting in inconsistent data inputs for detection models. Additionally, there are noticeable differences in experimental settings, and evaluation strategies and metrics lack standardization. To fill this gap, we present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions: 1) a unified data management system to ensure consistent input across all detectors, 2) an integrated framework for state-of-the-art methods implementation, and 3) standardized evaluation metrics and protocols to promote transparency and reproducibility. Featuring an extensible, modular-based codebase, DeepfakeBench contains 15 state-of-the-art detection methods, 9 deepfake datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations. Moreover, we provide new insights based on extensive analysis of these evaluations from various perspectives (e.g., data augmentations, backbones). We hope that our efforts could facilitate future research and foster innovation in this increasingly critical domain. All codes, evaluations, and analyses of our benchmark are publicly available at https://github.com/SCLBD/DeepfakeBench.

1 Introduction

Deepfake detection lacks a standardized, unified, comprehensive benchmark, making comparisons difficult and potentially misleading. DeepfakeBench addresses this gap with unified data management, integrated detector implementations, and standardized evaluation and analysis.

  • Deepfake detection methods are commonly categorized as naive, spatial, or frequency detectors, but their fair comparison remains difficult.
  • Inconsistent training configurations and evaluation standards produce divergent detection outcomes across methods.
  • Non-public source code limits reproducibility and comparability of reported detector results.
  • Detection performance can change substantially with seemingly minor settings, such as the number of selected video frames.
  • DeepfakeBench provides an extensible modular codebase with unified data processing, state-of-the-art detector implementations, and standardized evaluation and analysis.

2 Related Work

Prior work spans deepfake generation, detector architectures, surveys, and benchmarks. Existing benchmarks contribute useful resources but often remain dataset-specific and lack unified handling, training, evaluation, or analytical support.

  • Deepfake generation research prominently includes face-swapping and face-reenactment techniques based on learning-driven facial manipulation.
  • Deepfake detectors are organized into naive CNN-based, spatial representation-based, and frequency-domain approaches.
  • Frequency detectors target forgery clues such as phase-spectrum patterns, high-frequency noise, and adaptively learned frequency representations.
  • Related surveys examine deepfake technology, including its legal and ethical dimensions, manipulation techniques, and detection approaches.
  • Existing benchmarks often focus on their own datasets, leaving inconsistencies in cross-dataset data handling, training strategies, augmentations, and analysis tools.

3 Our Benchmark

DeepfakeBench combines nine recognized datasets and fifteen established detectors in an extensible benchmark with configurable data processing, training, and evaluation. Its modular codebase standardizes inputs and supports reproducible performance analysis.

  • Datasets and Detectors: The benchmark incorporates nine widely recognized deepfake datasets, including FaceForensics++, Celeb-DF, DFDC, UADFV, FaceShifter, and DeeperForensics-1.0.
  • Datasets and Detectors: Users can select training and testing dataset combinations, although FaceForensics++ is typically used for training and other datasets for testing.
  • Datasets and Detectors: DeepfakeBench reduces preparation overhead by addressing frame extraction, face cropping, face alignment, and organization across diverse dataset structures.
  • Datasets and Detectors: The benchmark implements fifteen established detectors selected for classic or advanced status and organized into naive, spatial, and frequency categories.
  • Codebase: Its modular codebase contains Data Processing, Training, and Evaluation and Analysis modules, with configurable preprocessing and automated experiment handling.
  • Codebase: Evaluation uses ACC, AUC, AP, and EER at the frame level, alongside ROC-AUC curves, radar charts, histograms, Grad-CAM, and t-SNE visualizations.

4 Evaluations and Analysis

DeepfakeBench evaluates detectors under standardized within-domain, cross-domain, and cross-manipulation settings, then analyzes augmentation, backbone, pre-training, and representation effects. Results show that generalization depends strongly on evaluation conditions and design choices.

  • Evaluation protocols: All detectors are trained on FF-c23 and evaluated using within-domain and cross-domain AUC, with average scores and top-three counts reported.The benchmark evaluates six within-domain and eight cross-domain testing datasets.
  • Evaluation protocols: Cross-manipulation evaluation changes only the forgery algorithm while keeping background and identity consistent, exposing detector generalization across manipulations.Figure 2 compares 10 detectors using uniformly scaled AUC heat maps.
  • Data augmentation: Removing all augmentations improves Xception and SPSL performance by approximately 2% on FF++_c23, whereas compression-related augmentations help on compressed FF++_c40 data.Compression and blur can degrade SPSL on DFD and DFDCP, while blur can reduce Xception performance by diminishing visual artifacts.
  • Backbone architecture: Xception and EfficientNet-B4 consistently outperform ResNet34 despite similar parameter counts, showing that backbone architecture materially affects detector performance.The comparison covers CORE, SPSL, UCF, and Face X-ray using AUC across within-domain and cross-domain datasets.
  • Backbone architecture: Adding depthwise separable convolution to ResNet produces significant improvements on many test datasets, while model depth and parameter count also influence performance.The ablation trains on FF++_c23 and evaluates frame-level AUC on other datasets.
  • Pre-training: Pre-trained models largely outperform non-pre-trained models, with gains of about 10% for Xception on DFDCP and EfficientB4 on DeepFakeDetection.The benefit is less pronounced for ResNet34, which the authors associate with its architectural design.
  • Representation analysis: t-SNE shows distinct learned feature representations across detectors, but Meso4 fails to clearly separate real and fake samples.The visualizations use detectors trained and tested on FF++ (c23).

5 Conclusions, Future Plans, and Societal Impacts

DeepfakeBench provides a modular platform for standardized, fair detector comparison and analysis, while identifying current scope and ethical boundaries. The benchmark currently focuses on frame-level detection and may be misused to refine deepfakes that evade detection.

  • DeepfakeBench offers a tailored processing pipeline, broad detector coverage, and standardized comparisons across models.
  • Researchers can compare new methods with existing ones more quickly, supporting faster development and iteration.
  • Comprehensive evaluations and analyses may inspire new research problems and future advances in deepfake detection.
  • DeepfakeBench currently provides algorithms and evaluations primarily at the frame level, with video-level detectors and metrics planned for future versions.
  • Transparent benchmarking may give malicious actors a blueprint for refining deepfakes to evade detection, motivating consideration of controlled access and continual updates.

6 Contents in Appendix

The appendix supplies additional details on data processing, algorithm implementations, and visualizations to deepen understanding of the benchmark.

  • Details of data processing: The appendix elaborates on face detection, cropping, alignment, and other data-processing steps.
  • Details of algorithms implementation and visualizations: It also describes algorithm implementations and provides additional visualizations for deeper analysis.

A.1 Details of Data Processing

The data-processing pipeline standardizes video frames into aligned face data, optional masks, landmarks, and metadata arranged for uniform dataset handling. It combines configurable sampling, detection, alignment, cropping, and structured storage.

  • Face Detection and Alignment: Dlib detects faces and facial landmarks, then uses an affine transformation to align and crop faces into a standardized pose.
  • Face Cropping and Landmark Extraction: Cropping includes a configurable margin around aligned faces, while extracted landmarks provide facial geometry for downstream analysis and training.
  • Mask Extraction: When available, masks indicating manipulated regions are aligned and cropped alongside the corresponding face data.
  • Frame Sampling: The pipeline extracts a subset of frames from each video, using fixed counts or fixed strides to balance computational demands and example diversity.
  • Saving Processed Data: Processed faces, landmarks, and optional masks are saved in structured per-video directories, with JSON metadata describing the rearranged dataset.
  • Arrangement: A unified JSON-based arrangement standardizes heterogeneous dataset structures, eliminating dataset-specific I/O code and reducing preprocessing duplication.

A.2 Details of Algorithms Implementation and Visualizations

DeepfakeBench implements 15 detectors spanning naive, spatial, and frequency approaches, including reimplementations where official code is unavailable. It also visualizes detector behavior and analyzes representations produced by FWA-generated data.

  • Algorithms Implementation: The benchmark organizes 15 implemented detectors into naive, spatial, and frequency categories.
  • Naive Detectors: Meso4 uses a CNN to target mesoscopic image properties, while MesoIncep uses more sophisticated Inception modules.
  • Naive Detectors: CNN-Aug uses ResNet-34 with JPEG compression and Gaussian blurring augmentations for GAN-generated image detection.
  • Naive Detectors: The benchmark includes Xception variants trained under raw, medium-compression, and high-compression video conditions, plus Capsule using VGG19-based capsule structures.
  • Spatial and Frequency Detectors: Spatial and frequency methods model artifacts through attention, representation consistency, reconstruction, disentanglement, frequency decomposition, phase spectra, or high-frequency noise.
  • Visualizations: Some detectors lacking public code were reimplemented from their original papers, including DSP-FWA and Face X-ray.
  • Visualizations: FWA-generated blended images appear distinct from other manipulation methods in t-SNE feature space, enlarging the forgery space and improving generalization capabilities.

A.3 Training Details and Full Experimental Results

DeepfakeBench standardizes dataset handling, training configurations, augmentations, and evaluation setup across its 9 included datasets. It supports reproducible comparisons through modular detector implementations and consistent experimental settings.

  • Dataset Splits: Dataset splits are standardized through dataset-specific rules, including duplicated real and fake data for DFD, UADFV, and parts of FF++ and FaceShifter.Celeb-DF, DFDCP, and DFDC use author-provided or method-based test splits, with validation sometimes matching test data.
  • Training Setup: Adam is used with a 0.0002 learning rate and batch size 32 for most experiments, while DSP-FWA and Face X-ray use batch size 16 for paired inputs.For DSP-FWA and Face X-ray, dynamically generated forgery images double the input size.
  • Training Setup: Naive detectors generally use ImageNet-pretrained official backbones, except Meso4 and MesoIncep, which lack ImageNet pretraining weights.The benchmark separately evaluates the effect of pretraining.
  • Evaluation Setup: Evaluation reports average top-3 metrics as the primary result, alongside top-1 results, and computes AUC, AP, and EER without using the validation set.Experiments are conducted in a standardized NVIDIA A100 environment.
  • Data Augmentation: The benchmark applies common augmentations including flips, rotations, isotropic resizing, color adjustments, FancyPCA, and image compression.Several augmentations are applied with probability 0.5, while HSV adjustment has no specified probability in the described code.

Full Experimental Results

The benchmark presents comprehensive detector evaluations across within-dataset and cross-dataset settings, using multiple metrics and training-time visualizations. These views expose performance trends and stability differences among detectors.

  • Evaluation Scope: 15 detectors are compared across 14 testing datasets using AUC, AP, and EER, with results selected from the highest-performing models on each testing dataset.The benchmark also stores predictions and labels to support additional metric computation.
  • Training Monitoring: Training-process visualizations track training loss, training accuracy, testing loss, AUC, AP, and EER through TensorBoard.Plotting metrics against training steps enables comparison across datasets and evaluation measures.
  • Stability Analysis: Some detectors show volatile testing metrics, so final Top-3 results average testing metrics rather than relying on a single potentially unstable observation.DSP-FWA and Face X-ray are visualized separately because their batch sizes differ from the other 13 detectors.

A.4 Other Analysis Results

Additional analyses examine frequency artifacts, detector design choices, and evaluation consistency. They report limited visible frequency artifacts in deepfake forgeries, differences between GAN and diffusion artifacts, and gains from adding spectrum features.

  • Frequency Analysis: Average spectra of 2,000 randomly sampled real and fake images are used to examine deepfake-forgery artifacts in the frequency domain.The visualization includes separate real, fake, and difference spectra.
  • Frequency Analysis: Deepfake forgeries do not exhibit obvious frequency artifacts, consistent with prior findings, potentially because resizing, blending, and MPEG compression perturb low-level statistics.These preprocessing and post-processing steps can prevent frequency patterns from emerging distinctly in the visualization.
  • GAN and Diffusion Artifacts: Diffusion-generated images exhibit fewer artifacts, whereas GAN-generated images display a noticeable checkerboard artifact pattern.The comparison uses the GenImage dataset and the benchmark’s frequency analysis tool.
  • Phase Spectrum: SPSL reaches an average cross-domain score of 78.75%, motivating spectrum-feature integration because phase spectra can capture up-sampling artifacts.The benchmark also evaluates iFWA in cross-manipulation settings.
  • Phase Spectrum: 80.35% average AUC is achieved by iFWA versus 73.16% for FWA in cross-data evaluation after adding the spectrum feature.The models are trained on FF++_c23 and tested on other datasets using frame-level AUC.
  • Detector Analysis: Naive detectors such as Xception and EfficientNetB4 can be competitive with more complex methods, while pretraining and augmentation further improve naive-detector performance.The analysis also warns that inconsistent procedures and metrics can make results and claims differ across studies.
Loading 2307.01426v2…