PaperlayerView exact revision on arXiv

Source-linked AI summary

DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection

Liming Jiang, Ren Li, Wayne Wu, Chen Qian, Chen Change Loy

arXiv:2001.03024v2cs.CVcs.LG

TL;DR

Face forgery detection needs datasets that capture realistic variation and support reliable evaluation of manipulated videos. DeeperForensics-1.0 addresses this gap with a large, diverse benchmark, a face-swapping framework, and baseline analysis; its results show that diverse training variants and video-level methods improve performance, although hidden-test accuracy remains limited.

  • Problem

    Face forgery detection requires effective countermeasures, but existing data and generated videos lack sufficient real-world variation and can remain visibly artificial.

  • Method

    The paper constructs DeeperForensics-1.0 with 60,000 videos and 17.6 million frames, using carefully collected source videos, DF-VAE face swapping, real-world distortions, and representative detector benchmarks.

  • Results

    Training-set variants and diverse distortions improve detection accuracy, while video-level methods show greater potential than image-level methods for real-world fake videos.

  • Takeaways & Limitations

    DeeperForensics-1.0 provides a higher-quality benchmark for real-world face forgery detection and supports research on training diversity and video-level analysis.

  • Takeaways & Limitations

    Accuracy on the challenging hidden test set remains not very high, leaving source-data quality and training-set diversity as future improvement directions.

Abstract

from arXiv · show

We present our on-going effort of constructing a large-scale benchmark for face forgery detection. The first version of this benchmark, DeeperForensics-1.0, represents the largest face forgery detection dataset by far, with 60,000 videos constituted by a total of 17.6 million frames, 10 times larger than existing datasets of the same kind. Extensive real-world perturbations are applied to obtain a more challenging benchmark of larger scale and higher diversity. All source videos in DeeperForensics-1.0 are carefully collected, and fake videos are generated by a newly proposed end-to-end face swapping framework. The quality of generated videos outperforms those in existing datasets, validated by user studies. The benchmark features a hidden test set, which contains manipulated videos achieving high deceptive scores in human evaluations. We further contribute a comprehensive study that evaluates five representative detection baselines and make a thorough analysis of different settings.

1. Introduction

Existing forgery datasets are limited in scale, quality, diversity, or train–test realism, motivating DeeperForensics-1.0 as a larger, more realistic benchmark with systematic perturbations and baseline evaluations.

  • Existing datasets contain too few, low-quality, or overly artificial videos, limiting training for real-world forgery detection.
  • Real-world face forgery detection remains difficult because realistic editing produces videos that vary widely and unpredictably.
  • DeeperForensics-1.0 contains 60,000 videos and 17.6 million frames, designed around quality, scale, and diversity.
  • The dataset uses consented data from 100 individuals, a novel DF-VAE enhancement method, and deliberate distortions and perturbations.
  • The benchmark evaluates five representative detection methods on the dataset and a hidden test set containing highly deceptive manipulated videos.

2. Related Work

Prior face-forgery datasets vary substantially in size and construction, while DeeperForensics-1.0 emphasizes consented high-resolution data, controlled collection, perturbation diversity, and difficult evaluation.

  • Earlier datasets range from 98 videos in UADFV to 4,000 fake and 1,000 real videos in FaceForensics++.
  • DeeperForensics-1.0 uses 100 consented paid actors and high-resolution 1920 × 1080 source videos with varied poses, expressions, and illumination.
  • DF-VAE is an end-to-end face-swapping method intended to improve the fidelity of generated videos.
  • The dataset introduces seven perturbation types at five intensity levels, producing 35 perturbations and allowing mixtures of perturbations.
  • The benchmark uses a hidden test set with high user-study deceptive scores and emphasizes video-level methods that incorporate temporal information.

3. A New Large-Scale Face Forensics Dataset

DeeperForensics-1.0 combines large-scale, diverse source and manipulated videos with a learned face-swapping framework designed for quality, style matching, scalability, and temporal continuity.

  • Dataset construction: 60,000 videos and 17.6 million frames form the dataset, including over 50,000 collected source videos and 10,000 manipulated videos.The manipulated portion contains 5 million frames, while the collected source data contains 12.6 million frames.
  • Dataset construction: Source-data collection improves face-swapping robustness to extreme poses because internet videos usually contain limited head-pose variation.The dataset therefore emphasizes richer source-face expressions, poses, and lighting conditions than target-face variation.
  • Dataset construction: 100 consented actors provide high-resolution source data spanning identities, poses, expressions, illuminations, and 3DMM blendshapes.The collection uses controlled indoor recording with seven HD cameras and 1920 × 1080 resolution.
  • DF-VAE: MAdaIN reduces style mismatch and boundary artifacts while the framework supports many-to-many face swapping without quality degradation across multiple identities.Temporal continuity is also treated as a design requirement for generated videos.
  • DF-VAE: DF-VAE is a conditional variational auto-encoder with structure extraction, disentangled structure and appearance representations, and fusion modules.Its end-to-end design uses unpaired source samples, separated encoders, and an explicit structure heatmap.
  • Dataset construction: Seven perturbation types applied at five intensity levels increase variation in manipulated videos to imitate real-world scenarios.The dataset is released for research on real-world face forgery detection.

4. Video Forgery Detection Benchmark

The benchmark uses a hidden test set designed to better approximate real-world face forgery, then evaluates detection under varied distortions and training-set variants. Results show that perturbation-aware training and video-level methods improve robustness beyond clean, distribution-matched evaluation.

  • Hidden test set: The hidden test set is designed to avoid close training-test distributions and simulate multiple sources, high-quality fakes, and diverse real-scene distortions.These factors target variability that standard benchmarks may not capture.
  • Hidden test set: The hidden test set contains 400 videos from multiple unknown face-swapping methods, diverse hidden distortions, and fakes fooling at least 50 of 100 human observers.Its labels remain hidden for evaluating detection models.
  • Cross-dataset evaluation: The baselines trained on DeeperForensics-1.0 achieve much better hidden-test performance than models trained on all four FaceForensics++ subsets.I3D obtains the best hidden-test performance when trained on the standard DeeperForensics-1.0 training set.
  • Perturbation evaluation: Nearly 100% accuracy occurs when models are trained and tested on the clean standard set, but accuracy decreases when test distortions alter the distribution.Most video-level methods except C3D are more robust to test perturbations than XceptionNet.
  • Perturbation evaluation: Matching distortions in training and testing increases accuracy, but randomized evaluation indicates that suitable training distortions can improve robustness to real-world perturbations.The matched setting is described as impractical because training and test distributions remain the same.
  • Training-set variants: Training on std+std/sing or std+std/rand increases hidden-test accuracy, while std+std/mix further increases accuracy across all detection baselines.I3D and ResNet+LSTM exceed 80% in some settings.
  • Implications: The study concludes that suitable training variants and diverse distortions are necessary, while video-level methods have greater potential than image-level methods for real-world fake videos.The hidden-test accuracy remains not very high, motivating improved data quality and diversity.

5. Discussion

The discussion presents DeeperForensics-1.0 as a large-scale benchmark for real-world face forgery detection and outlines expansion plans. Future work includes enlarging the data and hidden test set and developing better evaluation metrics.

  • Contributions: DeeperForensics-1.0 is proposed to facilitate face forgery detection research toward real-world scenarios.The dataset is intended to provide good quality, large scale, and high diversity.
  • Future work: The authors plan to collect more source and target videos and invite additional falsification methods that pass human testing for the hidden test set.These plans aim to expand DeeperForensics and its future hidden-test versions.
  • Future work: A better evaluation metric for face forgery detection methods is identified as an additional research direction.This is listed alongside dataset and hidden-test expansion.

A. Derivation

DF-VAE is trained with reconstruction, KL, content, style, and temporal objectives, combining pixel/SSIM fidelity with feature matching and temporal consistency.

  • Reconstruction loss: DF-VAE uses separate source and target reconstruction losses, with pixel and SSIM terms measuring reconstructed-face fidelity.The reconstruction objective includes MAE-based pixel loss and SSIM loss, with hyperparameters controlling the two reconstruction components.
  • Reconstruction loss: Pixel loss computes Mean Absolute Error after reconstruction.
  • Reconstruction loss: SSIM loss computes Structural Similarity between the reconstructed and original faces.
  • KL loss: The KL term uses the typical variational-autoencoder divergence, with latent dimensionality and variational mean and standard deviation defining its components.DF-VAE uses reparameterization to keep sampling differentiable through an auxiliary variable with independent marginal.
  • MAdaIN loss: MAdaIN loss trains the style-matching and fusion module jointly with the disentangled module using VGG-19 features.Style loss matches the mean and standard deviation of style features, while content loss measures Euclidean distance between target and swapped-face features.
  • Total objective: The total objective jointly trains all DF-VAE components with weighted loss terms, including a temporal consistency loss.The framework is described as an end-to-end many-to-many face swapping model.

C. Implementation Details

DF-VAE is implemented end-to-end with pretrained landmark extraction, optical-flow estimation, AdaIN-based style matching and fusion, and Adam optimization.

  • Implementation Details: DF-VAE uses pretrained stacked hourglass networks for landmarks, FlowNet 2.0 for optical flow, and AdaIN for style matching and fusion.The stacked hourglass configuration uses 4 stacks and 1 block.
  • Implementation Details: The learning rate is 0.00005 for all DF-VAE components, optimized with Adam using β1 = 0.5 and β2 = 0.999.Experiments are conducted on NVIDIA Tesla V100 GPUs.

D. User Study of Methods

The user study compares DF-VAE with three representative learning-based face-swapping baselines using matched source and target video samples.

  • Baselines: The baselines are DeepFakes, faceswap-GAN, and ReenactGAN, representing autoencoder, GAN, and boundary-latent-space architectures.DF-VAE uses the same fusion method as ReenactGAN for fair comparison.
  • Study setup: The study randomly selects 30 real DeeperForensics-1.0 videos as sources and 30 real FaceForensics++ videos as targets.Each method generates 30 fake videos.
  • Baselines: DeepFakes uses a shared encoder with two separate decoders to perform face swapping.
  • Baselines: faceswap-GAN extends a similar structure with paired discriminators to improve face-swapping quality.
  • Baselines: ReenactGAN performs many-to-one face reenactment by adapting the source-face boundary to the target-face boundary.A separate fusion method is applied to obtain swapped faces.

E. Quantitative Evaluation Metrics

The evaluation uses FID and IS to assess generated-image distribution similarity, visual quality, articulation, and diversity; DF-VAE outperforms all three baselines on both metrics.

  • FID: FID measures distribution similarity between generated and real images, correlates with visual quality, and assigns better quality to lower values.
  • IS: IS evaluates articulation and diversity, with higher values indicating better generation quality.
  • Results: DF-VAE outperforms DeepFakes, faceswap-GAN, and ReenactGAN in quantitative FID and IS evaluations.Table 7 reports the scores for all four methods.

F. Ablation Study

The ablations test temporal consistency and individual DF-VAE components, showing that temporal loss reduces re-rendering error and the full method produces high-fidelity swaps.

  • Temporal loss: Higher re-rendering error without temporal loss demonstrates the effectiveness of the temporal consistency constraint.Errors are measured by per-pixel Euclidean distance in RGB channels in a self-reenactment setting with known ground truth.
  • DF-VAE components: The DF-VAE ablation compares the full method with variants removing MAdaIN, hourglass structure extraction, or unpaired data construction.These variants are evaluated in a many-to-many face swapping setting.
  • DF-VAE components: The full DF-VAE method shows high-fidelity results in the many-to-many face swapping ablation.

G. Details of Benchmark Baselines

The benchmark evaluates four video-level and one image-level face forgery detection baseline, spanning 3D convolution, segment-based aggregation, recurrent temporal modeling, and frame-level classification.

  • Baseline selection: Five baselines comprise C3D, TSN, I3D, ResNet+LSTM, and XceptionNet for face forgery detection.Four methods operate at video level, while XceptionNet operates at image level.
  • Video-level baselines: C3D uses 3D convolutions to capture spatiotemporal video features from non-overlapping 16-frame clips.The input face images are resized to 112 × 112.
  • Video-level baselines: TSN models long-range temporal structure by fusing class scores from snippets sampled across three video segments.The experiment uses BN-Inception with only the RGB stream and 224 × 224 images.
  • Video-level baselines: I3D adds a temporal dimension to Inception-V1 filters and pooling kernels and uses randomly selected 64-frame snippets.Face images are resized to 224 × 224.
  • Video-level baselines: ResNet+LSTM extracts spatial features with ResNet-50 and aggregates temporal dependencies using an LSTM with 512 hidden units.An additional fully connected layer performs classification after temporal aggregation.
  • Image-level baseline: XceptionNet performs image-level detection with depthwise-separable convolutions and averages frame classification scores into a video prediction.The model uses 299 × 299 face images and trains without freezing any layer.

H. More Examples of Data Collection

The paper illustrates diverse source videos, common real-world perturbations, and component-level face-swapping results used to characterize DeeperForensics-1.0.

  • Source data: Collected source videos vary across identities, poses, expressions, emotions, lighting conditions, and 3DMM blendshapes.The source videos are intended for release to support further research.
  • Perturbations: DeeperForensics-1.0 includes seven perturbation types and mixtures of two, three, or four distortions.Examples combine Gaussian blur, JPEG compression, white Gaussian noise in color components, and color saturation change.
  • Temporal-loss visualization: Temporal-loss effectiveness is visualized with RGB per-pixel Euclidean error maps and mean errors displayed above the images.The color scale corresponding to error values appears beside the images.
  • DF-VAE visualization: The DF-VAE component ablation displays source and target faces, full-method outputs, and outputs produced after removing three components.The compared components are MAdaIN, hourglass structure extraction, and unpaired data construction.
Loading 2001.03024v2…