Source-linked AI summary
FaceForensics++: Learning to Detect Manipulated Facial Images
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner
TL;DR
Facial forgery detection lacks standardized evaluation under realistic compression and resizing, while manipulation traces can be difficult to detect. The paper introduces a large-scale benchmark and supervised CNN-based detectors for four facial manipulation methods, showing that trained detectors detect even low-quality fakes and outperform humans.
Problem
Facial forgery detection lacks a benchmark for standardized evaluation, and robustness to compression and resizing remains insufficiently addressed.
Method
The paper builds a benchmark and dataset from four facial manipulation methods, then trains supervised CNN-based forgery detectors for evaluation.
Results
Trained forgery detectors detect visually realistic manipulations, including low-quality video, and outperform human observers; XceptionNet performs best among tested variants.
Takeaways & Limitations
The publicly available dataset and standardized benchmark provide a basis for follow-up research and transfer learning in digital media forensics.
Abstract
from arXiv · showhide
The rapid progress in synthetic image generation and manipulation has now come to a point where it raises significant concerns for the implications towards society. At best, this leads to a loss of trust in digital content, but could potentially cause further harm by spreading false information or fake news. This paper examines the realism of state-of-the-art image manipulations, and how difficult it is to detect them, either automatically or by humans. To standardize the evaluation of detection methods, we propose an automated benchmark for facial manipulation detection. In particular, the benchmark is based on DeepFakes, Face2Face, FaceSwap and NeuralTextures as prominent representatives for facial manipulations at random compression level and size. The benchmark is publicly available and contains a hidden test set as well as a database of over 1.8 million manipulated images. This dataset is over an order of magnitude larger than comparable, publicly available, forgery datasets. Based on this data, we performed a thorough analysis of data-driven forgery detectors. We show that the use of additional domainspecific knowledge improves forgery detection to unprecedented accuracy, even in the presence of strong compression, and clearly outperforms human observers.
1. Introduction
The introduction frames facial manipulation as a widespread societal concern and distinguishes expression manipulation from identity manipulation. It presents a standardized benchmark, a large supervised dataset, and CNN-based detection intended to outperform human observers.
- Motivation: Facial manipulation has become ubiquitous and can enable harms such as defaming people by replacing one person’s face with another’s.The introduction identifies faces as a central target of current manipulation methods.
- Manipulation categories: Facial forgeries comprise expression manipulation, exemplified by real-time Face2Face expression transfer, and identity manipulation through face swapping.DeepFakes performs face swapping using deep learning, whereas simple computer-graphics approaches can run in real time.
- Detection approach: The authors train supervised convolutional neural networks to learn image features for automatically and reliably detecting facial manipulations.They report that the resulting detectors outperform human observers by a significant margin.
- Benchmark: The paper proposes an automated benchmark for standardized facial-manipulation detection under random compression and random dimensions, including a human baseline.The benchmark evaluates state-of-the-art detectors and a facial-manipulation-specific detection pipeline.
- Contributions: The work contributes more than 1.8 million manipulated images from 1,000 videos with pristine sources and target ground truth, plus broad detector evaluation and a tailored method.The dataset supports supervised learning, while the evaluation covers hand-crafted and learned forgery detectors in various scenarios.
2. Related Work
Related work spans facial manipulation and synthesis methods, multimedia-forensics techniques, face-manipulation detectors, and forensic datasets. The paper positions its database of more than 1.8 million images from 4000 fake videos as substantially larger than existing datasets.
- Face Manipulation Methods: Facial manipulation research progressed from Video Rewrite and early automatic face swapping to real-time facial reenactment using 3D reconstruction and image-based rendering.These methods alter mouth movements, swap faces, or modify facial movements in commodity video streams.
- Face Manipulation Methods: Deep-learning synthesis methods use GANs, feature interpolation, and Fader Networks to age faces, generate viewpoints, and alter attributes such as skin color, age, mustache, or smiling.These approaches target facial attributes and appearance transformations.
- Multimedia Forensics: Multimedia forensics evolved from hand-crafted statistical and physics-based artifacts toward supervised and unsupervised CNN-based solutions for image and video authenticity.Face-specific detectors address computer-generated, morphed, spliced, swapped, and DeepFake faces using blinking, color, texture, shape, and learned inconsistency cues.
- Multimedia Forensics: Existing face-manipulation detectors report impressive results, but robustness issues often remain unaddressed.Methods may exploit synthesis artifacts such as eye blinking or learn subtle low-level and high-level inconsistencies.
- Forensic Analysis Datasets: More than 1.8 million images from 4000 fake videos form the paper’s database, an order of magnitude larger than existing datasets.Earlier forensic datasets were often manually created under controlled conditions, and some contained only limited image manipulations or video coverage.
3. Large-Scale Facial Forgery Database
FaceForensics++ extends the original FaceForensics dataset into a large-scale benchmark built from 1,000 Internet videos manipulated by four state-of-the-art facial manipulation methods. It also provides pixel-level ground-truth masks to support forgery localization.
- FaceForensics++ extends the preliminary FaceForensics dataset as a core contribution of the paper.
- 1,000 pristine YouTube videos are manipulated using four automated state-of-the-art face manipulation methods.The videos were collected in the wild to imitate realistic scenarios.
- The dataset combines two computer graphics-based methods, Face2Face and FaceSwap, with two learning-based methods, DeepFakes and NeuralTextures.All four methods require source and target actor video pairs as input.
- Pixel-level ground-truth masks identify modified pixels and support training forgery localization methods.
- Face2Face: Face2Face transfers source-video expressions onto a target video while preserving the target person's identity.The adapted system reconstructs the face for resynthesis under different illumination and expressions.
4. Forgery Detection
Forgery detection is evaluated as per-frame binary classification, comparing human observers with automated detectors that exploit face-region information. Domain-specific preprocessing and learned networks improve robustness, while compression, NeuralTextures, and smaller training corpora remain challenging.
- The evaluation uses fixed training, validation, and test splits of 720, 140, and 140 videos, respectively, with all results reported on the test set.
- User Study: 204 participants provide the human baseline by classifying 60 randomly selected pristine or manipulated images under 2-, 4-, or 6-second viewing limits.Images vary in quality and manipulation method, with a 50:50 split between pristine and fake images.
- User Study: Human accuracy decreases from 68.7% to 58.7% as video quality declines, while different time limits produce no significant performance differences.Figure 4 reports 68.69% on raw videos, 66.57% on high-quality videos, and 58.73% on low-quality videos.
- Automated Detection: Face tracking extracts a conservative 1.3× enlarged crop around the tracked face before classification, improving performance over using the whole image.The pipeline feeds the extracted face region into a learned classification network, while evaluated variants include hand-crafted and learned-feature methods.
- Automated Detection: All detectors perform highly on raw inputs, but compression especially reduces hand-crafted and shallow-CNN performance; deeper neural networks handle compression better, with XceptionNet achieving the strongest results.On the combined manipulation dataset, automated detectors outperform human performance by a large margin, including comparison with a full-image XceptionNet baseline.
- Robustness Analysis: Detection accuracy is lower for NeuralTextures because its separately trained manipulation models create greater artifact variation, and performance increases as the XceptionNet training corpus grows.Increasing training data is particularly important for low-quality video footage.
5. Benchmark
The paper introduces a competitive facial forgery detection benchmark built from additional videos manipulated by four methods under realistic postprocessing conditions. It evaluates previously trained low-quality models and makes the benchmark publicly available for standardized comparison.
- Benchmark construction: The benchmark uses 1000 additional videos, with a subset manipulated using each of four methods and repeatedly obscured through varied resizing, compression, and bit-rate conditions.These transformations are intended to reflect postprocessing applied to uploaded videos, such as on social networks.
- Evaluation: The evaluation uses low-quality versions of previously trained models as baselines and generally supplies pre-extracted face regions, except for Full Image XceptionNet.Detection results are reported separately for each manipulation method.
- Benchmark release: The benchmark is publicly available to support standardized comparisons in subsequent facial forgery detection research.The authors express hope that the release will enable consistent evaluation of follow-up work.
6. Discussion & Conclusion · Appendix
The paper concludes that trained, learning-based detectors can identify state-of-the-art facial manipulations, including low-quality video, and introduces a large dataset and standardized benchmark for future research. The appendix documents data acquisition, classification experiments, and hyperparameters.
- 6. Discussion & Conclusion: Trained forgery detectors identify visually realistic state-of-the-art facial manipulations, including challenging low-quality video that humans and hand-crafted features struggle with.The paper attributes this capability to learning-based approaches and domain-specific knowledge.
- 6. Discussion & Conclusion: The authors introduce a standardized benchmark and a novel manipulated-face video dataset that exceeds existing resources in scale.The benchmark, image data, and trained models are publicly available.
- 6. Discussion & Conclusion: The benchmark is intended to support follow-up work on manipulation detectability and robustness to post-processing such as compression.The dataset and benchmark are already used by other researchers, with transfer learning identified as a key forensic-community interest.
- 6. Discussion & Conclusion: Future detection methods must handle newly emerging manipulation techniques with little to no training data.The discussion highlights transfer learning as particularly relevant to this challenge.
- Appendix: FaceForensics++ evaluates facial-manipulation detectors on a large-scale dataset generated with four different manipulation methods.The study also proposes an automated benchmark for comparing future approaches.
- Appendix: The supplemental appendix documents pristine-data acquisition and the exact binary-classification experiment counts.It also describes manipulation-classification evaluation and lists hyperparameters for manipulation and detection techniques.
A. Pristine Data Acquisition
The benchmark’s pristine videos were collected from YouTube and screened to support reliable facial manipulation. Selection required front-facing, unobstructed faces and sufficiently long target sequences, yielding 1,000 manually screened sequences with 509,914 images.
- Source selection: Pristine videos were collected in the wild from YouTube for a realistic evaluation scenario.
- Source selection: Videos were selected using face- and news-related tags, including “face”, “newscaster”, “newsprogram”, “interview”, and “blog”.
- Quality criteria: Target faces had to be nearly front-facing and free of occlusions to prevent manipulation failures and strong artifacts.
- Quality criteria: 280 frames was the minimum sequence length for methods requiring sufficiently many training images, excluding FaceSwap.
- Quality criteria: 1,000 video sequences containing 509,914 images remained after manual screening for quality and face occlusions.
B. Forgery Detection
The section reports forgery-detection accuracies across manipulation-specific and all-method training, analyzes training-corpus size, and presents user-study results by viewing device. Results cover raw and compressed data for DeepFakes, Face2Face, FaceSwap, NeuralTextures, and pristine images.
- Manipulation-specific detection: Table 4 reports manipulation-specific detector accuracies for raw and compressed datasets across DeepFakes, Face2Face, FaceSwap, and NeuralTextures.The detectors are trained on their respective manipulation methods.
- All-method training: Table 5 reports detection accuracies for detectors trained on all manipulation methods and evaluated on each manipulation method or pristine data.The listed categories are DeepFakes, Face2Face, FaceSwap, NeuralTextures, and Pristine.
- Training corpus size: Table 6 analyzes training-corpus size using XceptionNet detectors trained on single manipulation methods or all manipulation methods.The categories include DeepFakes, Face2Face, FaceSwap, NeuralTextures, and All.
- User study: Table 7 reports user-study results by viewing device, with 99 participants using a PC and 105 using a mobile phone.Results cover DeepFakes, Face2Face, FaceSwap, NeuralTextures, and pristine images.
C. Classification of Manipulation Method
The XceptionNet classifier distinguishes four manipulation methods from pristine images using five output classes. Its accuracy is highest on raw data and declines as compression quality decreases.
- C. Classification of Manipulation Method: Five class probabilities enable XceptionNet to classify four manipulation methods and pristine images.The network is trained on the full dataset of pristine and manipulated images.
- C. Classification of Manipulation Method: 99.03% accuracy is achieved on raw data.This result comes from training on the full dataset containing all pristine and manipulated images.
- C. Classification of Manipulation Method: 95.42% accuracy is achieved with high quality compression, declining from raw-data performance.The high-quality compression result is lower than the 99.03% raw-data accuracy.
- C. Classification of Manipulation Method: 80.49% accuracy is achieved on low quality images.This is the lowest reported accuracy among raw, high-quality-compressed, and low-quality images.
D. Hyperparameters
This section documents the hyperparameters used for the manipulation methods and forgery-detection classification approaches to support reproducibility.
- D. Hyperparameters: The hyperparameters are organized into manipulation methods and classification approaches for forgery detection.The section provides these implementation details for reproducibility.
D.1. Manipulation Methods
The benchmark uses DeepFakes and NeuralTextures as learning-based manipulation methods, while the remaining methods use their default parameters.
- DeepFakes and NeuralTextures are learning-based manipulation methods; the other methods use default parameters.
D.2. Classification Methods
The forgery-detection pipeline compares five convolutional-neural-network classifiers trained with Adam under method-specific learning-rate and batch-size settings. Evaluation uses repeated validation checks, fixed per-video image counts, and procedures addressing frame-count and real/fake imbalance.
- Classification Methods: Five classification approaches based on convolutional neural networks are evaluated in the forgery-detection pipeline using the Adam optimizer.The approaches use different learning-rate and batch-size parameters.
- Classification Methods: 10−5 learning-rate with batch-size 16 is used for the Cozzolino et al. network, while Bayar and Stamm uses 10−5 learning-rate with batch-size 64.These settings are reported for two of the evaluated network proposals.
- Classification Methods: Validation accuracies are computed ten times per epoch, and training stops after 10 consecutive checks without a change in validation accuracy.Validation and test accuracies are computed on 100 images per video.
- Classification Methods: 270 images per video are used for training evaluation to account for frame-count imbalance, while binary-task real/fake imbalance is also addressed.Fake images are roughly four times more numerous than pristine images.