Source-linked AI summary
FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner
TL;DR
Reliable detection of realistic facial video manipulation is hindered by limited forensic data and weakened traces after compression. FaceForensics introduces a large Face2Face-generated dataset with classification, segmentation, and generative-refinement benchmarks. It establishes broad forensic baselines, while refinement improves visual quality but only slightly affects detection by models trained on the forged outputs.
Problem
Limited forensic training data hampers deep-network manipulation detection and makes evaluation difficult, while compression can remove low-level traces.
Method
FaceForensics builds a large Face2Face-based facial-manipulation dataset and benchmarks classification, segmentation, and supervised generative refinement.
Results
The dataset exceeds existing public forensic datasets by orders of magnitude and provides strong modern-deep-learning detection baselines; refinement improves visual quality but only slightly encumbers detection.
Takeaways & Limitations
FaceForensics supports realistic forensic analysis and evaluation of manipulation refinement using known ground truth across detection and segmentation tasks.
Takeaways & Limitations
The refiner mainly improves visual quality, while its effect on detection is measured for deep-learning methods trained exactly on the forged output data.
Abstract
from arXiv · showhide
With recent advances in computer vision and graphics, it is now possible to generate videos with extremely realistic synthetic faces, even in real time. Countless applications are possible, some of which raise a legitimate alarm, calling for reliable detectors of fake videos. In fact, distinguishing between original and manipulated video can be a challenge for humans and computers alike, especially when the videos are compressed or have low resolution, as it often happens on social networks. Research on the detection of face manipulations has been seriously hampered by the lack of adequate datasets. To this end, we introduce a novel face manipulation dataset of about half a million edited images (from over 1000 videos). The manipulations have been generated with a state-of-the-art face editing approach. It exceeds all existing video manipulation datasets by at least an order of magnitude. Using our new dataset, we introduce benchmarks for classical image forensic tasks, including classification and segmentation, considering videos compressed at various quality levels. In addition, we introduce a benchmark evaluation for creating indistinguishable forgeries with known ground truth; for instance with generative refinement models.
1 Introduction
Realistic facial video manipulation threatens trust in digital communication, while limited forensic training data constrains detection research. FaceForensics addresses this gap with a large dataset and benchmarks spanning detection, segmentation, compression, and generative refinement.
- Real-time facial video editing can undermine trust in digital communication and complicate filtering manipulated content on social networks.
- Limited, mostly manually created forensic datasets constrain deep-network training and make method comparison difficult.
- Over 500,000 frames from 1004 videos form a Face2Face-generated dataset covering source-to-target and self-reenactment manipulations.
- The dataset benchmarks forgery classification and pixel-level segmentation on videos compressed at various quality levels.
- Self-reenactment supports supervised evaluation of generative refinement using indistinguishable-image generation with known ground truth.
2 Related Work
Prior work spans increasingly capable face manipulation, multimedia forensics, and datasets for authenticity analysis. The review emphasizes that robustness remains difficult because compression, resizing, and targeted post-processing can remove manipulation traces.
- Face Manipulation Methods: Face manipulation research progressed from video rewriting and face replacement to real-time facial reenactment and expression transfer.
- Face Manipulation Methods: Deep-learning methods synthesize faces by changing viewpoints, age, skin color, facial attributes, and expressions.
- Multimedia Forensics: Multimedia forensics combines handcrafted artifact features with CNN-based approaches to assess image and video authenticity.
- Multimedia Forensics: Compression and resizing can launder manipulation traces, while targeted post-processing creates additional counter-forensic challenges.
- Datasets: Existing manipulation datasets were generally small or image-focused, with few realistic video datasets providing ground truth.
3 The FaceForensics Dataset
FaceForensics is built from 1004 YouTube videos and processed with an automated Face2Face reenactment pipeline. It provides source-to-target manipulations with pixel masks and self-reenactment pairs with ground truth for generative modeling.
- The dataset contains 1004 videos representing unique identities and is divided into source-to-target and self-reenactment variants.
- Data Collection: Videos were collected from YouTube, filtered for face-containing sequences, and manually screened for quality.
- Data Processing: An automated Face2Face variant reconstructs a temporary 3D identity, tracks expressions, and estimates identity and static texture from selected poses.
- Source-to-Target Dataset: Source-to-target reenactment transfers 76 Blendshape expression coefficients and stores source, target, output, and per-pixel forgery-mask data.
- Source-to-Target Dataset: The source-to-target pipeline uses randomly chosen videos and a person-specific mouth database that can limit expression coverage and cause distortions.
- Self-Reenactment Dataset: Self-reenactment uses the same video as source and target to produce manipulated and ground-truth image pairs for generative training.
4 Forgery Classification Task
The forgery classification task detects whether a face is manipulated on a per-frame basis, comparing forensic, CNN-based, and transfer-learning methods under varying compression. Performance is relatively high without compression but declines for compressed videos, with deep networks more robust than handcrafted features and shallow CNNs.
- Forgery classification is posed as binary per-frame detection of whether an image contains a forged face.
- The benchmark evaluates classification accuracy with no compression, light H.264 compression, and strong H.264 compression.Light and strong compression use quantization parameters 23 and 40, respectively.
- The compared methods include handcrafted steganalysis features with SVM, CNN-based forensic models, two-stream networks, and transfer-learning XceptionNet.
- Without compression, all methods achieve relatively high performance, while compression particularly reduces handcrafted-feature and shallow-CNN performance.
- Deep neural networks handle compressed videos better, with XceptionNet slightly outperforming Zhou et al.'s method.
5 Forgery Segmentation Task
The forgery segmentation task localizes manipulated pixels rather than only classifying faces. The benchmark focuses on deep-learning methods and evaluates precision–recall behavior across uncompressed and compressed videos, where performance deteriorates as compression increases.
- Forgery segmentation performs pixel-level localization of manipulated images and is described as a very challenging task.
- Camera-artifact methods were discarded because they produced unsatisfactory results even on uncompressed frames, so evaluation focuses on deep-learning methods.
- XceptionNet is adapted for localization, while methods from Cozzolino et al. and Rahmouni et al. already support localization without further adaptation.
- At test time, the network processes 128 × 128 patches with stride 16 and assigns each patch's manipulation probability to its central 16 × 16 region.
- Precision–recall performance degrades with increasing compression because false positives and false negatives become more frequent.
- At the highest compression rate, only XceptionNet continues to provide good results, while the other methods produce useless heatmaps.
6 Refinement Task
The paper refines Face2Face forgeries with a supervised autoencoder trained on self-reenactment data, improving visual quality but only slightly reducing detector performance under compression.
- 521,406 manipulated frames with target ground truth pairs support supervised refinement training.
- The autoencoder uses skip connections and is pretrained unsupervised on unlabeled VGGFace2 before supervised fine-tuning on FaceForensics self-reenactment data.The architecture predicts a same-resolution output and uses known target ground truth during fine-tuning.
- Face2Face artifacts around the chin, cheek, nose, and mask boundaries are significantly improved by the refinement network.
- 6.1 Perceptual Evaluation of the Refinement: Human accuracy is about 50% on highly compressed images, while refinement makes visual differences harder for human observers to detect.Participants viewed raw and refined forgeries under multiple compression conditions.
- 6.2 Quantitative Evaluation: The autoencoder slightly lowers detection accuracy on compressed data, but XceptionNet still detects refined fakes with high accuracy.The comparison uses the same classifier architecture and training-data amount for refined and unrefined fakes.
- 6.2 Quantitative Evaluation: Visual quality remains an inadequate evaluation metric, motivating future refinement with generative adversarial networks.
7 Conclusions
The paper contributes a large manipulated-video dataset and benchmarks for forgery identification and segmentation. It also shows that autoencoder refinement improves visual quality while only slightly encumbering deep-learning detection.
- The dataset exceeds existing publicly available forensic datasets by orders of magnitude and supports benchmarks for forged-image identification and segmentation.
- Handcrafted approaches are highly challenged by realistic compression, whereas modern deep-learning architectures provide a strong baseline for facial-manipulation detection.
- The supervised autoencoder improves forgery appearance but only slightly reduces detection performance for deep-learning methods trained on the forged output data.
A User Study Interface
The user-study interface presents randomly selected real and fake images for three seconds, with participants choosing whether each image is fake or real.
- Images are randomly drawn from no-, easy-, and hard-compression sets with equal numbers of fake and real examples for each participant.
- Participants view each image for three seconds before selecting either fake or real.
B Forgery Segmentation Examples
The segmentation examples compare three CNN-based methods across real and manipulated frames, showing that XceptionNet remains more effective as compression increases.
- The examples compare Rahmouni et al., Cozzolino et al., and XceptionNet using heatmaps for original and manipulated frames.Fake inputs additionally include ground-truth masks.
- On uncompressed videos, XceptionNet best localizes manipulated regions while producing few false positives on real videos.
- Under easy compression, XceptionNet still provides reliable segmentation of altered pixels.
- At stronger compression, segmentation becomes poor, but XceptionNet remains much better than the compared methods.Rahmouni et al. and Cozzolino et al. produce almost random heatmaps on compressed videos.