Source-linked AI summary
Face X-ray for More General Face Forgery Detection
Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, Baining Guo
TL;DR
Face forgery detectors often overfit to known manipulation techniques, leaving detection of unseen forgeries unresolved. The paper introduces face X-ray, a blending-boundary representation trainable without state-of-the-art fake images, and reports strong generalization to unseen forgeries. Its scope is limited when images are entirely synthetic or heavily compressed.
Problem
Supervised face forgery detectors can perform well on known techniques but experience significant performance drops on unseen manipulation methods.
Method
Face X-ray represents intrinsic discrepancies at blending boundaries and can be trained using blended images composed only from real images.
Results
The framework remains effective on unseen face forgeries and largely exceeds competing state-of-the-art face forgery detectors.
Takeaways & Limitations
A detector based on the common blending step can generalize across most existing face manipulation algorithms without using their generated fake images for training.
Takeaways & Limitations
The method relies on a blending step and suffers performance degradation on low-resolution or heavily compressed images.
Abstract
from arXiv · showhide
In this paper we propose a novel image representation called face X-ray for detecting forgery in face images. The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources. It does so by showing the blending boundary for a forged image and the absence of blending for a real image. We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image. For this reason, face X-ray provides an effective way for detecting forgery generated by most existing face manipulation algorithms. Face X-ray is general in the sense that it only assumes the existence of a blending step and does not rely on any knowledge of the artifacts associated with a specific face manipulation technique. Indeed, the algorithm for computing face X-ray can be trained without fake images generated by any of the state-of-the-art face manipulation methods. Extensive experiments show that face X-ray remains effective when applied to forgery generated by unseen face manipulation techniques, while most existing face forgery detection or deepfake detection algorithms experience a significant performance drop.
1. Introduction
Face forgery detection must generalize beyond known manipulation methods because supervised detectors often overfit to technique-specific artifacts. Face X-ray instead represents blending boundaries, enabling detection across many manipulations and remaining effective on unseen forgeries.
- Face forgery detection is important because realistic facial manipulations can evade human judgment and create trust and security concerns.
- Existing supervised detectors reach around 98% accuracy on known techniques but suffer significant performance drops on unseen manipulations.
- Face X-ray exploits blending between an altered face and an existing background, revealing intrinsic discrepancies at the blending boundary.
- The greyscale face X-ray distinguishes forged from real images and identifies the blending boundary when present.
- Face X-ray requires only a blending step, can be trained using composites of real images, and remains effective on unseen manipulation methods.
- Experiments report high accuracy on unseen forgeries and performance exceeding competing state-of-the-art face forgery detectors.
2. Related Work
Related face-forensics methods target manipulation-specific artifacts or require forged examples for supervision, limiting their generalization to new forgery types.
- Recent face manipulation techniques leave few visual clues, making face forgery detection increasingly challenging.
- Existing approaches detect artifacts from manipulation methods, whereas the proposed approach focuses on face X-ray and blending boundaries.
3. Face X-Ray
Face X-ray detects forgery by locating the blending boundary introduced when altered faces are composited into background images. It can be defined and trained using soft masks and pairs of real images, without relying on manipulation-specific artifacts or fake training images.
- Most facial manipulation methods detect, synthesize, and blend a target face into the original image.
- Existing detectors often learn artifacts from known manipulations, causing degraded performance on unseen fake images.The paper instead targets the blending boundary introduced at the third stage of manipulation.
- The blended image is modeled as IM = M ⊙IF + (1 −M) ⊙IB, where M is a soft mask delimiting the manipulated region.IF is the manipulated foreground and IB supplies the background; ⊙ denotes element-wise multiplication.
- Face X-ray is a greyscale image that reveals a blending boundary for manipulated inputs and is zero everywhere for real inputs.It represents intrinsic discrepancies between images from different sources.
- A binary mask would produce a blank face X-ray, so the method converts binary masks to soft masks with a 3×3 Gaussian kernel.Soft masks preserve nontrivial boundary responses through Bi,j = 4 · Mi,j · (1 −Mi,j).
- Nontrivial training face X-rays are generated by blending two real images with landmark-based matching, deformed face-region masks, and color correction.The generated samples are produced dynamically during training, avoiding manipulated images for supervision.
4. Face Forgery Detection Using Face X-Ray
The detection framework predicts a face X-ray from an input image and then classifies the image as real or blended. It trains these predictions jointly with pixelwise and classification losses, emphasizing face X-ray learning.
- The framework uses a convolutional network to predict a face X-ray and classify the input as real or blended.The classifier operates on the predicted face X-ray rather than directly on the image.
- The face X-ray predictor is a fully convolutional network, while classification uses global average pooling, a fully connected layer, and softmax.
- Training uses cross-entropy losses for face X-ray prediction and real-versus-blended classification.The face X-ray loss measures prediction accuracy over the feature-map pixels.
- λ = 100 balances the losses and forces the network to focus more on face X-ray prediction.The framework is trained end-to-end using backpropagation.
5. Experiments
Experiments evaluate face X-ray for generalization to unseen facial manipulations and datasets. Results show that detecting blending evidence, especially with blended-image training, improves performance beyond conventional supervised classifiers.
- Experimental Setup: FaceForensics++ provides training data from four manipulation methods, while BI consists of blended samples composited from real images.The evaluation also tests DFD, DFDC, and Celeb-DF, including synthesized videos with reduced visual artifacts.
- Generalization Ability: Known-manipulation AUC exceeds 99%, but Xception performance drops drastically on unseen manipulations.This indicates that supervised binary classification can overfit manipulation-specific artifacts.
- Generalization Ability: Face X-ray improves generalization by targeting blending evidence rather than manipulation-specific artifacts and by enabling automatically composited training samples.The framework can be trained with blended images made only from real images, without state-of-the-art fake images.
- Generalization Ability: 99.17%, 98.57%, 98.21%, and 98.13% AUC are achieved on DF, F2F, FS, and NT using only blended images.The results show competitive detection accuracy without fake images generated by the evaluated manipulation methods.
- Benchmark Results on Unseen Datasets: On unseen DFD, DFDC, and Celeb-DF datasets, blended-image training outperforms Xception, with additional out-of-distribution fake images further improving performance in most cases.The reported benchmark metrics are AUC, AP, and EER.
- Benchmark Results on Unseen Datasets: Predicted face X-rays capture the shape of ground-truth difference maps across multiple facial manipulation methods and generated blended images.Ground truth is computed from the absolute element-wise difference between manipulated and corresponding real images, then converted to grayscale and normalized.
6. Limitations
The framework has limitations tied to its blending assumption, adversarial adaptation, and image quality. Performance drops substantially on heavily compressed, low-resolution images.
- Blending assumption: Face X-ray may fail on entirely synthetic images because the method relies on a blending step.The authors note that many realistic fake-news videos nevertheless use blending during post-processing.
- Adversarial robustness: Adversarial samples can be developed against the detector as face forgery creation and detection evolve in an arms race.
- Image quality: 87.35% and 61.6% overall AUC are obtained on HQ and LQ FF++ data, respectively.Heavier compression weakens forgery evidence and makes classification more challenging.
7. Conclusion
The paper introduces face X-ray as forgery evidence based on blending-boundary discrepancies. It develops a detector trainable without state-of-the-art fake images and evaluates its generalization to unseen forgeries.
- Contribution: Face X-ray uses intrinsic image discrepancies across blending boundaries as novel evidence for face forgery detection.
- Training: The proposed detector can be trained without fake images generated by state-of-the-art face manipulation methods.
- Evaluation: Extensive experiments show accurate discrimination of unseen forged images and reliable prediction of blending regions.