Source-linked AI summary
DeepFake Detection Based on the Discrepancy Between the Face and its Context
Yuval Nirkin, Lior Wolf, Yosi Keller, Tal Hassner
TL;DR
Face-swapping detectors need cues that remain useful when generated images become more realistic and artifact-specific flaws change. This paper compares identity signals from the manipulated face and its unchanged context, using their discrepancy alongside conventional artifact cues. It reports state-of-the-art results on major benchmarks and generalization to unseen manipulation methods.
Problem
Face-swapping methods alter apparent identity while leaving surrounding context unchanged, creating a need for detection cues less dependent on technique-specific artifacts.
Method
The method segments face and context regions, applies separate identity-recognition networks, and compares their pseudo-probability outputs with a swapping-artifact network.
Results
The method outperforms existing state-of-the-art schemes on FaceForensics++, Celeb-DF-v2, and DFDC and generalizes to unseen manipulation methods.
Takeaways & Limitations
The face-context identity discrepancy is a complementary detection signal that can be combined with artifact-based classifiers.
Takeaways & Limitations
The approach assumes manipulation affects the internal face while leaving surrounding context unchanged, and challenging low-contrast or blurry images remain a boundary.
Abstract
from arXiv · showhide
We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions. These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real vs. fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++, Celeb-DF-v2, and DFDC benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.
1 INTRODUCTION
Face swapping changes the inner face while leaving identity-bearing context largely unchanged, creating discrepancies that can detect manipulated images. The proposed method compares separate face and context identity representations and complements artifact-based detectors.
- Motivation: Face swapping is distinct from face reenactment because it changes apparent identity, motivating manipulation cues specific to identity changes.The paper focuses on swapping scenarios in which the apparent identity is changed, rather than expression or lip-motion reenactment.
- Motivation: The method assumes manipulation affects the internal face while the surrounding context remains unchanged.The context includes regions such as the head, neck, ears, and hair outside the internal face.
- Motivation: Context alone provides strong identity cues, making mismatches between manipulated faces and unchanged context detectable.The paper reports that its findings support the context assumption and align with prior reports that context can identify faces.
- Approach: The proposed signal compares two identity vectors produced separately from the face region and its context to seek identity discrepancies.The face and context are represented using two independently trained face-recognition networks.
- Approach: The comparison requires no prior knowledge of the source or target identity in the image.The method compares representations of one or two unknown identities derived from the face and context.
- Contribution: The cue is designed to generalize across manipulation methods and complement artifact detectors because it relies on face-swap design rather than technique-specific artifacts.The authors report state-of-the-art performance on FaceForensics++, Celeb-DF-v2, and DFDC, plus results on additional swapping benchmarks.
2 RELATED WORK
Prior work spans face manipulation methods, deep-learning detection approaches, and benchmark construction. The paper positions its cue as less dependent on incidental artifacts or synthesis-method biases than conventional artifact-focused detection.
- Face manipulation techniques: Face manipulation methods evolved from semi-automatic techniques to deep-learning and GAN-based systems producing near photo-realistic images and videos.The related work covers early privacy, recreation, and entertainment uses alongside later DeepFake applications.
- Detection methods: Detection research commonly trains classifiers to distinguish real from fake media, using handcrafted features, deep networks, temporal cues, attention, landmarks, or multiple cues.The cited approaches address detection and, in some cases, manipulated-region localization.
- Benchmarks: Standard benchmarks include FaceForensics, DeepFake-TIMIT, Celeb-DF, VTD, FaceForensics++, DFD, and DFDC.These benchmarks include multiple manipulation techniques rather than relying on a single synthesis method.
- Detection methods: Using few synthesis methods can introduce dataset biases tied to generation artifacts or training data.The paper motivates an approach intended to be invariant to these incidental biases rather than targeting particular artifacts.
3 RECOGNITION OF FACES AND THEIR CONTEXT
The recognition stage separates a detected crop into face and context regions, then trains dedicated networks to identify people from each region. Their final pseudo-probability outputs are compared because context retains useful identity information but independently learned intermediate representations do not align well.
- 3 RECOGNITION OF FACES AND THEIR CONTEXT: The method focuses on the segmented face and its surrounding context as separate sources of identity cues.Two complementary face-recognition networks are used for these application-specific regions.
- 3.1 Detecting and segmenting faces: Preprocessing enlarges detected face boxes by 20%, resizes crops to 299×299 pixels, and uses segmentation to produce face image I_f and complementary context image I_c.The segmentation mask separates foreground face pixels from background context pixels.
- 3.2 Recognition networks: Network E_f maps face pixels and E_c maps context pixels to pseudo-probability vectors over the same subject classes.Both networks use Xception-based recognition architectures and are trained on VGGFace2 identities.
- 3.2 Recognition networks: Context-only recognition remains a strong identity cue, although the internal-face network is more accurate.This pattern appears in evaluation on VGGFace2 and LFW, supporting the use of context in discrepancy detection.
- 3.2 Recognition networks: The method compares final per-subject pseudo-probabilities rather than penultimate-layer activations because independently trained face and context networks produce poorly matched intermediate representations.Figure 3 contrasts the two representation choices on LFW verification accuracy.
- Method overview: The full pipeline subtracts face and context pseudo-probability vectors, combines the difference with a swapping-artifact representation, and sends both to classifier D.The method overview includes face identification networks E_f and E_c, swapping network E_s, and final classifier D.
4 FAKE DETECTION USING FACES VS. CONTEXT
The method combines identity-based face–context discrepancy cues with dedicated detectors for swapping and reenactment, then feeds their signals into a final real-versus-fake classifier. Specialized training, frozen recognition networks, and face-selection rules support inference across multiple manipulation types.
- Architecture: The approach combines recognition networks Ef and Ec with binary Xception networks Es for swapping and Er for reenactment detection.Er is optional in the illustrated architecture and is trained separately for reenactment manipulations.
- Training: Recognition-network weights are kept fixed after pretraining because leaving them unfrozen reduces accuracy when generalizing to unseen manipulation methods.Es and Er are fine-tuned during final training while Ef and Ec remain frozen.
- Face–context discrepancy: Ef and Ec process face and context images separately to compute a discrepancy feature vector vd.The face discrepancy network predicts whether the face and context share the same identity.
- Signal combination: The system concatenates vd, vs, and vr, then passes the result to classifier D for a real-versus-fake binary prediction.D uses an initial linear layer, batch normalization, ReLU, and a final linear layer, trained with logistic loss.
- Training: Es and Er are pretrained on manipulation-specific FaceForensics++ subsets, with Es using FaceSwap and Deepfakes and Er using Face2Face and NeuralTextures.Training uses compressed C23 and C40 videos rather than raw videos.
- Inference: At inference, faces smaller than 64 pixels are discarded, segmentation thresholds remove false detections, and the minimum patch score is used when only one face is manipulated.If filtering removes all detections, fallback rules retain the largest or maximal-pixel face patch.
5 EXPERIMENTAL RESULTS
The method is evaluated on standard face-swapping benchmarks, an image benchmark, and unseen swapping methods. It achieves leading benchmark performance and improves generalization over the baseline, especially when artifacts are scarce.
- Datasets and evaluation: The evaluation covers FaceForensics++, DFDC, Celeb-DF-v2, and an additional test set using two swapping methods.The additional methods are FSGAN and Nirkin et al.; the experiments assess generalization to techniques not included in FaceForensics++.
- Datasets and evaluation: Frame-level AUC is used for comparing methods on frames containing detected faces.AUC avoids selecting classification thresholds for per-frame outputs.
- Face-swap detection: The method achieves the best AUC scores on all reported face-swap benchmarks.On FaceForensics++ DeepFakes, performance is similar to the state of the art because accuracy is saturated; Celeb-DF-v2 results indicate improved generalization.
- FaceForensics++ image benchmark: Our total accuracy outperforms previous methods by a large margin on the FaceForensics++ image benchmark.Per-category accuracy is threshold-dependent, so the Total score is the meaningful comparison.
- Generalization and ablation: The proposed method significantly outperforms the baseline on both unseen swapping methods, with a larger gap for FSGAN faces.FSGAN produces fewer apparent artifacts, whereas the 3DMM-based method produces artifacts more similar to those in known methods.
- Generalization and ablation: End-to-end training generalizes less effectively because it disrupts the face and context classifiers’ roles in extracting aligned identity representations.The face discrepancy signal alone is not competitive with fake-detection networks but contributes to the overall method.
6 CONCLUSION
The paper concludes that common design patterns in face identity manipulation enable a detection cue based on face–context discrepancies. This cue complements conventional artifact-based classifiers and is intended to remain useful as visual quality improves.
- Conclusion: Recent face manipulation methods follow similar patterns that can support a shared detection cue.The proposed cue uses commonalities across identity-manipulation methods rather than method-specific artifacts.
- Conclusion: The face–context cue is complementary to conventional real/fake classifiers and can be used alongside them.The qualitative DFDC examples include fakes detected by the proposed method but missed by XceptionNet.
- Conclusion: Avoiding the cue would require broader integration of the new identity into the image, according to the paper’s conclusion.The authors contrast this requirement with artifact detectors’ susceptibility to improvements in generated-image quality.
APPENDIX A SEGMENTATION NETWORK DETAILS
The segmentation network extracts the face region used to separate facial content from surrounding context. It uses a modified U-Net and training augmentations intended to improve robustness.
- Architecture: The face segmentation network uses a U-Net architecture with bilinear interpolation followed by convolution for upsampling.Inputs are cropped and resized to 256x256 resolution.
- Training: Training uses a face segmentation dataset generated with publicly available code and applies geometric, color, flip, and blur augmentations.The reported training setup uses batch size 48 and 40,000 iterations per epoch.
APPENDIX B TRAINING DETAILS
The appendix specifies optimization settings and staged training procedures for the identity, manipulation-specific, and full-pipeline networks.
- Optimization: Training used four NVIDIA Tesla P100 GPUs and an Intel Xeon CPU with 64 cores for 90 epochs using Adam optimization.The learning rate was halved every ten epochs; initial learning rate and batch size varied by training stage.
- Identity networks: Identity networks were trained on cropped and augmented VGGFace2 images resized to 299x299 for the Xception architecture.Crops used squared, 20%-extended bounding boxes and horizontal flipping with probability 0.5.
- Identity networks: Face and context identity networks received complementary inputs by replacing the excluded region with a constant color.Both networks used batch size 192 and initial learning rate 0.0002.
- Pretraining: Manipulation-specific networks Es and Er were pretrained on designated FaceForensics++ video subsets using randomly selected detected-face frames.Training used 40,000 iterations per epoch, batch size 96, and initial learning rate 0.0002.
- Full pipeline: In final pipeline training, Es, Er, and D were optimized while Ef and Ec remained frozen, except in the end-to-end ablation.The discriminator was randomly initialized and trained from scratch with batch size 64, 20,000 iterations per epoch, and initial learning rate 0.0001.
APPENDIX C RUNTIME PERFORMANCE
Runtime was measured for the complete pipeline apart from face detection, using a single Tesla V100 GPU and an eight-core Xeon CPU.
- Measurement scope: Runtime testing included segmentation and preprocessing but excluded the face detection step.The detector used was DSFD, which was not optimized for runtime and was originally designed for images rather than videos.
- Detection stage: The face detection stage could potentially be replaced by a face-tracking algorithm for runtime use.The passage notes that many tracking algorithms can run faster, but the comparison is not quantified.